Pinebookpro virtualization

Sharing my virtualization configurations and howtos here with others in case anybody would find them usable.

Currently a full 64bit support for KVM is only possible with Manjaro Linux (https://wiki.pine64.org/index.php/Pinebook_Pro_Software_Release#Manjaro_ARM) which I run off a microSD card (faster the better, use minimum of 100 mb/s read and 64 GB or more in size)

We are going to use mostly QEMU with KVM support for running the native arm64 systems, HYPERION-HERCULES for running mainframe simulator for z/OS or MVS  and lastly the SIMH machine simulator for running VAX systems (OpenVMS and UNIX)

Qemu

git clone https://github.com/qemu/qemu.git
cd qemu
./configure   
make -j5 
su
# make install

Hyperion – Hercules

git clone https://github.com/SDL-Hercules-390/hyperion 
git clone https://github.com/SDL-Hercules-390/crypto
git clone https://github.com/SDL-Hercules-390/decNumber
git clone https://github.com/SDL-Hercules-390/SoftFloat
git clone https://github.com/SDL-Hercules-390/telnet

We first need to build crypto, decnumber, softfloat and telnet

mkdir crypto64.Release 
cd crypto64.Release 
cmake ../crypto 
make
cd .. 
mkdir decNumber64.Release 
cd decNumber64.Release 
cmake ../decNumber 
make
cd ..
mkdir SoftFloat64.Release 
cd SoftFloat64.Release 
cmake ../SoftFloat 
make
cd ..
mkdir telnet64.Release
cd telnet64.Release
cmake ../telnet
make

Now we need to copy the compiled aarch64 libs to the proper hyperion directory

cd ..
mkdir hyperion/telnet/lib/aarch64
mkdir hyperion/SoftFloat/lib/aarch64
mkdir hyperion/decNumber/lib/aarch64 
mkdir hyperion/crypto/lib/aarch64

cp crypto64.Release/libcrypto64.a hyperion/crypto/lib/aarch64
cp decNumber64.Release/libdecNumber64.a hyperion/decNumber/lib/aarch64
cp SoftFloat64.Release/libsSoftFloat64.a hyperion/SoftFloat/lib/aarch64
cp telnet64.Release/libtelnet64.a hyperion/telnet/lib/aarch64
cd hyperion
./configure 
make -j4
su 
# make install

Next we need to compile x3270 terminal emulator

cd ..
wget http://x3270.bgp.nu/download/03.06/suite3270-3.6ga8-src.tgz
gunzip suite3270-3.6ga8-src.tgz 
tar -xvf suite3270-3.6ga8-src.tar
cd suite3270-3.6 
./configure

Will fail because we have an old config.guess .. download a recent one from GCC

rm config.guess
wget https://raw.githubusercontent.com/gcc-mirror/gcc/master/config.guess

./configure 
make -j4 
# make install

SIMH

git clone https://github.com/simh/simh
cd simh
make -j5

All simh binaries will be in simh/BIN directory

Preparing the Pinebook-Pro Manjaro Linux for virtualization

Manjaro aarch64 kernel is compiled with KVM support so we can take full advantage of /dev/kvm acceleration via qemu-system-aarch64  or qemu-system-arm. Once we have the hypervizor and simulators compiled, here are some examples how I handle networking in the VMs

For the networking on the Arch Linux we need to download uml_utilities and compile them, since they do not ship on the Manjaro Linux (arch) since we need the tunctl command from there to setup the network

$ wget http://user-mode-linux.sourceforge.net/uml_utilities_20070815.tar.bz2
$ bunzip uml_utilities_20070815.tar.bz2 
$ tar -xvf uml_utilities_20070815.tar
$ cd tools-20070815 
$ make 

The make will error on uml_net/hoct.c:215 undefined 
reference to ` makedev`   But this does not really matter, the tunctl binary will be built which we can copy over to /usr/bin like this  $ su # cd tunclt # cp tunctl /usr/bin

Also we would need to install the bridge-utils which are installable via pacman package and net-tools (because Im too dumb to learn ip) manager like this

# pacman -S bridge-utils net-tools

Once we have all the above in place we can prepare the network sharing script for Qemu, just make sure we execute this after each system boot

#Setup tap and bridge 
#change the username below to match yours
tunctl -t tap0 -u user  
ifconfig tap0 up
brctl addbr br0
brctl setfd br0 0
ifconfig br0 10.0.2.2 netmask 255.255.255.0 broadcast 10.0.2.255 up
brctl addif br0 tap0 
ifconfig tap0 0.0.0.0
sysctl net.ipv4.ip_forward=1
iptables --table nat -A POSTROUTING --out-interface wlan0 -j MASQUERADE

Also make sure you have this script as /etc/qemu-ifup and executable

#! /bin/sh
# Script to bring a network (tap) device for qemu up.
# The idea is to add the tap device to the same bridge
# as we have default routing to.

# in order to be able to find brctl
PATH=$PATH:/sbin:/usr/sbin
ip=$(which ip)

if [ -n "$ip" ]; then
ip link set "$1" up
else
brctl=$(which brctl)
if [ ! "$ip" -o ! "$brctl" ]; then
echo "W: $0: not doing any bridge processing: neither ip nor brctl utility not found" >&2
exit 0
fi
ifconfig "$1" 0.0.0.0 up
fi

switch=$(ip route ls |
awk '/^default / {
for(i=0;i<NF;i++) { if ($i == "dev") { print $(i+1); next; } }
}'
)

switch=br0

# only add the interface to default-route bridge if we
# have such interface (with default route) and if that
# interface is actually a bridge.
# It is possible to have several default routes too
for br in $switch; do
if [ -d /sys/class/net/$br/bridge/. ]; then
if [ -n "$ip" ]; then
ip link set "$1" master "$br"
else
brctl addif $br "$1"
fi
exit # exit with status of the previous command
fi
done

echo "W: $0: no bridge for guest interface found" >&2


Also please bear in mind that for KVM acceleration to work we only need to use the "BIG" 2 cores of the Pinebookpro; run the following script to disable them before Qemu KVM usage only 

#!/bin/bash
echo 0 > /sys/devices/system/cpu/cpu0/online 
echo 0 > /sys/devices/system/cpu/cpu1/online 
echo 0 > /sys/devices/system/cpu/cpu2/online 
echo 0 > /sys/devices/system/cpu/cpu3/online

To re-enable normal CPUs (all 6 cores) on the Pinebookpro execute the floowing script

#!/bin/bash
echo 1 > /sys/devices/system/cpu/cpu0/online 
echo 1 > /sys/devices/system/cpu/cpu1/online 
echo 1 > /sys/devices/system/cpu/cpu2/online 
echo 1 > /sys/devices/system/cpu/cpu3/online

Bear in mind that KVM acceleration only works with the first 4 cores offline !

Running Virtual Machines on the PinebookPro

Here are some examples of installer/loader scripts of various systems I have used for KVM accelerated qemu

NetBSD -current arm64   ( http://www.invisible.ca/arm/)

wget https://releases.linaro.org/components/kernel/uefi-linaro/latest/release/qemu64/QEMU_EFI.fd
/usr/local/bin/qemu-system-aarch64 \
-cpu host -M virt,accel=kvm -smp 2 \
-m 512 -nographic \
-bios QEMU_EFI.fd \
-drive file=netbsd.img,if=none,id=drive0,format=raw \
-device virtio-blk-device,drive=drive0 \
-serial tcp::4441,server,telnet,wait \
-net nic -net tap

Connect to the remote machine qemu console via telnet 4441

Here is an example of how it runs on the actual hardware


FreeBSD -current arm64 (http://ftp.freebsd.org/pub/FreeBSD/snapshots/arm64/aarch64/ISO-IMAGES/13.0/)

wget https://releases.linaro.org/components/kernel/uefi-linaro/latest/release/qemu64/QEMU_EFI.fd
./qemu-system-aarch64 \
-cpu host -M virt,accel=kvm \
-m 2048 -nographic \
-bios QEMU_EFI.fd \
-drive file=freebsd.img,format=raw,id=drive1 \
-serial tcp::4441,server,telnet,wait \
-net nic -net tap

Connect to the remote machine qemu console via telnet 4441

Here is an example how it runs on the hardware

 


CentOS 8 arm64 (http://isoredirect.centos.org/centos/8/isos/aarch64/)

wget https://releases.linaro.org/components/kernel/uefi-linaro/latest/release/qemu64/QEMU_EFI.fd

Install.sh

qemu-system-aarch64 \
-cpu host -M virt,accel=kvm -m 1024 -nographic \
-drive if=pflash,format=raw,file=QEMU_EFI.img \
-drive if=pflash,file=varstore.img \
-drive if=virtio,file=disk.img \
-drive if=virtio,format=raw,file=centos8.iso

Run.sh

qemu-system-aarch64 \
-cpu host -M virt,accel=kvm -m 1024 -nographic \
-drive if=pflash,format=raw,file=QEMU_EFI.img \
-drive if=pflash,file=varstore.img \
-drive if=virtio,file=disk.img \
-net nic -net tap

Windows 10 arm64

PLEASE NOT THAT AS OF LATEST Manjaro + Panfrost drivers (Linux kernel 5.5.0) this does not work) – the below setup should be OK for the Vanilla Manjaro from around early December 2019 (Linux kernel 5.4.) – Investigating what happened :)

Update: 29.01.2020 – Tested the simulation with 5.4 kernel from Manjaro 19.12 and all works fine https://osdn.net/projects/manjaro-arm/storage/pbpro/kde-plasma/19.12/Manjaro-ARM-kde-plasma-pbpro-19.12.img.xz   – so for now do not upgrade to 5.5.0 kernel until its sorted out if you want to run Win10 arm64 under accelerated Qemu ;)

Its sufficient to disable linux-pinebookpro update in /etc/pacman.conf as follows

IgnorePkg = linux-pinebookpro

And upgrade the system normally afterwards,

Getting the Win10 arm64 ISO  https://www.woaproject.net/viewtopic.php?t=3

Then extract the zip file with the aria

mv 19541.1000_arm64_en-us_professional_7da33442_convert.zip ARIA/
unzip 19541.1000_arm64_en-us_professional_7da33442_convert.zip 
Archive:  19541.1000_arm64_en-us_professional_7da33442_convert.zip
  inflating: aria2_download_windows.cmd  
  inflating: aria2_download_linux.sh  
  inflating: ConvertConfig.ini       
 extracting: files/convert_config_linux  
  inflating: files/aria2c.exe        
  inflating: files/convert.sh        
  inflating: files/convert_ve_plugin  
  inflating: files/7zr.exe           
  inflating: files/uup-converter-wimlib.7z

chmod +x aria2_download_linux.sh
./aria2_download_linux.sh

this will take some time (faster on SSD drivers ofcourse) 

Total translation table size: 2048
Total rockridge attributes bytes: 0
Total directory bytes: 0
Path table size(bytes): 10
Max brk space used d5000
1854834 extents written (3622 MB)

Once downloaded rename it to win10.iso

mv 19541.1000_PROFESSIONAL_ARM64_EN-US.ISO win10.iso

Create a Win10 working directory and create empty disk image to hold the installation

Working QEMU_EFI.img - USE AT YOUR OWN RISK !!!!
wget http://45.76.81.249/Pinebook64/pinebook-pro/QEMU_EFI.img

Below does not seem to work and just hangs. 
wget https://releases.linaro.org/components/kernel/uefi-linaro/latest/release/qemu64/QEMU_EFI.fd
wget https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.173-2/virtio-win-0.1.173.iso
qemu-img create -f qcow2 disk.img 20G

Install.sh

#!/bin/bash
export QEMU_AUDIO_DRV=pa

qemu-system-aarch64 \
-cpu host \
-enable-kvm \
-M virt-2.12 \
-smp 2 \
-m 2G \
-bios QEMU_EFI.img \
-device ramfb \
-device ich9-usb-ehci1 \
-device usb-kbd \
-device usb-mouse \
-device usb-tablet \
-device usb-storage,drive=windows \
-drive if=none,id=windows,media=cdrom,file=win10.iso \
-device usb-storage,drive=drivers \
-drive if=none,id=drivers,media=cdrom,file=virtio-win-0.1.173.iso \
-device virtio-blk,drive=system -vnc :1\
-drive if=none,id=system,format=qcow2,file=disk.img

Connect to the simulation via vncviewer ->  vncviewer :1

From my experience I recommend to use the onscreen keyboard, because the frequent emulated keyboard/mouse freezes are quite frustrating

Networking works best with a physical USB 1GB realtek dongle identified by Linux as productid=0x8153. During the installation make sure to load the virtio-win-0.1.173.iso drivers so that the disk.img gets detected by the installer !

Run.sh

#!/bin/bash
export QEMU_AUDIO_DRV=pa

qemu-system-aarch64 \
-cpu host \
-enable-kvm \
-M virt-2.12 \
-smp 2 \
-m 2G \
-bios QEMU_EFI.img \
-device ramfb \
-device ich9-usb-ehci1 \
-device usb-kbd \
-device usb-mouse \
-device usb-tablet \
-device usb-storage,drive=drivers \
-drive if=none,id=drivers,media=cdrom,file=virtio-win-0.1.173.iso \
-device virtio-blk,drive=system \
-drive if=none,id=system,format=qcow2,file=disk.img -vnc :1 \
-device usb-ehci,id=ehci -device usb-host,bus=ehci.0,vendorid=0x0bda,productid=0x8153

And use VNC connector to localhost :1 to finish the setup, please note that the virtio network drivers are causing BSOD on Win10 arm64, so its better to use a real USB ethernet dongle on the Pinebook-pro woth the Realtek chip identified as 0x8153, once you have the network up in the VM, use RemoteDesktop or  freerdp2 to connect to Windows10 over the network for best performance.

Here is a live recording of some Metasploit framerowk meterpreter testing against it through FreeRDP and running on the Pinebookpro

NON aarch64 systems in simulators on the PinebookPro

Here are example loaders and installers for non arm/aarch64 systems which are fun to play around with (and of course you can run this on more powerful x86_64 platform ) I just wanted to show how powerful the Pinebookpro actually is

AmigaOS 4.1  qemu-system-ppc  (search for Sam460InstallCD-53.58.iso on the internet) 

qemu-system-ppc -machine sam460ex -rtc base=localtime -drive if=none,id=cd,file=./Sam460InstallCD-53.58.iso,format=raw -device ide-cd,drive=cd,bus=ide.1 -drive file=./disk.img,format=raw,id=disk -device ide-hd,drive=disk,bus=ide.0 -netdev user,id=mynet0 -net nic,model=rtl8139 -net tap

MorphOS 3.12 qemu-system-ppc

wget http://www.morphos-team.net/morphos-3.12.iso

Install.sh

qemu-system-ppc -machine mac99,via=pmu -m 512 \
-vga none -device sm501 \
-cdrom morphos-3.12.iso -boot d \
-hda disk.img \
-prom-env "boot-device=cd:,\mac_ppc32\boot.img" \
-bios openbios-qemu.elf -serial stdio \
-netdev user,id=mynet0 -device sungem,netdev=mynet0 -net nic -net tap

Run.sh

qemu-system-ppc -machine mac99,via=pmu -m 512 \
-vga none -device sm501 \
-cdrom morphos-3.12.iso -boot c \
-hda disk.img \
-prom-env "boot-device=hd:,boot.img" \
-bios openbios-qemu.elf -serial stdio \
-netdev user,id=mynet0 -device sungem,netdev=mynet0 -net nic -net tap

AIX 7.2 qemu-system-ppc64  (IBM prorietary OS)

You can get the System Diagnostics live CD from https://worthdoingbadly.com/aixqemu/

Or if you have AIX 7.2 you can use it onwards from Technology Level 3 and Service Pack 1 onwards in qemu-system-ppc64 since IBM added virtio-scsi support to the AIX kernel

Install.sh

qemu-system-ppc64 -cpu POWER8 -machine pseries -m 2048 -serial stdio -drive file=disk.img,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom aix.iso -prom-env "boot-command=dev / 0 0 s\" ibm,aix-diagnostics\" property boot cdrom:\ppc\chrp\bootfile.exe -s verbose" -net nic -net tap -display vnc=:1

Run.sh

qemu-system-ppc64 -cpu POWER8 -machine pseries -m 2048 -serial telnet::4441,server -drive file=disk.img,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom AIX71/cd1.iso -prom-env boot-command='boot disk: ' -net nic -net tap -vga none -nographic

Overall performance is not bad, bear in mind that 2 GB RAM is minimum to run AIX 7.2 under qemu-system-ppc64, it takes roughly about 15 minutes to boot on the Pinebook-pro and is quite usable afterwards.

HERCULES – HYPERION

Mainframe z/OS 1.10 (strictly experimental) 

hercules.cnf

ARCHMODE z/Arch
ALRF ENABLE
cckd RA=2,RAQ=4,RAT=2,WR=2,GCINT=5,GCPARM=0,NOSTRESS=0,TRACE=0,FREEPEND=-1
CNSLPORT 3270
CONKPALV (3,1,10)
CPUMODEL 3090
CPUSERIAL 012345
DIAG8CMD ENABLE
ECPSVM YES
LOADPARM 0A95DB..
LPARNAME HERCULES
MAINSIZE 1024
MOUNTED_TAPE_REINIT DISALLOW
NUMCPU 6 
OSTAILOR Z/OS
PANRATE 80
PGMPRDOS LICENSED
SHCMDOPT NODIAG8
SYSEPOCH 1900
TIMERINT 50
TZOFFSET +1400
YROFFSET 0

HERCPRIO 0
TODPRIO -20
DEVPRIO 8
CPUPRIO 0

# Display Terminals

0700 3270
0701 3270
0702 3270
0703 3270
0704 3270

#********************************************************************
# CTCI COMMUNICATION DEVICES *
#********************************************************************
0E20.2 CTCI 192.168.11.101 192.168.11.102


# DASD Devices

0A81 3390 cckd/zares1.cckd sf=shadow/zares1_*
0A82 3390 cckd/zares2.cckd sf=shadow/zares2_*
0A83 3390 cckd/zadb81.cckd sf=shadow/zadb81_*
0A84 3390 cckd/zadb82.cckd sf=shadow/zadb82_*
0A85 3390 cckd/zadb83.cckd sf=shadow/zadb83_*
0A86 3390 cckd/zadb84.cckd sf=shadow/zadb84_*
0A87 3390 cckd/zadb91.cckd sf=shadow/zadb91_*
0A88 3390 cckd/zadb92.cckd sf=shadow/zadb92_*
0A89 3390 cckd/zacic1.cckd sf=shadow/zacic1_*
0A8A 3390 cckd/zadis1.cckd sf=shadow/zadis1_*
0A8B 3390 cckd/zadis2.cckd sf=shadow/zadis2_*
0A8C 3390 cckd/zadis3.cckd sf=shadow/zadis3_*
0A8D 3390 cckd/zadis4.cckd sf=shadow/zadis4_*
0A8E 3390 cckd/zadis5.cckd sf=shadow/zadis5_*
0A8F 3390 cckd/zadis6.cckd sf=shadow/zadis6_*
0A90 3390 cckd/zaims1.cckd sf=shadow/zaims1_*
0A91 3390 cckd/zaprd1.cckd sf=shadow/zaprd1_*
0A92 3390 cckd/zaprd2.cckd sf=shadow/zaprd2_*
0A93 3390 cckd/zaprd3.cckd sf=shadow/zaprd3_*
0A94 3390 cckd/zaprd4.cckd sf=shadow/zaprd4_*
0A95 3390 cckd/zasys1.cckd sf=shadow/zasys1_*
0A96 3390 cckd/zauss1.cckd sf=shadow/zauss1_*
0A97 3390 cckd/zawas1.cckd sf=shadow/zawas1_*
0A98 3390 cckd/zawas2.cckd sf=shadow/zawas2_*
0A99 3390 cckd/zawas3.cckd sf=shadow/zawas3_*
0A9A 3390 cckd/sares1.cckd sf=shadow/sares1_*

Execute Hercules

/usr/local/bin/hercules -f hercules.cnf

Open 2 x3270 terminals and connect them both to localhost so that Hercules logo shows up , below are some basic commands to run the simulator and the system, Im not goint to cover the z/OS here in deeper detail.

IPL the 0E20 device in Hercules by first doing

Press ESC to go to IPL menu in HerculesShift+l   
h 

During first run we need to initialize the system like so 

R 00,I 

Wait till you get either the TSO login or fix the TSO logon by running 
the following in the primary console window 

R 01,RETRY or 
R 02,RETRY

The network is tricky to get configured and the whole Mainframe simulation stuff needs a separate blog entry I believe, hopefully if I have time I will create a specific article about exactly that.

But at the end our Pinebook-pro runs Mainframe quite well under hercules :)

About astr0baby

Please run Adblock or similar... we have been told to do so since Carl Sagan wrote the Contact .
Gallery | This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Pinebookpro virtualization

  1. Pingback: PinePhones start shipping – all you want to know | PINE64

  2. Pingback: Pinebook Pro virtualization | 0ddn1x: tricks with *nix

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.