Enabling KVM in aarch64 Debian 9.6 for accelerated virtualization of CentOS 7.6 aarch64

Centos 7.6 aarch64 howto

Armbian guys are great in what they do with the sunxi64 kernel for the Pinebook. I have been experimenting with the Pinebook for 2 months now and so far I have used the native Ubuntu 18.04 aarch64 distro, NetBSD -current from Jared McNeill (http://www.invisible.ca/arm/) and the Armbian for Pinebook (https://www.armbian.com/pinebook-a64/)

I have so far best experience with the Armbian pinebook-a64 Linux distribution and here I would like to describe the process how to get full KVM support in the Armbian Bionic based on the 4.19.x kernel

Unfortunately the current 4.19.2-sunxi64 kernel does not have KVM virtualisation support enabled so any attempt on running aarch64 virtual machines in QEMU on the Pinebook will be horribly slow.

Installation of the Armbian Stretch 4.19.x distribution is straight forward, and I have used the image that I have transferred to a microSD card  64BG (100 mb/s as minimum speed). All this can be easily done on the current Linux machine via following command

x86_64 LAPTOP

$ wget https://dl.armbian.com/pinebook-a64/Debian_stretch_next.7z
$ su  
# dd if=Armbian_5.69_Pinebook-a64_Debian_stretch_next_4.19.13.img of=/dev/sd(X) <- check what your USB microSD adaper is

Once the image is transferred, please follow the official documentation how to setup Networking and all other things, which I’m not going to cover here. It is straight forward.

So once we have the Pinebook booted and connected to the internet, make sure you get all packages updated

aarch64 PINEBOOK

# apt-get update
# apt-get upgrade

we can go on and install latest Qemu, so for this we need to enable the testing repository in /etc/apt/sources.list

aarch64 PINEBOOK

root@pinebook:/etc/apt# cat sources.list
deb http://httpredir.debian.org/debian  testing main contrib non-free
#deb-src http://httpredir.debian.org/debian stretch main contrib non-free

deb http://httpredir.debian.org/debian stretch-updates   main contrib non-free
#deb-src http://httpredir.debian.org/debian stretch-updates main contrib non-free

deb http://httpredir.debian.org/debian stretch-backports  main contrib non-free
#deb-src http://httpredir.debian.org/debian stretch-backports main contrib non-free

deb http://security.debian.org/ stretch/updates testing main contrib non-free
#deb-src http://security.debian.org/ stretch/updates main contrib non-free

deb http://httpredir.debian.org/debian jessie main contrib non-free

So next we can install qemu-system which will pull all softmmu suport (x86,sparc,ppc,mips,arm) but for accelerated qemu emulation we obviously need only the qemu-system-arm (this includes both qemu-system-arm and qemu-system-aarch64)   We will also need to install qemu-utils and qemu-efi-aarch64

aarch64 PINEBOOK

# apt-get install qemu-system qemu-efi-aarch64

So now we are ready to build our customized Armbian Stretch kernel 4.19.x based on mainline, since the stock one does not include KVM support.

So for this I have followed this guide https://docs.armbian.com/Developer-Guide_Build-Preparation/

What we will need is the following:

I have used the following config on my Linux Mint 19.1 – make sure you create a large enough VDI disk to hold all the build files, minimum is 20 GB, 50 GB is recommended.

Also Im using the following host-only-adapter network setup so that I can access the VM from my host and vice-versa easily via ssh/scp

For this to work I have a following network configuration script I run after I start VirtualBox manager (to get the vboxnet0 interface registered first)

x86_64 LAPTOP

root@panasonic:/home/user# cat networking.sh 
#Setup tap and bridge 
tunctl -t tap0 -u user
ifconfig tap0 up
brctl addbr br0
brctl addif br0 vboxnet0 
brctl setfd br0 0
ifconfig br0 netmask broadcast up
brctl addif br0 tap0 vboxnet0 
ifconfig tap0
sysctl net.ipv4.ip_forward=1
iptables --table nat -A POSTROUTING --out-interface wlan0 -j MASQUERADE

Internal IP address for the Guest VMs are always static in the following format

DNS:  or

Setup the OpenSSH server and configure the network on the Guest Ubuntu VM accordingly.

So once we have everything ready we can  start preparing the Guest Ubuntu VM  for the kernel build

Login, elevate to root and run

VirtualBox Ubuntu GUEST

# apt-get -y -qq install git
# git clone --depth 1 https://github.com/armbian/build
# cd build

And run the compile.sh script

VirtualBox Ubuntu GUEST

# ./compile.sh

Select to build U-boot and  Kernel packages

Choose to show Kernel options so that we can enable the KVM support

Choose Pinebook-a64

After a while you should come to the Kernel menuconfig

Make sure to set the following Virtualisation option on (space bar)

And the following subsection should be enabled as built-on and modules

After some time the kenrel packages would be made available in the following directory which we will scp out of the VirtualBox VM and transfer to the Pinebook via scp

Back on the Pinebook we need to install these packages via apt

aarch64 PINEBOOK 

-rw-r--r-- 1 user user 44152 Jan 11 23:36 linux-dtb-next-sunxi64_5.70_arm64.deb
-rw-r--r-- 1 user user 10422904 Jan 11 23:35 linux-headers-next-sunxi64_5.70_arm64.deb
-rw-r--r-- 1 user user 16571572 Jan 11 23:35 linux-image-next-sunxi64_5.70_arm64.deb
-rw-r--r-- 1 user user 320580692 Jan 11 23:36 linux-source-next-sunxi64_5.70_all.deb
-rw-r--r-- 1 user user 246032 Jan 11 23:35 linux-u-boot-next-pinebook-a64_5.70_arm64.deb

# dpkg -i linux-dtb-next-sunxi64_5.70_arm64.deb
# dpkg -i linux-headers-next-sunxi64_5.70_arm64.deb
# dpkg -i linux-image-next-sunxi64_5.70_arm64.deb

Reboot the Pinebook to load the new KVM enabled Kernel, once up we should verify we are running the new Kernel

aarch64 PINEBOOK

root@pinebook:/home/user# uname -a
Linux pinebook 4.19.13-sunxi64 #5.70 SMP Sat Jan 12 00:10:12 CET 2019 aarch64 GNU/Linux

Next we prepare the virtualisation environment where we want to run Centos 7.6

Before we actually load QEMU on the Pinebook we need to modify slightly the stock /etc/qemu-ifup script as follows

#! /bin/sh
# Script to bring a network (tap) device for qemu up.
# The idea is to add the tap device to the same bridge
# as we have default routing to.

# in order to be able to find brctl
ip=$(which ip)

if [ -n "$ip" ]; then
ip link set "$1" up
brctl=$(which brctl)
if [ ! "$ip" -o ! "$brctl" ]; then
echo "W: $0: not doing any bridge processing: neither ip nor brctl utility not found" >&2
exit 0
ifconfig "$1" up

switch=$(ip route ls |
awk '/^default / {
for(i=0;i<NF;i++) { if ($i == "dev") { print $(i+1); next; } }


# only add the interface to default-route bridge if we
# have such interface (with default route) and if that
# interface is actually a bridge.
# It is possible to have several default routes too
for br in $switch; do
if [ -d /sys/class/net/$br/bridge/. ]; then
if [ -n "$ip" ]; then
ip link set "$1" master "$br"
brctl addif $br "$1"
exit # exit with status of the previous command

echo "W: $0: no bridge for guest interface found" >&2

And prepare the following “network” helper script to enable internet connection sharing bethween Host and Quest on the Pinebook

You will however need to install some tools first for this to work

aarch64 PINEBOOK 
# apt-get install uml-utilities
# apt-get install bridge-utils

And then finally save below shell script, make it executable and run it

#Setup tap and bridge 
tunctl -t tap0 -u user
ifconfig tap0 up
brctl addbr br0
brctl setfd br0 0
ifconfig br0 netmask broadcast up
brctl addif br0 tap0 
ifconfig tap0
sysctl net.ipv4.ip_forward=1
iptables --table nat -A POSTROUTING --out-interface wlan0 -j MASQUERADE

Load the above script each time you boot the Pinebook if you wish to run Qemu.

On the Pinebook prepare a directory to hold some files, we will be using the linaro UEFI image

aarch64 PINEBOOK 

$ mkdir -p ~/KVM/centos
$ cd ~/KVM/centos
$ wget http://mirror.vpsnet.com/centos-altarch/7.6.1810/isos/aarch64/CentOS-7-aarch64-Minimal-1810.iso
$ mv CentOS-7-aarch64-Minimal-1810.iso centos.iso
$ wget http://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/latest/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.img.gz
$ gunzip QEMU_EFI.img.gz 
$ qemu-img create -f qcow2 centos.img 50G 
$ qemu-img create -f qcow2 varstore.img 64M

And next we prepare the qemu loader shell script for this simulation in the KVM/centos working directory and make it executable

/usr/bin/qemu-system-aarch64 \
-cpu host -M virt,accel=kvm -m 1024 -nographic \
-drive if=pflash,format=raw,file=QEMU_EFI.img \
-drive if=pflash,file=varstore.img \
-drive if=virtio,format=raw,file=disk.img \
-drive if=virtio,format=raw,file=centos.iso \
-net nic -net tap

This is it, and you should be able to fully emulate with almost native speed CentOS 7.6 aarch64 on the Debian 9.6 aarch64 Pinebook

Configuration of CentOS is pretty standard.

FreeBSD 13 aarch64 howto

The above method also works for FreeBSD 13 aarch64  ( http://ftp.freebsd.org/pub/FreeBSD/snapshots/arm64/aarch64/ISO-IMAGES/13.0/FreeBSD-13.0-CURRENT-arm64-aarch64-PINEBOOK-20190110-r342911.img.xz ) except the qemu loader is slightly different

/usr/bin/qemu-system-aarch64 \
-cpu host -M virt,accel=kvm -m 1024 -nographic \
-drive if=pflash,format=raw,file=QEMU_EFI.img \
-drive if=pflash,file=varstore.img \
-drive if=virtio,file=freebsd.img \
-net nic -net tap

Where the freebsd.img is the downloaded img file from the above URL

During initial boot, you need to break the DHCP load by CTRL+C  as well as during the default route config also CTRL+C  we will set these up later

Login as root:root

P.S to save up some space on the default image  rm -rf /usr/lib/debug  

Network can be configured easily as follows

aarch64 FreeBSD

# ifconfig vtnet0 up 
# route add default
# echo "nameserver" > /etc/resolv.conf

To make this permanent add the following to /etc/rc.conf

ifconfig_vtnet0="inet netmask"

To use secondary disk vtblk1 use this loader script (and create disk2.img raw image first)

aarch64 PINEBOOK

# qemu-disk create -f raw disk2.img 2G   (For 2 Gigabytes)
/usr/bin/qemu-system-aarch64 \
-cpu host -M virt,accel=kvm -m 1024 -nographic \
-drive if=pflash,format=raw,file=QEMU_EFI.img \
-drive if=pflash,file=varstore.img \
-drive if=virtio,bus=0,unit=0,format=raw,file=freebsd.img \
-drive if=virtio,bus=0,unit=1,format=raw,file=disk2.img \
-net nic -net tap

On emulated FreeBSD it is then simple to add this new virtual disk as follows:

aarch64 FreeBSD 

# gpart create -s GPT vtbd1 
# gpart add 0t freebsd-ufs -a 1M vtbd1 
# newfs -U /dev/vtbd1p1 
# mkdir /disk
# mount /dev/vtbd1p1  /disk

Here is a dmesg from FreeBSD 13 aarch64 running under KVM qemu-system-aarch64 on Debian 9.6 aarch64 Pinebook

root@pinebook:/home/user/KVM/FreeBSD# ssh
Password for root@freebsd:
Last login: Sun Jan 13 22:56:36 2019
FreeBSD 13.0-CURRENT r342911 GENERIC

Welcome to FreeBSD!

root@freebsd:~ # dmesg
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-CURRENT r342911 GENERIC arm64
FreeBSD clang version 7.0.1 (tags/RELEASE_701/final 349250) (based on LLVM 7.0.1)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
KLD file umodem.ko is missing dependencies
FreeBSD/SMP: Multiprocessor System Detected: 1 CPUs
random: unblocking device.
random: entropy device external interface
MAP 785d0000 mode 2 pages 64
MAP 78610000 mode 2 pages 160
MAP 786b0000 mode 2 pages 160
MAP 7bc20000 mode 2 pages 400
MAP 7bdb0000 mode 2 pages 592
MAP 4000000 mode 0 pages 16384
MAP 9010000 mode 0 pages 1
kbd0 at kbdmux0
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
psci0: <ARM Power State Co-ordination Interface Driver> on acpi0
gic0: <ARM Generic Interrupt Controller> iomem 0x8000000-0x8000fff,0x8010000-0x8010fff on acpi0
gic0: pn 0x2, arch 0x2, rev 0x1, implementer 0x43b irqs 288
gic0: frame: 0 8020000 1 64 80
gicv2m0: <ARM Generic Interrupt Controller MSI/MSIX> mem 0x8020000-0x8020fff on gic0
generic_timer0: <ARM Generic Timer> irq 34,35,36 on acpi0
Timecounter "ARM MPCore Timecounter" frequency 24000000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 24000000 Hz quality 1000
cpu0: <ACPI CPU> on acpi0
uart0: <PrimeCell UART (PL011)> iomem 0x9000000-0x9000fff irq 0 on acpi0
uart0: console (9600,n,8,1)
virtio_mmio0: <VirtIO MMIO adapter> iomem 0xa000000-0xa0001ff irq 1 on acpi0
virtio_mmio1: <VirtIO MMIO adapter> iomem 0xa000200-0xa0003ff irq 2 on acpi0
virtio_mmio2: <VirtIO MMIO adapter> iomem 0xa000400-0xa0005ff irq 3 on acpi0
virtio_mmio3: <VirtIO MMIO adapter> iomem 0xa000600-0xa0007ff irq 4 on acpi0
virtio_mmio4: <VirtIO MMIO adapter> iomem 0xa000800-0xa0009ff irq 5 on acpi0
virtio_mmio5: <VirtIO MMIO adapter> iomem 0xa000a00-0xa000bff irq 6 on acpi0
virtio_mmio6: <VirtIO MMIO adapter> iomem 0xa000c00-0xa000dff irq 7 on acpi0
virtio_mmio7: <VirtIO MMIO adapter> iomem 0xa000e00-0xa000fff irq 8 on acpi0
virtio_mmio8: <VirtIO MMIO adapter> iomem 0xa001000-0xa0011ff irq 9 on acpi0
virtio_mmio9: <VirtIO MMIO adapter> iomem 0xa001200-0xa0013ff irq 10 on acpi0
virtio_mmio10: <VirtIO MMIO adapter> iomem 0xa001400-0xa0015ff irq 11 on acpi0
virtio_mmio11: <VirtIO MMIO adapter> iomem 0xa001600-0xa0017ff irq 12 on acpi0
virtio_mmio12: <VirtIO MMIO adapter> iomem 0xa001800-0xa0019ff irq 13 on acpi0
virtio_mmio13: <VirtIO MMIO adapter> iomem 0xa001a00-0xa001bff irq 14 on acpi0
virtio_mmio14: <VirtIO MMIO adapter> iomem 0xa001c00-0xa001dff irq 15 on acpi0
virtio_mmio15: <VirtIO MMIO adapter> iomem 0xa001e00-0xa001fff irq 16 on acpi0
virtio_mmio16: <VirtIO MMIO adapter> iomem 0xa002000-0xa0021ff irq 17 on acpi0
virtio_mmio17: <VirtIO MMIO adapter> iomem 0xa002200-0xa0023ff irq 18 on acpi0
virtio_mmio18: <VirtIO MMIO adapter> iomem 0xa002400-0xa0025ff irq 19 on acpi0
virtio_mmio19: <VirtIO MMIO adapter> iomem 0xa002600-0xa0027ff irq 20 on acpi0
virtio_mmio20: <VirtIO MMIO adapter> iomem 0xa002800-0xa0029ff irq 21 on acpi0
virtio_mmio21: <VirtIO MMIO adapter> iomem 0xa002a00-0xa002bff irq 22 on acpi0
virtio_mmio22: <VirtIO MMIO adapter> iomem 0xa002c00-0xa002dff irq 23 on acpi0
virtio_mmio23: <VirtIO MMIO adapter> iomem 0xa002e00-0xa002fff irq 24 on acpi0
virtio_mmio24: <VirtIO MMIO adapter> iomem 0xa003000-0xa0031ff irq 25 on acpi0
virtio_mmio25: <VirtIO MMIO adapter> iomem 0xa003200-0xa0033ff irq 26 on acpi0
virtio_mmio26: <VirtIO MMIO adapter> iomem 0xa003400-0xa0035ff irq 27 on acpi0
virtio_mmio27: <VirtIO MMIO adapter> iomem 0xa003600-0xa0037ff irq 28 on acpi0
virtio_mmio28: <VirtIO MMIO adapter> iomem 0xa003800-0xa0039ff irq 29 on acpi0
virtio_mmio29: <VirtIO MMIO adapter> iomem 0xa003a00-0xa003bff irq 30 on acpi0
virtio_mmio30: <VirtIO MMIO adapter> iomem 0xa003c00-0xa003dff irq 31 on acpi0
virtio_mmio31: <VirtIO MMIO adapter> iomem 0xa003e00-0xa003fff irq 32 on acpi0
pcib0: <Generic PCI host controller> on acpi0
pci0: <PCI bus> on pcib0
pcib0: pci_host_generic_core_alloc_resource FAIL: type=4, rid=16, start=0000000000000080, end=000000000000009f, count=0000000000000020, flags=1400
pcib0: pci_host_generic_core_alloc_resource FAIL: type=4, rid=16, start=0000000000000040, end=000000000000007f, count=0000000000000040, flags=1800
virtio_pci0: <VirtIO PCI Network adapter> mem 0x10042000-0x10042fff,0x8000000000-0x8000003fff at device 1.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 52:54:00:12:34:56
virtio_pci1: <VirtIO PCI Block adapter> mem 0x10041000-0x10041fff,0x8000004000-0x8000007fff at device 2.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci1
vtblk0: 2560MB (5242880 512 byte sectors)
virtio_pci2: <VirtIO PCI Block adapter> mem 0x10040000-0x10040fff,0x8000008000-0x800000bfff at device 3.0 on pci0
vtblk1: <VirtIO Block Adapter> on virtio_pci2
vtblk1: 0MB (385 512 byte sectors)
acpi_button0: <Power Button> on acpi0
acpi0: Could not update all GPEs: AE_NOT_CONFIGURED
cryptosoft0: <software crypto>
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass
CPU 0: ARM Cortex-A53 r0p4 affinity: 0
Instruction Set Attributes 0 = <AES+PMULL,SHA1,SHA2,CRC32>
Instruction Set Attributes 1 = <>
Processor Features 0 = <AdvSIMD,Float,EL3 32,EL2 32,EL1 32,EL0 32>
Processor Features 1 = <0>
Memory Model Features 0 = <4k Granule,64k Granule,MixedEndian,S/NS Mem,16bit ASID,1TB PA>
Memory Model Features 1 = <>
Memory Model Features 2 = <32b CCIDX,48b VA>
Debug Features 0 = <2 CTX Breakpoints,4 Watchpoints,6 Breakpoints,PMUv3,Debug v8>
Debug Features 1 = <0>
Auxiliary Features 0 = <0>
Auxiliary Features 1 = <0>
Trying to mount root from ufs:/dev/ufs/rootfs [rw]...
WARNING: WITNESS option enabled, expect reduced performance.
lo0: link state changed to UP
vtnet0: link state changed to UP



About astr0baby

Please run Adblock or similar... we have been told to do so since Carl Sagan wrote the Contact .
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.