Minimising the virtual machine monitor
Published: 2016-07-02 (last updated: 2017-02-23)
- Update (2016-10-19): all has been merged upstream now!
- Update (2016-10-30):
static_website_tlsworks (TLS,HTTP,network via tap device)!
- Update (2017-02-23): no more extra remotes, Mirage3 is released!
Xen is a hypervisor, providing concrete device drivers for the actual hardware of a physical machine, memory management, scheduling, etc. The initial release of Xen was done in 2003, since then the code size and code complexity of Xen is growing. It also has various different mechanisms for virtualisation, hardware assisted ones or purely software based ones, some where the guest operating system needs to cooperate others where it does not need to cooperate.
Since 2005, Intel CPUs (as well as AMD CPUs) provide hardware assistance for virtualisation (the VT-x extension), since 2008 extended page tables (EPT) are around which allow a guest to safely access the MMU. Those features gave rise to much smaller hypervisors, such as KVM (mainly Linux), bhyve (FreeBSD), xhyve (MacOSX), vmm (OpenBSD), which do not need to emulate the MMU and other things in software. The boot sequence in those hypervisors uses kexec or multiboot, instead of doing all the 16 bit, 32 bit, 64 bit mode changes manually.
MirageOS initially targeted only Xen, in 2015 there was a port to use rumpkernel (a modularised NetBSD), and 2016 solo5 emerged where you can run MirageOS on. Solo5 comes in two shapes, either as
ukvm on top of KVM, or as a multiboot image using
virtio interfaces (block and network, plus a serial console). Solo5 is only ~1000 lines of code (plus dlmalloc), and ISC-licensed.
A recent paper describes the advantages of a tiny virtual machine monitor in detail, namely no more venom like security issues since there is no legacy hardware emulated. Also, each virtual machine monitor can be customised to the unikernel running on top of it: if the unikernel does not need a block device, the monitor shouldn't contain code for it. The idea is to have one customised monitor for each unikernel.
While lots of people seem to like KVM and Linux, I still prefer FreeBSD, their jails, and nowadays bhyve. I finally found some time, thanks to various cleanups to the solo5 code base, to finally look into porting solo5 to FreeBSD/bhyve. It runs and can output to console.
These instructions are still slightly bumpy. If you've a FreeBSD with bhyve (I use FreeBSD-CURRENT), and OCaml and opam (>=1.2.2) installed, it is pretty straightforward to get solo5 running. First, I'd suggest to use a fresh opam switch in case you work on other OCaml projects:
opam switch -A 4.04.0 solo5 (followed by
eval `opam config env` to setup some environment variables).
You need some software from the ports:
opam install mirage mirage-logs solo5-kernel-virtio mirage-bootvar-solo5 mirage-solo5 should provide you with a basic set of libraries.
Now you can get the mirage-skeleton repository, and inside of
mirage configure --no-opam --virtio followed by
make. There should be a resulting
Once that is in place, start your VM:
sudo grub-bhyve -M 128M console > multiboot (host)/home/hannes/mirage-skeleton/console/mir-console.virtio > boot sudo bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -l com1,stdio -m 128M console
The following output will appear on your controlling terminal:
| ___| __| _ \ | _ \ __ \ \__ \ ( | | ( | ) | ____/\___/ _|\___/____/ multiboot: Using memory: 0x100000 - 0x8000000 TSC frequency estimate is 2593803000 Hz Solo5: new bindings STUB: getenv() called hello world hello world hello world hello world solo5_app_main() returned with 0 Kernel done. Goodbye!
Network and TLS stack works as well (tested 30th October).
- I'm not happy to require
ldfrom the ports (but the one in base does not produce sensible binaries with
- Via twitter, bhyve devs suggested to port ukvm to ubhyve. This is a great idea, to no longer depend on virtio, and get more speed. Any takers?
- Debugging via gdb should be doable somehow, bhyve has some support for gdb, but it is unclear to me what I need to do to enter the debugger (busy looping in the VM and a gdb remote to the port opened by bhyve does not work).
I managed to get solo5 to work with bhyve. I even use clang instead of gcc and don't need to link
libgcc.a. :) It is great to see further development in hypervisors and virtual machine monitors. Especially thanks to Martin Lucina for getting things sorted.
Other updates in the MirageOS ecosystem
There were some busy times, several pull requests are still waiting to get merged (e.g. some cosmetics in mirage as preconditions for treemaps and dependency diagrams), I proposed to use
sleep_ns : int64 -> unit io instead of the
sleep : float -> unit io (nobody wants floating point numbers); also an RFC for random, Matt Gray proposed to get rid of
CLOCK (and have a
PCLOCK and a
MCLOCK instead). Soon there will be a major MirageOS release which breaks all the previous unikernels! :)