VirtualBox On FreeBSD

Intro

You probably know the situation: You love FreeBSD, you love ZFS as a filesystem (maybe even as the root filesystem?), but when it comes to virtualisation, FreeBSD users don't have a lot of choice.
That said, VirtualBox, which runs on quite a few platforms and is free, does perform rather well and offers many features, which most people aren't aware of: iSCSI support, Teleporting (aka live migration, even cross-platform), Virtio Net support, Ballooning, built-in solid VNC support (better than Xen's!), highly configurable device support (from disk controller types over chipsets and NICs).

Getting it to run on a FreeBSD 9.0 amd64 server (without any GUI) was less complicated than I thought. However, there are some minor things, which can easily be missed in the relevant chapter of the FreeBSD Handbook and the FreeBSD Wiki, and neither of them actually tells you how to start your first VM, which is anything but intuitive. (That can be automated later, though.)

Let me walk you through the required steps from installation to getting your first VM on the way.

If you'd like to comment, please do it on my related blog post. Thank you!

Prerequisites

I'm assuming that your OS is FreeBSD 9.0, amd64. Your system is up to date, and you do have the full source tree in /usr/src and the ports in /usr/ports.
Only if this is not the case, run:

cd /usr/src
csup -h cvsup.freebsd.org /usr/share/examples/cvsup/standard-supfile
portsnap fetch extract

After that, install VirtualBox:

cd /usr/ports/emulators/virtualbox-ose
make install clean

#only if you're using csh shell:
rehash

On a headless server, which is the focus of this wiki page, you want to disable QT4 and NLS support in the port configuration.
However, if you are walking through this on a desktop and want a GUI, you'll need to keep QT4/NLS support. This requires some extra steps later on, which I'm not going to cover in this document, though, because I haven't got a FreeBSD desktop to verify it with.

The whole compilation will take a while, and you'll be asked to configure and compile a number of dependencies. However, it should run through from start to finish just fine.

Once this is done, load the new kernel modules in this order:

kldload vboxdrv
kldload vboxnetflt
kldload vboxnetadp

We'll add all this to the boot loader and startup configuration files later, when we're sure everything is working fine.
If you have had a play with VirtualBox on FreeBSD before, you'll notice that these manual steps are covered in rc scripts, too. I deliberately choose the manual path here to clarify what's happening in the background.

If everything worked well thus far, we can carry on with the VM creation.

Setting up a first VM

You should have an ISO image of the guest OS you'd like to run. I'm using FreeBSD in this example, too, though the Jail evangelists will start throwing things at me if they read this (and rightly so)

Before we continue, we need to understand that VirtualBox uses a sort of internal registry, which contains references to common network settings, paths and of course the VMs. Therefore, each VM must explicitly be "registered" (storage devices too, though it happens implicitly for them).

This creates a registry entry for a VM called "testMachine":

VBoxManage createvm --name testMachine --ostype FreeBSD_64 --register

There are plenty of different OS types to choose from. You can get a full list with VBoxManage list ostypes. The ostype is essentially choosing a number of default settings, some of which you will want to override in most cases. Let's do that now:

VBoxManage modifyvm testMachine --memory 1024 --ioapic on --cpus 1 \
     --chipset ich9 --nic1 bridged --nictype1 82540EM --bridgeadapter1 em0

In this step we change the Memory to 1GB, switch on IOAPIC (without this, FreeBSD 9 will panic during boot!), assign one virtual CPU, choose the Intel ICH9 chipset, and assign a single bridged interface, which will share the em0 host adapter.
The chosen card is an Intel PRO/1000 Desktop, which worked best for me. Your mileage may vary. You can choose between 5 cards and the virtio driver. (More on virtio later.)

em0 is my Intel card in the host system. Your device may be a different one, for example re0 (Realtek), nfe0 (NVidia), and so on. Run ifconfig to find the name for your interface.
By the way, you can of course use a vlan interface in --bridgeadapter as well, if you like. It works just fine.
And another tip: You probably want to set up a DHCP server somewhere on your network/vlan as it simplifies spinning up VMs greatly.

Next, you'll need a disk of course. Let's use VBox's sparse image files for now, which grow over time. (Logically this isn't the ideal choice for a server setup, because it will eventually scatter fragments of your image all over the disk as the file grows, but right now it's nice and easy. You can use iSCSI, raw devices and other common disk image types, if you like.)

VBoxManage createhd --filename testMachine.vdi --size 10240 

VBoxManage storagectl testMachine --name "SATA Controller" --add sata \
     --controller IntelAhci --sataportcount 4

VBoxManage storageattach "testMachine" --storagectl "SATA Controller" \
     --port 0 --device 0 --type hdd --medium testMachine.vdi

This looks like a lot of effort to add a disk to the VM, and it surely is. However, what you're actually doing here is to create a disk image, create a controller of your choice in the VM, and attach the disk image to it.
VirtualBox allows a lot of flexibility here; the tradeoff is lack of simplicity. You'll probably want to wrap it into a shell script for your use-cases later.

Next up: CDRom to boot and install the guest OS from:

VBoxManage storagectl testMachine --name "IDE Controller" --add ide --controller PIIX4

VBoxManage storageattach testMachine --storagectl "IDE Controller" \
     --port 1 --device 0 --type dvddrive --medium FreeBSD-9.0-RELEASE-amd64-disc1.iso

Here again we add a controller (this time IDE not SATA), attach the CDRom drive to it, and put the FreeBSD installation disk into the virtual tray.

That's all for the preparation.

Let's go – Start the VM

First, we're using the manual approach. FreeBSD's VirtualBox port comes with neater rc scripts, but they'll also background the process, which is not a good idea right now.

So let's start our VM already:

VBoxHeadless --startvm testMachine --vnc --vncport 5001 --vncpass <yourpassword>

Note that we add a VNC server here, which listens on port 5001 and uses <yourpassword> for authentication.
You'll notice the VNC messages right after you start your VM:

20/10/2012 13:15:36 Listening for VNC connections on TCP port 5001
20/10/2012 13:15:36 Listening for VNC connections on TCP6 port 5001

If you see this and can't connect with your preferred VNC viewer, make sure to check your firewall settings.

The console output on the host will give some diagnostic details once you're connected via VNC. All other host diagnostics go into the log file, in this case: VirtualBox VMs/testMachine/Logs/VBox.log. It's interesting that the logs are rotated with every single VM start, which makes it very easy to find the relevant bits for your current (or previous) session.

If everything went well and you're connected via VNC now, you should see FreeBSD booting into the installer just fine. You can install FreeBSD as usual. No pitfalls or caveats here.

When you're finished with the installation and the VM wants to reboot, make sure to hit CTRL-C in the host session (VBoxHeadhles) to interrupt the execution and avoid a reboot (it will otherwise boot from CD again).

Change boot order after setup

Obviously there's no point booting from CDRom for future sessions (unless the VM runs off a LiveCD). Make sure it boots from disk from now on:

VBoxManage modifyvm testMachine --boot1 disk

Now if you start it again as shown above, it should boot from disk into your freshly installed guest:

VBoxHeadless --startvm testMachine --vnc --vncport 5001 --vncpass <yourpassword>

Tip: For future setups you probably want to modify your VMs before installing the guest OS like this:

VBoxManage modifyvm testMachine --boot1 disk --boot2 dvd

This ensures that it will always boot from disk, unless the disk isn't bootable (or is empty), which is the case when you first install the guest OS.

Shutdown

Obviously CTRL-C in the host session is like a powercut for the VM. There's a more elegant way to shut down the VM. For example you can simulate pressing the power button, which will trigger a clean shutdown in FreeBSD:

VBoxManage controlvm testMachine acpipowerbutton

Security

It's obviously best practice to not run things as root, unless you absolutely have to. In case of VirtualBox you do not have to. Here's how to run VirtualBox VMs as any user of your choice...

At this point, you can already run VMs as any user who is member of the group vboxusers. However, you cannot currently use any form of network device in unprivileged VMs. Changing that is simple:

chown root:vboxusers /dev/vboxnetctl
chmod 0660 /dev/vboxnetctl

And to persist these changes, add the following lines to /etc/devfs.conf:

own     vboxnetctl root:vboxusers
perm    vboxnetctl 0660

That's all there is to it. Try it. Create a user, who you add to vboxusers, log in as them, and go through the normal VM creation steps above.
You should be able to do it just fine.

Starting/Stopping properly

You will have a set of VMs that you want to keep running, even after a server restart, and which should therefore be included in your normal startup/shutdown procedure. Luckily the port maintainers have thought of this too, and treat it very similar to how you'd deal with jails. Here an example for this particular VM.

First add this to /boot/loader.conf to load the virtualbox kernel module:

vboxdrv_load="YES"

From /etc/rc.conf:

# this is required for networking (replaces loading remaining modules manually)
vboxnet_enable="YES"

# start VirtualBox 
vboxheadless_enable="YES"

# VMs to start (whitespace-separated list):
vboxheadless_machines="test"

# definition for VM with acronym "test":
vboxheadless_test_name="testMachine" 
vboxheadless_test_user="carsten" 
vboxheadless_test_flags="--vnc --vncport 5001 --vncpass passwd"
vboxheadless_test_stop="acpipowerbutton"

I used "acpipowerbutton" as a shutdown method in this example. The default, if you omit that line, is to save the state and pause the VM. Next time the vboxheadless service starts, it will revive the VM and continue where it left off, which is a lot more graceful procedure, if you are planning on running a number of VMs. To add even more grace (or to define a certain order if VMs depend on one another), you can delay the start of VMs by using vboxheadless_<name>_delay in rc.conf

After you've got these changes in place, you can refer the VM by it's chosen shortname (here "test", which points to "testMachine") when invoking the rc scripts manually to start or stop individual VMs.
For example:

service vboxheadless start test
service vboxheadless stop test

Without a given name, the rc scripts will start or stop all of your defined VMs.

Another useful command is service vboxheadless status which lists all of your registered VMs together with their current status (running, powered off).

Other useful things

Virtio Network Driver in a FreeBSD guest

The Virtio driver performs a lot better than the Intel (em) and AMD PCNet (pcn) drivers, at least within a FreeBSD guest. Actually it's easily twice as fast in my tests.
To enable it, you will need the portstree and kernel sources in the guest, and then compile like this:

cd /usr/ports/emulators/virtio-kmod && make clean install

Once that's done, add these lines to your /boot/loader.conf:

virtio_load="YES"
virtio_pci_load="YES"
if_vtnet_load="YES"
virtio_balloon_load="YES"

Then shutdown your VM and change the network device types for the VM:

VBoxManage modifyvm testMachine --nictype1 virtio

Next time you start your VM, the NIC will be renamed to vtnet[0-8]. Make sure to amend rc.conf accordingly.

You'll see in many places that you should put virtio_blk_load="YES" into your loader.conf as well. This is however not relevant for VirtualBox, because it does not currently support Virtio-Blk controllers. No need to load a module which will never be used.
Should you plan to migrate the VM to Linux KVM (naughty you!), this additional module is useful, though. (Disks will then be renamed to vtbd*). Maybe they'll add it to VirtualBox some day. Might be worth keeping in mind or adding it commented out.

Cloning disks from, say, a Xen Linux VM

Essentially you don't need to do much at all to run Xen VMs in VBox. However, these prerequisites will help:

if you previously booted a Xen-specific kernel in your VM, you may want to switch to a generic one before converting
make sure boot loader menu's timeout is long enough for you to connect via VNC and see it (VirtualBoxes boot fast!)

Remove any references to the hvc0 console from the kernel boot parameters or your VNC window will stay blank after moving on from boot menu

All these changes can be made in /boot/grub/menu.lst before converting the image. Or if the timeout is set long enough, you can change these settings at boot time.

Once you think your VM is ready for conversion, this is how you transform the image:

VBoxManage convertfromraw  /iscsi/test2 test2.vdi --format VDI

You can then attach this disk to a VM's storage controller as described earlier, and chances are that it will boot just fine from it, if you attached it as the first disk device.

This works fine for CentOS VMs, which previously ran on Xen. However, if you see problems mounting root during startup, it's likely because fstab is referencing devices by name (like xvda, xvdb etc) rather than their UUIDs. Change them to sda, sdb etc, because VirtualBox doesn't know anything about Xen-specific devices.

Graphics, shared folders, clipboard etc.

According to the FreeBSD Wiki, there are a number of features, so-called Guest Additions, which are interesting in particular when you run Windows VMs on desktop environments. I haven't tried them, because I haven't got a FreeBSD desktop to play with at the moment.

For servers, they include only one useful feature, that it synchronisation of the guest's time with the host. For me personally that's not worth compiling a load of stuff. NTP will do for now.

However, the above link looks straight-forward enough, if you want to give it a go.

Remote Desktop vs. VNC

To my knowledge, VRDE (aka RDP) for FreeBSD is not supported by Oracle. If this has changed or someone has reverse-engineered it, please point me into the right direction.

However, VNC will usually be enough, and is very easy to configure. (See configuration examples above)
If anybody needs RDP for their Windows guests, it's probably best to use Windows built-in "Remote Desktop Access", if VNC isn't good enough.

Sending CTRL-ALT-DEL

This could occasionally come in handy for a soft reboot without logging into SSH or VNC

VBoxManage controlvm testMachine keyboardputscancode 1d 38 53

Snapshots

The snapshot functionality of VirtualBox goes a bit further than just cloning a disk image. It actually includes the whole configuration of the VM as well, which is very useful if you are trying to find the ideal configuration for your scenario.

However, please keep this in mind:

Whenever you create a snapshot with

VBoxManage snapshot <vm> take <snapname>

you MUST append --pause to the command.
Otherwise the snapshot will fail in most cases. This does not only leave you with a ruined snapshot, but it also freezes the VM.
So, always use this:

VBoxManage snapshot <vm> take <snapname> --pause

I'm not quite sure what causes this, but I can reproduce that both for ZFS and UFS as the host's medium where images are stored on. The error I'd get would be along the lines of:

0%...
Progress state: NS_ERROR_FAILURE
VBoxManage: error: Failed to take snapshot. 
Error message: Failed to save the machine state to 
'/carsten/VirtualBox VMs/centostest/Snapshots/2012-10-21T09-50-04-510753000Z.sav' 
(VERR_SSM_LIVE_GURU_MEDITATION)

I believe --pause ensures a consistent state, which is safer than trying to create a snapshot on the fly anyway. And the actual pause is in the order of milliseconds in my tests. In most cases that will be perfectly acceptable.

(Even if it took a few seconds, it would be fine. You won't be running mission-critical things on just a single VM, would you?)