Xen DomU configuration examples - FreeBSD and CentOS

Introduction

This is not a complete tutorial on setting up Xen and its VMs (domains) from scratch. Just some examples, which work. Most of them use the newer pv-grub loader.
It's scarcely documented anywhere how to configure DomU's for FreeBSD, or the documentation is outdated/inaccurate/didn't work for me. What brought me a big step forward was this howto in FreeBSD's forums. And thanks to an AMI image for EC2 by Colin Percival, I was able to de-mystify the magic behind pv-grub/FreeBSD. For CentOS it was then pretty easy and straight-forward (examples below).

To avoid confusion about the versions I'm referring to (that troubled me while searching for other howtos/tutorials), I'm using:

CentOS 5.5 x86_64 Dom0 (irrelevant here; just for completeness)
Linux Kernel 2.6.18-194, not containing the new PVops stuff, which has been introduced in 2.6.31
Xen 3.4.3 from Gitco

The FreeBSD version used in these examples is 8.2-RC2, CentOS is again version 5.5.

FreeBSD i386

FreeBSD/i386 works in PV mode. Configuration is simple and would look like this:

memory = 256

name = "freebsd-gold"
vif = [ 'mac=00:16:3e:2d:82:8f,bridge=xenbr0.1' ]

disk = [ 'phy:/dev/disk2/freebsd-gold-1,hdb,w',
                'phy:/dev/disk2/freebsd-gold-2,hdc,w' ]

vcpus = 1
kernel="/root/freebsd/kernel"
extra = " vfs.root.mountfrom=ufs:ad1s1a,machdep.idle_mwait=0,boot_verbose=1"

The kernel has to be compiled with KERNCONF=XEN

Alternatively, one could use a hda disk (not partitioned; omitted in the above example on purpose), formatted with ext2 and containing:

/boot
/boot/kernel/kernel
/boot/grub/menu.lst

In this case, menu.lst would contain something along the lines of this:

default=0
timeout=3
#hiddenmenu
title FreeBSD kernel i386
    root (hd0)
    kernel /boot/kernel/kernel vfs.root.mountfrom=ufs:ad1s1a,machdep.idle_mwait=0,boot_verbose=1

Here again, the kernel must be compiled with KERNCONF=XEN.

This would then obviously utilise the pv-grub loader, hence the configuration would look slightly different:


memory = 256
name = "freebsd-gold"
vif = [ 'mac=00:16:3e:2d:82:8f,bridge=xenbr0.1' ]

disk = [ 'phy:/dev/disk2/freebsd-pvgrub-kernselector,hda,r', 'phy:/dev/disk2/freebsd-gold-1,hdb,w',
                'phy:/dev/disk2/freebsd-gold-2,hdc,w' ]

vcpus = 1
kernel="/usr/lib/xen/boot/pv-grub-x86_32.gz"
extra = "(hd0)/boot/grub/menu.lst"

The extra parameter tells pv-grub where to find the menu file, and the kernel's extra parameters need to be specified therein as shown earlier.

FreeBSD amd64

FreeBSD/amd64 is currently not supporting full PV mode. It has to be run in HVM but adds at least PV drivers for block devices and network cards, hence performing better than pure HVM. In order for this to work, FreeBSD's kernel needs to be compiled with KERNCONF=XENHVM (note: XENHVM for amd64, XEN for i386).
If running a GENERIC kernel, the system will still be operational, but the drivers lack PV support and are therefore slower.

The HVM Xen configuration file is slightly more complex, incorporating the HVM loader and a device model:

#  -*- mode: python; -*-

import os, re

arch_libdir = 'lib'
arch = os.uname()[4]
if os.uname()[0] == 'Linux' and re.search('64', arch):
    arch_libdir = 'lib64'

kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
device_model = '/usr/' + arch_libdir + '/xen/bin/qemu-dm'

memory = 768
name = "mx2"
vif = [ 'mac=00:16:3e:2d:82:93,bridge=xenbr0.1' ]
disk = [ 'phy:/dev/disk2/mx,hda,w' ]

vcpus = 1

vnc=1
vnclisten="0.0.0.0"
vncpasswd="passwd"
vncdisplay=1

serial='pty'

The normal console won't work. Therefore this configuration also contains settings for the VNC server. In this case it would listen on all interfaces/IPs, and be available on port 5900 + vncdisplay = 5901. There are a few more configuration options around VNC, but I wanted to keep it simple here.

Anyway, once set up and networking configured, SSH will always work just fine

CentOS x86_64

Linux is (obviously) very straight forward here. Assuming that the image has been created with "default partitioning scheme" (from CentOS installer; that is ext3 for /boot and / in VolGroup00/LogVol00), a 64bit configuration would look as simple as:

name = "test"
memory = "256"
disk = [ 'tap:aio:/xen-images/test.img,xvda,w' ]
vif = [ 'mac=00:16:3e:50:c0:c2,bridge=xenbr0' ]
vcpus=1

kernel="/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra = "(hd0,0)/grub/menu.lst"

In this example I'm using a partitioned image file, not a logical volume (LVM) as I did with FreeBSD above. Of course you can and should use it here, too. The utilisation of a file is just an additional example. Mind the tap:aio: prefix, instead of the commonly used file:, which is reportedly a lot slower.

If your partitioning scheme does not contain a dedicated /boot partition, you would use (hd0,0)/boot/grub/menu.lst instead. Or, if you have an unpartitioned image file, you'd replace (hd0,0) with (hd0)

Make sure that your kernel in grub's menu.lst is a xen-aware kernel, and that you add the console parameter there:

title CentOS (2.6.18-194.el5xen)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-194.el5xen ro root=/dev/VolGroup00/LogVol00 console=xvc0
        initrd /initrd-2.6.18-194.el5xen.img

CentOS i386

Here exactly the same as above applies, except that the kernel configuration line differs slightly:

kernel="/usr/lib/xen/boot/pv-grub-x86_32.gz"

That's it. You can take it from here. Happy virtualising

Share