Introduction

The purpose of this little tutorial is, to give a quick start on installing Xen 3.4.3 in a CentOS 5.5 base environment.
I'm assuming that your system is currently running a clean installation of CentOS 64bit (you haven't tinkered with loads of 3rd party repos or compiled half of the system on your own).

In this tutorial we're going to set up Xen 3.4.3, and get a CentOS guest running via network install (fetching files from a mirror, not PXE or netboot).

It's got potential to break things, in particular networking, so you better do this either while having physical access to the machine or by using KVM-over-IP like Raritan Dominion KX-II (often called "Lara" in Germany) or Dell's DRAC management card.

Installing Xen 3.4.1 and the latest Xen-aware CentOS Linux kernel

We're using gitco's RPMs here, but you could as well build it yourself, if you wanted to (beyond the scope of this article).

At the time of writing, the repository was a bit flaky, which came down to DNS issues at their end (more precisely one NS down, the other one responding occasionally). Add this to your /etc/hosts file, if yum complains about not being able to resolve the hostname: 188.40.79.101 www.gitco.de

Now add the repository to yum and install what we need:

cd /etc/yum.repos.d
wget http://www.gitco.de/repo/GITCO-XEN3.4.3_x86_64.repo
yum install xen kernel-xen vconfig

This will install some 40 packages, some of which certainly aren't required. You can refine that by disabling services (one example: checkconfig avahi-daemon off) or really re-compiling the RPMs yourself and removing unnecessary dependencies. The iSCSI service and GlusterFS are other contained packages, which aren't required, although both are definitely worth a shot.

Now change /boot/grub/menu.lst slightly to ensure the Xen-aware kernel will be booted next time:

default=1 should read default=0

Now on CentOS libvirt creates a NAT bridge, which is good in some cases, but personally I don't do NAT here (routing is done by appropriate devices at the entry point of my network; the Xen boxes are running within local address space). Let's get rid off it:

rm /etc/libvirt/qemu/networks/autostart/default.xml 

After the reboot, which we will have to do anyway, the virtbr0 interface will be gone.

Now go ahead and configure eth0 and eth1 as you usually would.

Then change /etc/xen/xend-config.sxp. Comment out the bridging bit. We'll cover that in a second. This way Xen doesn't touch your interfaces any more.

# (network-script network-bridge)

Time to reboot now.

Server should come up with peth0 and peth1 (the actual physical interfaces) and the two bridges xenbr0 and xenbr1.
Moreover, xend and all that stuff should be running and uname -a will show the xen kernel.

xm list will show Domain-0 running.

Well done.

Networking

A word on networking. The way how Xen works is this: It takes the first NIC (eth0), renames it to peth0, and then creates a bridge called eth0. Puzzled? For the second NIC, or if the NICs are in "wrong" order (eth0 is assumed to be the default outgoing interface), we need to tweak it. Plus, if we want to use VLANs, CentOS will be thoroughly confused by that. Let's just do it ourselves using standard OS tools, giving us full control and making us independent from possible further naming changes in Xen. xenbr0 from previous versions was far more straight-forward and clear than renaming your bridge to something which used to be a name for the actual NICs.

Here's what the configurations in /etc/sysconfig/network-scripts could look like, assuming that you've got eth0 as your default device for outgoing traffic:

*** ifcfg-xenbr0

DEVICE=xenbr0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
IPADDR=10.100.0.102
NETMASK=255.255.255.0
DELAY=0
STP=off

*** ifcfg-eth0

DEVICE=eth0
BRIDGE=xenbr0
HWADDR=00:23:CD:B2:02:D1
ONBOOT=yes
BOOTPROTO=none
HOTPLUG=no

*** ifcfg-xenbr0.4094

DEVICE=xenbr0.4094
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=none
DELAY=0
STP=off

*** ifcfg-eth0.4094

DEVICE=eth0.4094
BRIDGE=xenbr0.4094
VLAN=yes
ONBOOT=yes
BOOTPROTO=none

Restart networking, or reboot, or bring up interfaces manually as you usually would.

In this example the bridge xenbr0 gets the IP addresses assigned, not the NIC eth0.
xenbr0.4094 bridges VLAN 4094 with the NICs of VMs, which are supposed to connect to it, so the bridge itself does not need to have an IP address.

Create your first domain / VM

Now the first VM – or in Xen terminology Domain – can be created. Let's use CentOS. Surely, you could use virt-install, which ships with CentOS/RedHat, and creates VM config files for you by asking you a couple of simple questions. But then again, I'd like to remain in control, and probably you do as well. It's not exactly rocket-science to create the configuration manually.

Before we start, let's get Xen-aware kernels which contain the installer. That's by far the easiest and quickest way to get started:

http://mirror.centos.org/centos/5.5/os/i386/images/xen/ or http://mirror.centos.org/centos/5.5/os/x86_64/images/xen/ (depends on which architecture you prefer for your VM). Download kernel and initrd file.

I've put those into /root/kernel32 and /root/kernel64 respectively. Then you can create a domain configuration file /etc/xen/test:

#bootloader="/usr/bin/pygrub"
kernel = "/root/kernel64/vmlinuz"
ramdisk = "/root/kernel64/initrd.img"
#pae = 1

extra = "text console=xvc0"
name = "test"
memory = "256"
disk = [ 'tap:aio:/var/lib/xen/images/test.img,xvda,w']
vif = [ 'bridge=xenbr0', 'bridge=xenbr1.4094' ]
vcpus=1

This is pretty much the simplest possible setup. Most of the setting explain themselves. Note that the pae line only applies to 32bit kernels. Actually, it may not be required at all, but at least it reminds you that if you use 32bit, your kernel must be a PAE-enabled kernel, even if your VN's allocated memory is far below 4GB. The bootloader line comes into play later.

As for the disk, I've used a file in this example for simplicity. You could as well use LVM partitions (certainly more elegant and allows for hot snapshots). Obviously the file needs to be created as well. 2GB will do:

dd if=/dev/zero of=/var/lib/xen/images/test.img bs=1024 count=2048K

Now start and attach the console with

xm create test -c 

Go through the normal install process. If you're not familiar with net installs, select HTTP and these settings for site/mirror when asked where to install from:

site:      mirror.centos.org
directory: /centos/5.5/os/x86_64

At the very end where it's telling you that it's going to reboot the VM, leave the console (CTRL+]) and shutdown the VM:

xm shutdown test

If you rebooted it straight away, you'd end up with the installer again. We need to change the configuration first:

• remove the kernel and ramdisk lines from /etc/xen/test
• uncomment the pygrub bootloader line
• start the VM again with xm create test

Once you're logged in again, you'd probably want to grab the MAC addresses to assign them statically, like this (for the VM's eth0):

vif = [ 'mac=00:16:3e:3e:10:37,bridge=xenbr0', 'bridge=xenbr1.4094' ]

Just grab the mac addresses from ifconfig's output and paste them here. Otherwise your VM would get a new mac for each virtual NIC assigned with every restart.
The VM's second NIC (eth1) is attached to the VLAN by the way. This is transparent to the VM itself.
The VLAN tag is stripped off the ethernet frame when entering the bridge, which is why each VLAN needs to have its own bridge, obviously. Do not attach multiple VLANs to the same bridge. If you do so, you are compromising security as you're creating a link between two VLANs, which aren't supposed to communicate directly.

Useful Tools

ebtables - Bridge Firewall

You certainly want to have that!

wget http://packages.sw.be/ebtables/ebtables-2.0.8-1.el5.rf.x86_64.rpm
rpm -Uhv ebtables-2.0.8-1.el5.rf.x86_64.rpm

Further Reading

IYou might be interested in Xen DomU configuration examples - FreeBSD and CentOS.

Shortcuts