Convert Citrix XenServer images to plain Xen

Introduction

I'm currently trying to get rid of Citrix XenServer implementation in favour of plain Xen. I've got a couple of reasons for that, but most importantly it's about control and about avoiding downtime.
Four examples:

• Ever tried to replace a broken management NIC of a server in a pool? What you'd expect is to shut it down, replace the NIC, and then it would figure that out and change all MAC references (or in the worst case you could do that quickly). Not happening. It's a total nightmare and caused a lot of downtime. I ended up wiping the server, re-installing XenServer and restoring all VMs and VLANs manually.
• When you delete snapshots or VMs, you don't get your space back. You actually have to manually reclaim it, which – don't laugh! – can only be done while the affected VM is at least suspended, aka offline, as explained in that linked document. It will cost a bloody awful lot of time, because in a production environment you can't just go ahead without letting people know what you do, even if it's only affecting a single VM out of two and hence wouldn't cause service outage.
• NIC management (and bridges) is all way too static and over-engineered. And although XenServer is using CentOS 5.4 as a basis, it's not using any of its built-in tools and configurations.
• VLAN support: For a long time XenServer has only supported VLAN tagging on the default NIC, not on additional NICs. Since XenServer 5.6 you can't reliably work around that any more. Plus, ebtables (iptables for bridge interfaces) doesn't work on XenServer 5.6 any more. The kernel lacks support, although it used to be present in earlier versions. (Citrix custom kernel, not the Linux kernel itself)

These are four frustrating examples out of many more. Don't get me wrong, XenServer works brilliantly as long as you don't need to do uncommon tasks. Wait, is NIC replacement uncommon really? Never mind
Surely some things will be more difficult using plain Xen, but I'd rather write my own toolset and know what's happening behind the curtains.

Citrix has made some effort to complicate things (and formats), but thanks to this article it wasn't too difficult to make sense of it.

Get me out of here!

So let's get our images off of XenServer back onto the plain Xen environment. Aforementioned URL quotes a xenmigrate.py script, which isn't linked anywhere (or I'm just blind), but with the description of the file format, we can get started using plain bash anyway...

Once you've exported your VM (or its snapshot, or a template) to an .xva file, copy it onto your Xen box. To summarize the .xva format:

• it's a tar file
• it contains a folder
• the folder contains chunks of 1MB each
• they can be concatenated, but blank space needs to be filled

What I'm intending to do is to write a bash script, which does exactly that:

• check which is the biggest chunk number
• create a zero-filled 1MB file (to fill blank space)
• count from 0 to max chunk
• if chunk exists, append it; if not, append blank file
• done

In bash it looks like this:

#!/bin/bash

dd if=/dev/zero of=blank bs=1024 count=1k
test -f image.img && rm -f image.img
touch image.img

max=`ls ???????? | sort | tail -n1`

for i in `seq 0 $max`; do
        fn=`printf "%08d" $i`
        echo -n "$fn of $max"
        if [ -f "$fn" ]; then
                echo " - appending chunk"
                cat $fn >> image.img
        else
                echo " - filling blank"
                cat blank >> image.img
        fi
done

rm -f blank

echo "Done."

It's really simple and not (yet) extracting lvm volumes etc. It's merely to give you an idea of how to easily get your data back to Xen. You can refine yourself, I'm sure.

To use it, simply extract the backup.xva (or whatever you called it), and go into the resulting directory Ref:x. Then you run the script:

tar xf backup.xva
cd Ref:x

/path/to/convert.sh

Get a coffee. This may take a moment. In the end you'll have image.img, ready to be used as a disk in Xen. Your relevant configuration file would look like:

bootloader="/usr/bin/pygrub"
extra = "text console=xvc0"
name = "imported"
memory = "256"
disk = [ 'tap:aio:/root/image.img,xvda,w' ]
vif = [ 'bridge=xenbr0' ]
vcpus=1

Your instinct probably tells you to run something along the lines of mount -o loop ... in order to check the image first.
Not so fast! It contains a disk dump, not an ISO image or a single ext3 partition.

You need to use parted to find the offsets, for example:

# parted image.img 
GNU Parted 1.8.1
Using /root/Ref:6/image.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit                                                             
Unit?  [compact]? B                                                       
(parted) print                                                            

Model:  (file)
Disk /root/Ref:6/image.img: 10737418239B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start       End           Size          Type     File system  Flags
 1      32256B      106928639B    106896384B    primary  ext3         boot 
 2      106928640B  10733990399B  10627061760B  primary               lvm  

So in this case the offset is 32256B. You'd mount it as follows:

mount -o loop,offset=32256B image.img /mnt

Mount it read-only (or better unmount it altogether) before starting a Xen domain from it. Otherwise you're likely to cause mess there.

Conclusion

I'm not taking side against Citrix here at all. For many people/companies it's simply great and straight-forward. However, while we're still running some XenServer clusters and will be running them for a while, I intend to migrate back to plain Xen over time. That's because of all those little nuisances and limitations, which are caused solely by XenServer's toolstack, not actually by Xen.

If you came to read this article for the same reasons, you now know how to get started and can take it from there. Far less complicated than it seemed.
Obviously LVM is preferred over using files. That way you could then create snapshots of running systems etc. I may add that bit later.

Shortcuts