Move CentOS VM from ESXi to XenServer

Alright, this is a rather sophisticated thing to do, and it took me quite a while to figure out how. But as XenServer 5.5 is free (as opposed to vSphere, which is required to get the same functionality), migration may become an option for many people, who currently pay a yearly subscription fee.
Seriously, many users of the free VMware ESXi might consider migrating, because the (also free) XenServer 5.5 offers a lot more features.

Version numbers in this howto:
Source-Hypervisor: ESXi 4.0
Target-Hypervisor: XenServer 5.5 (XEN 3.3), XenConvert 2.0.1
CentOS guest: 5.3 64bit, default disk layout including LVM

This is the list of things I am going to cover in this howto:

adding a XEN-enabled kernel
exporting the VM from VMware to disk (OVF format)
importing the VM into XenServer
changing boot manager configuration
changing VM settings using the xe tool
amending console settings
installing XenServer Tools (optional, but recommended for performance monitoring, suspending, hot-plug, and live-migration between different servers in a pool)
some trouble-shooting tips at the end of this manual

Pre-Requisites

These things are required for this tutorial:

CentOS guest on ESXi must use LVM! If you don't use LVM, you will need to fiddle with boot devices, root devices and /etc/fstab in the target environment, because ESXi emulates SCSI devices, whereas XEN uses ATA
vCenter Converter
XenConvert
CentOS 5.x CD 1 or the DVD as an ISO file in a location where it can be accessed by XenServer (NFS or Samba share, also called ISO library), just in case something goes wrong and we need to fix it on the target host
a lot of time

I recommend having a shared storage attached to your XenServer pool (iSCSI or NFS, for example with Openfiler or with your own FreeBSD-based iSCSI target), because otherwise you won't be able to migrate VMs between servers while they are running!

Before you start the migration, you may want to make sure that all packages are up to date, running yum update. If a new kernel is to be installed, reboot the machine after updating everything, to ensure everything is still working as expected.

Of course, it's advisable to take a snapshot of the VM to enable yourself to roll back if necessary.

When everything is up to date, you're ready to go...

Installing a XEN kernel

First, let's get a XEN kernel into the system:

yum install kernel-xen
yum install kmod-xfs-xen    # if you are currently using the XFS filesystem

The version should match the one currently running (uname -r), but if not, it doesn't matter. You can make a initial ram-disk (initrd) without running the same version.
As of the time of writing, I've got these two kernels in /boot:

-rw-r--r-- 1 root root 1888956 Jul 14 11:59 vmlinuz-2.6.18-128.2.1.el5
-rw-r--r-- 1 root root 1993616 Jul 14 12:31 vmlinuz-2.6.18-128.2.1.el5xen

The XEN kernel version is important. Copy the kernel version and export it to the environment (easier not to mis-spell or mess with things later on then, especially when you use copy & paste to apply the steps from this how-to):

export kv="2.6.18-128.2.1.el5xen"

Now, we need to get rid off the SCSI drivers in the XEN kernel's initial ram-disk, because they will conflict with the hardware drivers XenServer provides. (You could of course ignore this, but then you won't get xenblk [XEN block device support with full para-virtualization], hence an impact on performance can be expected.)

Let's build a new initial ram-disk for the target kernel:

yum install mkinitrd    
mkinitrd --omit-scsi-modules --with=xennet --with=xenblk --preload=xenblk \
   /boot/initrd-$kv.img $kv

# only if you use the XFS filesystem on important partitions, use this instead:
mkinitrd --omit-scsi-modules --with=xennet --with=xenblk --preload=xenblk \
   --with=xfs --preload=xfs /boot/initrd-$kv.img $kv

Now let's modify /boot/grub/menu.lst. You will find an entry for the previously installed XEN kernel. Other manuals suggest to modify it. I'd recommend to just add a new section before all other sections (less fiddly):

title CentOS-Xen (2.6.18-128.2.1.el5xen-mod)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.18-128.2.1.el5xen ro root=LABEL=/
        initrd /boot/initrd-2.6.18-128.2.1.el5xen.img

A bit further up in the menu.lst file, you will find the default value. Set it to 0 (zero), if you inserted the section on top of all others. Or just count the sections, starting with 0, and fill in the appropriate value. (Not so important anyway, as you can always choose manually during the boot process.)
Note, that I changed the title in menu.lst. Otherwise it's easy to boot the wrong version.

If you have messed around with /etc/fstab and are using non-labelled or non-LVM filesystems, that is giving device names in the /etc/fstab file, it's time to change that now. In the Xen environment, your devices will get new names, as a rule of thumb:

sdaX becomes xvdaX
sdbX becomes xvdbX
...

So sda1 will be xvda1 etc...
The safest way is certainly to comment out unnecessary partitions for the time being.

Alright, now we're done here. Time to transfer the VM over to the XenServer host...

Don't bother to reboot and try out this kernel on ESXi! It will not run in a non-XEN environment!

Copying the ESXi guest to XenServer

After you've shut down the guest instance, fire up VMware vCenter Converter and choose to convert a VMware Infrastructure VM to a Virtual Appliance (Folder format). The interface is quite intuitive and straight-forward. It takes quite a long time though, and the estimated time is always wrong, so don't trust the converter there.

When this step has finished, you can move on to the next step, that is running XenConvert to import the machine.

You must select one of the drives in the XenServer pool to be the default storage! Otherwise XenConvert will fail with rather cryptic error messages ("key not found in dictionary")

In XenConvert select the conversion from Open Virtualisation Package Format (OVF) to XenServer. Then choose your previously saved file, and go through the other few questions you will be asked. Very simple procedure.

Then again, you need a lot of time.

You can dramatically speed up the whole process by using a Windows VM on the same network as both the ESX and the XEN server, preferably with GBit Ethernet. Don't even think of doing that through a DSL or broadband connection from home.

Preparing the VM for the first boot on XenServer

Do not start the VM straight away! There's still some work to do...

Firstly, you need to get SSH access into any XenServer in the target pool of the guest VM. You can also use the XenCenter console, but SSH is nicer, I think.
Anyway, once you are in the shell, you need to tweak some settings related to the VM (I assume it's called centos here):

# 1. get the UUID 
xe vm-list name-label=centos params=uuid

# 2. get rid off the HVM boot node. we want to go PV!
xe vm-param-set uuid=<vm uuid> HVM-boot-policy=

# 3. switch to pygrub (the normal grub cannot read XEN kernel images)
xe vm-param-set uuid=<vm uuid> PV-bootloader=pygrub

# 4. set display arguments
xe vm-param-set uuid=<vm uuid> PV-args="console=hvc0 xencons=hvc"

# 5. discover virtual disk's interface UUID (the VBD, not the VDI)
xe vm-disk-list uuid=<vm uuid>

# 6. set device as bootable
xe vbd-param-set uuid=<vbd uuid> bootable=true

Now you need to restart XenCenter (the client, not any of the servers), because otherwise it might lose console connection during the boot process.

Getting excited? Well, here we go...

Starting the imported guest on XenServer

If your guest had a public IP address on a public NIC assigned, you may want to double-check that the settings in the Network tab match your requirements. Providing that the XenServer is on the same network (VLAN) as the ESXi machine, and that you didn't fire up the old ESXi guest, it's very likely that your guest will be online as-is in a moment.
In case you had a sophisticated network setup in place, you may want to simplify it before booting (or leave it disconnected from any networks) to avoid interference.

Anyway, let's go. Press the magic "Start" button!

Does it boot and present you a login prompt in the console? Well done!
(If not, this is the time where you might need the CentOS CD to boot the VM in rescue mode...)

Once you're logged in, you will not fix networking first thing. Do that later, because with XenServer Tools installed, you can do that while the machine is running!

Installing XenServer Tools

This one is straight forward. Unlike other manuals' suggestions, we are not going to install the whole bunch including kernel, because we of course prefer to use our own kernel, which we can update without relying on Citrix's release cycle.
All we need is the xe-guest-utilities RPM package, which can be found on the xs-tools.iso image. Fortunately, you don't need to do much. Just right-click on your new VM in XenCenter's server view, and choose "Install XenServer Tools". Above the console window it will then show you (white on grey background) where the iso has been assigned to. Usually the device is called /dev/xvdd or /dev/xvdb (xvd = xen virtual device).

So let's mount and check, what's in there:

$ mount /dev/xvdd /mnt
$ cd /mnt/Linux

These are the two RPMs, we're interested in:

-r--r--r-- 1 root root 57720 Jun  6 06:27 xe-guest-utilities-5.5.0-458.i386.rpm
-r--r--r-- 1 root root 57486 Jun  6 06:27 xe-guest-utilities-5.5.0-458.x86_64.rpm

Choose the one, which is right for your platform (nowadays, most likely the latter), and install it:

 
# first, dry run
rpm -Uhv --test xe-guest-utilities-5.5.0-458.x86_64.rpm
# if no problems reported, go ahead:
rpm -Uhv xe-guest-utilities-5.5.0-458.x86_64.rpm

You could also install the encryption utilities contained in the rhel5x directory, but I haven't figured out yet, what they are needed for. If someone knows, I'm happy to update this howto.

But do not install the kernel RPMs.

Now, let's check if the tools are registered properly, and then start the service. Reboot not required.

# check registered services for xe:
$ chkconfig --list | grep xe

# this is what we want to find there:
xe-linux-distribution   0:off   1:off   2:on    3:on    4:on    5:on    6:off

# now let's start the service, without reboot (as we haven't replaced the kernel)
$ service xe-linux-distribution start

DONE! Now you should see the virtualisation status in the VM's general tab, and also the memory usage within the VM. Besides, a "Suspend" button should have appeared for the VM in XenCenter. Furthermore you can live-migrate the VM (given that it doesn't rely on host-internal networks, and all VMs are on NFS or SAN/iSCSI devices).

Troubleshooting

The VM doesn't start

Did you receive a message like [Errno 2] No such file or directory from /usr/bin/pygrub?
Then here's the way to review your bootloader configuration file.
With your VM's UUID, run:

xe-edit-bootloader -u <vm-uuid> -p <partition_number>

The VM's UUID you know from the previous steps, and the partition number you should also know (from the old configuration). If you don't know it, try. It wont break anything.

You need to boot a rescue CD

That's of course possible. Just remember the step from above where we set HVM-boot-policy=
Now we revert to the default, which is "BIOS order":

xe vm-param-set uuid=<vm uuid> HVM-boot-policy="BIOS order"

Remember to set it back, after you are finished with your rescue operations

No login prompt after successful boot

This one took me hours to figure out, especially as it seems to happen only for CentOS migrations, where the current version is 5.3, but originally it was 5.2 or older. Maybe I can save you some time here.

Go to the General tab of your VM and click on the Properties button. Then to Startup Options. In the OS Boot Parameters field, add the word single in the end. This will ensure that you end up in runlevel 1 after the next reboot. (Single user, no login required)

Reboot. When you see the prompt, make a copy of /etc/inittab and edit the original file. Remove the whole block introduced with Run gettys in standard runlevels, and replace it with this line: