root@host:~# cat /etc/network/interfaces 
### Hetzner Online AG - installimage
# Loopback device:
auto lo
iface lo inet loopback
 
# device: eth0
auto eth0
iface eth0 inet static
  address 213.x.x.6
  broadcast 213.x.x.31
  netmask 255.255.255.224
  gateway 213.x.x.1
 
# default route to access subnet
up route add -net 213.x.x.0 netmask 255.255.255.224 gw 213.x.x.1 eth0
 
auto eth0:1
iface eth0:1 inet static
  address 78.x.x.1
  broadcast 78.x.x.7
  netmask 255.255.255.248

The plan is that the guests will get 78.x.x.2, 78.x.x.3, etc. and will use 78.x.x.1 as gateway (you'll see this later) - that's why I've assigned it to the host.

Well, after you have found out that Xen kernel is loaded with

uname -r
xm list

you'll be ready to proceed.

Now we want to make a domU with Ubuntu Server 8.04 with the program debootstrap. First identify the partitions you want to use. For the root filesystem I'll use /dev/lvmstore/lv1 and for swap I'll use /dev/sda6. To prepare the partitions, please

mkfs.xfs /dev/lvmstore/lv1
mkswap /dev/sda6

Now it's time to install the guest system. First mount the root file system like this:

mount /dev/lvmstore/lv1 /mnt

Now just install it with this simple command (depending on your environment you might have to change the --arch-parameter):

debootstrap --arch amd64 hardy /mnt http://archive.ubuntu.com/ubuntu

Now you should copy a couple of files:

cp /etc/resolv.conf /mnt/etc/resolv.conf
cp /etc/apt/sources.list /mnt/etc/apt/sources.list
mkdir /mnt/lib/modules/`uname -r`
cp -R /lib/modules/`uname -r`/* /mnt/lib/modules/`uname -r`/

Now you should change a couple of files. Now

vi /mnt/etc/network/interfaces

should have the content

root@host:~# cat /mnt/etc/network/interfaces 
auto lo
iface lo inet loopback
 
auto eth0
iface eth0 inet static
  address 78.x.x.2
  broadcast 78.x.x.7
  netmask 255.255.255.248
  gateway 78.x.x.1

Here you see that I'm using the host as gateway.

And

vi /mnt/etc/hostname

to change the hostname of the guest.

We also need to change the fstab - mine looks like this (remember to change xfs to ext3 or similar if you're not using xfs):

root@host:~# cat /mnt/etc/fstab 
/dev/sda1 /    xfs  defaults 1 2
/dev/sda2 none swap sw       0 0

The names /dev/sda1 and /dev/sda2 is assigned on a Xen configuration file we'll get to that i a minute.

Now we prepare the network bridge at the host by:

vi /etc/sysctl.conf

and change

#net.ipv4.ip_forward=1

to

net.ipv4.ip_forward=1

We also start the network bridge:

/etc/xen/scripts/network-bridge start

Now reboot the host to enable the changes in /etc/sysctl.conf. After reboot please check that the bridge is okay:

brctl show

Now we create the file /etc/xen/domU1.conf with this content:

root@host:~# cat /etc/xen/domu1.cfg 
kernel      = '/boot/vmlinuz-2.6.24-19-xen'
ramdisk     = '/boot/initrd.img-2.6.24-19-xen'
memory      = '512'
maxmem      = '1536'
 
root        = '/dev/sda1 ro'
disk        = ['phy:/dev/lvmstore/lv1,sda1,w','phy:/dev/sda6,sda2,w']
 
name        = 'domU1'
 
vif	     = ['bridge=eth0']
 
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
 
vcpus	     = '2'
extra       = 'xencons=tty1'

This is the specs of the guest. This is a configuration making you able to ballon the amount of memory to 1,5 GB instead of the initial 512 MB. Please view man xmdomain.cfg for further details.

Now start the guest (the -c means that the console is attached to the guest immediately):

xm create /etc/xen/domu1.cfg -c

When the environment is started, login with the same password as on your host. Now check the network is okay, change the password, and update the system, respectively:

ping 78.x.x.1
passwd
apt-get update
apt-get upgrade
apt-get dist-upgrade

You'll probably see some locales errors. For my case (da_DK) it's fixed by this:

locale-gen da_DK.UTF8
tzselect
echo "export LANG=C" >> ~/.bashrc

Please refer to https://wiki.ubuntu.com/DebootstrapChroot for details regarding this and debootstrap in general.

To shut it down, use

init 0

You can start the guest without the -c-parameter to start it without the console. Then you can use

xm list

to check that it's running.

At this point I'd strongly recommend to shut the guest down, reboot the host (dom0) and start everything again. This is to verify that everything is also working after a reboot (before using it in production).

I followed http://www.howtoforge.com/high-performance-xen-on-ubuntu-8.04-amd64 when I installed the guest, but I've done it slightly different and avoided some of the errors.

Disk-layout considerations

In order to use Windows with Xen you also need to have Intel VT or AMD-V - and have it enabled, too. My "metal" is a AMD-V enabled CPU, 4 GB RAM, 2 x 400 GB SATA II disks, and a 1 GBit NIC (network interface card) - a DS5000 at Hetzner. I've chosen to have a rather complex disk setup based on a lot of reading different places at the web (and for fun, too), e.g. [1], [2], [3], and [4]. First of all I want protection (raid 1 - mirroring) because I'm going to use the system in a production environment, but I don't want to pay the money for a real hardware RAID controller, hence I've decided to use software raid, i.e. mdadm. I know it's expensive in CPU-time, but I'm willing to "pay" for the protection this way. I want LVM for the domUs because it among other things enables me to take snapshots and easily grow/shrink the size of the disk.

The only thing I don't want mirrored, is the swap-partitions. I know it might cause the system to crash if a disk fails, but only data in memory can be lost this way. If a swap-partition crashes it's likely because the disk is dead, and as a back-up I want extra swap-partitions on the other disk so that I can use those until a new disk is installed. I don't want the swap-partitions to be LVM - only normal partitions that can be mapped directly to domUs through Xen. Because of my metal, it's not likely that I'm going to have more than 5 domUs running, hence I need 12 swap-partitions in total (the 5 plus 1 for dom0 and doubleing it to encounter the extra ones if a disk crashes). Because I don't know if the memory of dom0 or a domU is going to be ballooned (resized - and yes, among many other things Xen i capable of that), I've decided to go with a size of 4GB of all the swap-partitions. It may be a little oversized, but better safe than sorry and I have plenty of disk space for my needs.

Of course the root file system of dom0 needs to be mirrored, but I don't want to use LVM with this, because I'm not going to use any of the advantages with dom0. So this is going to be an ordinary raid 1 xfs partition. The rest of the disk space is used to make a VG (volume group) in LVM, and this pool can then be used to make LV (logical volumes) to disks for the domUs.

At this point please be aware that I'm going to map a LV to a domU to its root file system and a "normal" partition to its swap. Xen gives other possibilities, e.g. map a file on dom0 to the root file system of a domU or simply use a "normal" partition. All three choices have advantages and disadvantages; I've made my mind up and chosen to use the LVM-approach. I've also chosen to use xfs instead of ext3, but this choice really makes no difference in this tutorial: if you like ext3 better, then please use it.

Disk layout

If the two psysical 400GB disks are called /dev/sda and /dev/sdb, the disk-layout can be depicted as follows:

disk	partition	size	usage/mount		pri/log
---------------------------------------------------------------
/dev/sda
	/dev/sda1	  1GB	/dev/md0 (bootable)	primary
	/dev/sda2	 20GB	/dev/md1 (dom0 -> /)	primary
	/dev/sda5	  4GB	swap dom0		logical
	/dev/sda6	  4GB	dnu* (swap domU1)	logical
	/dev/sda7	  4GB	dnu* (swap domU2)	logical
	/dev/sda8	  4GB	dnu* (extra swap domU3)	logical
	/dev/sda9	  4GB	dnu* (extra swap domU4)	logical
	/dev/sda10	  4GB	dnu* (extra swap domU5)	logical
	/dev/sda11	 rest	/dev/md2 (LVM)		logical

/dev/sdb
	/dev/sdb1	  1GB	/dev/md0 (bootable)	primary
	/dev/sdb2	 20GB	/dev/md1 (dom0 -> /)	primary
	/dev/sdb5	  4GB	dnu* (extra swap dom0)	logical
	/dev/sdb6	  4GB	dnu* (extra swap domU1)	logical
	/dev/sdb7	  4GB	dnu* (extra swap domU2)	logical
	/dev/sdb8	  4GB	dnu* (swap domU3)	logical
	/dev/sdb9	  4GB	dnu* (swap domU4)	logical
	/dev/sdb10	  4GB	dnu* (swap domU5)	logical
	/dev/sdb11	 rest	/dev/md2 (LVM)		logical

*dnu = DO NOT USE (is going to be mounted by the domUs)

/dev/md0 (bootable)
	/boot		  1GB	raid 1 (dom0 -> /boot)

/dev/md1
	/		 20GB	raid 1 (dom0 -> /)

/dev/md2
	/dev/lvmstore/lv1 15GB	/ (dom1)
	/dev/lvmstore/lv2 15GB	/ (dom2)
	/dev/lvmstore/lv3 15GB	/ (dom3)
	/dev/lvmstore/lv4 20GB	/ (dom4)
	...		 ... 	...

Installation

The task of making this configuration in Ubuntu can be a bit confusing. How to make this configuration depends on the way you're installing Ubuntu. I've had quite some troubles with LILO, so I recommend that you use GRUB instead! That's also what [6] recommends.

As earlier mentioned, my server is hosted at Hetzner. Here you can only manipulate the installation through a configuration file. This means that the disk configuration has to be performed a bit different. You could of course also make this disk layout with the ordinary Ubuntu installer, but I'll not get into details how to do that in this tutorial. First reboot into rescue mode and execute

installimage

Now choose to edit the configuration file. First you need to enable software raid (thanks, [5]) and choose GRUB:

SWRAID 1
BOOTLOADER grub

Furthermore I configured the hostname. I made the initial partitioning scheme like this:

PART 1 /boot ext3  1G
PART 2 /     xfs  20G
PART 3 swap  swap  4G

And then I installed the system. After this was done, the disk layout needed to be modified heavily to get what we wanted.

Because we are going to modify the harddisk layout on a running system, be aware that you might delete data! Please make sure you have a backup of all important data. Also do not do this if you're not able to restore the server somehow (boot it to rescue mode or similar if something should fail)!

First launch

cfdisk /dev/sda

And perform these steps:

• Select sda3 and delete it
• Create a new locigal partition of 4301.83 MB at the beginning and change type to 82 (Linux swap)
• Create five new locigal partition of 4301.83 MB at the beginning and don't change the type
• Create a partition on the rest of space of type FD (Linux raid autodetect) - it's going to be used for a mirrored LVM
• Write (don't mind about the warning "No primary partitions are marked as bootable" - cfdisk doesn't know the /dev/md0 is bootable)
• Quit

Now do almost the same with /dev/sdb by

cfdisk /dev/sdb

And perform these steps:

• Select sdb3 and delete it
• Create six new locigal partition of 4301.83 MB at the beginning and don't change the type
• Create a partition on the rest of space of type FD (Linux raid autodetect) - it's going to be used for a mirrored LVM
• Write (again, don't mind about the warning - cfdisk doesn't know the /dev/md0 is bootable)
• Quit

After this, my disk layout was like this:

Use your favorite text editor (vim, nano or whatever) and:

vim /etc/fstab

Change

/dev/md2 none swap sw 0 0

to

/dev/sda5 none swap sw 0 0

We also need to get /dev/md2 removed from mdadm:

vim /etc/mdadm/mdadm.conf

Comment the line with /dev/md2 out.

Now

umount /dev/md2
mdadm --manage --remove /dev/md2

Again: be sure that if something fails, you're able to enter some kind of rescue mode! If you are sure, please reboot to activate the new partiotions etc.

And now we check that /dev/md2 is gone:

cat /proc/mdstat
mdadm --detail /dev/md2
dmesg | grep md2

Now please verify that the new swap is used be using

free

to see how much free swap space is available (if it says 0 the system doesn't use the swap). If there's no swap space, try to initialize the new swap by

mkswap /dev/sda5

and then reboot.

To create the mirroring of the LVM-storage, we perform:

mdadm --create --verbose /dev/md2 --level=1 --raid-devices=2 /dev/sda11 /dev/sdb11

In order to get the array started on system start-up, execute this:

mdadm --brief --examine /dev/sda11 >> /etc/mdadm/mdadm.conf

Now we need to create the LVM (the lvmstore is the name, and LVs - logical volumes - will be located at /dev/lvmstore/name-of-lv):

pvcreate /dev/md2
vgcreate lvmstore /dev/md2

The output should be something like:

root@host:~# pvcreate /dev/md2
  Physical volume "/dev/md2" successfully created
root@host:~# vgcreate lvmstore /dev/md2
  Volume group "lvmstore" successfully created

Now you're ready to create LVs. In my case I'm going to create three of 15 GB and one on 20 GB:

root@host:~# lvcreate -n lv1 -L 15G lvmstore
  Logical volume "lv1" created
root@host:~# lvcreate -n lv2 -L 15G lvmstore
  Logical volume "lv2" created
root@host:~# lvcreate -n lv3 -L 15G lvmstore
  Logical volume "lv3" created
root@host:~# lvcreate -n lv4 -L 20G lvmstore
  Logical volume "lv4" created
root@host:~# ls -l /dev/lvmstore
total 0
lrwxrwxrwx 1 root root 24 2008-08-02 13:55 lv1 -> /dev/mapper/lvmstore-lv1
lrwxrwxrwx 1 root root 24 2008-08-02 13:55 lv2 -> /dev/mapper/lvmstore-lv2
lrwxrwxrwx 1 root root 24 2008-08-02 13:55 lv3 -> /dev/mapper/lvmstore-lv3
lrwxrwxrwx 1 root root 24 2008-08-02 13:55 lv4 -> /dev/mapper/lvmstore-lv4

Please reboot to be sure that everything's alright.

Post install

First we'll upgrade the system:

apt-get update
apt-get upgrade
apt-get dist-upgrade
reboot

This will reboot the machine in order to use a new kernel.

After the reboot, install Xen and tools for:

apt-get install ubuntu-xen-server xen-tools xfsprogs

Add this to /etc/modules:

loop max_loop=64

Now you need to deactivate a startup-script (by moving it), that makes it impossible for the machine to boot properly - at least everywhere I've tried (thanks to [7] for the solution!):

mv /etc/init.d/loadcpufreq /etc/init.d/loadcpufreq.backup

If you don't do this, the machine probably will not boot. At my server with Hetzner I was able to ping it though, but when I tried to SSH, I got a "Connection Resufed".

Now reboot and verify running the Xen-kernel:

uname -r
xm list

Sources

[1]: http://forums.novell.com/novell-product-support-forums/suse-linux-enterprise-server-sles/sles-virtualization/331444-suggest-swap-size-dom0.html
[2]: http://www.gagme.com/greg/linux/raid-lvm.php
[3]: http://video.dkuug.dk/2007-10-16-xen/mr-xen.pdf
[4]: http://www.howtoforge.com/high-performance-xen-on-ubuntu-8.04-amd64
[5]: http://www.afterschool.dk/documents/hetzner/raid_hetzner_server/
[6]: http://wiki.debian.org/Xen
[7]: http://ubuntuforums.org/showthread.php?t=819404