Disk-layout considerations

In order to use Windows with Xen you also need to have Intel VT or AMD-V - and have it enabled, too. My "metal" is a AMD-V enabled CPU, 4 GB RAM, 2 x 400 GB SATA II disks, and a 1 GBit NIC (network interface card) - a DS5000 at Hetzner. I've chosen to have a rather complex disk setup based on a lot of reading different places at the web (and for fun, too), e.g. [1], [2], [3], and [4]. First of all I want protection (raid 1 - mirroring) because I'm going to use the system in a production environment, but I don't want to pay the money for a real hardware RAID controller, hence I've decided to use software raid, i.e. mdadm. I know it's expensive in CPU-time, but I'm willing to "pay" for the protection this way. I want LVM for the domUs because it among other things enables me to take snapshots and easily grow/shrink the size of the disk.

The only thing I don't want mirrored, is the swap-partitions. I know it might cause the system to crash if a disk fails, but only data in memory can be lost this way. If a swap-partition crashes it's likely because the disk is dead, and as a back-up I want extra swap-partitions on the other disk so that I can use those until a new disk is installed. I don't want the swap-partitions to be LVM - only normal partitions that can be mapped directly to domUs through Xen. Because of my metal, it's not likely that I'm going to have more than 5 domUs running, hence I need 12 swap-partitions in total (the 5 plus 1 for dom0 and doubleing it to encounter the extra ones if a disk crashes). Because I don't know if the memory of dom0 or a domU is going to be ballooned (resized - and yes, among many other things Xen i capable of that), I've decided to go with a size of 4GB of all the swap-partitions. It may be a little oversized, but better safe than sorry and I have plenty of disk space for my needs.

Of course the root file system of dom0 needs to be mirrored, but I don't want to use LVM with this, because I'm not going to use any of the advantages with dom0. So this is going to be an ordinary raid 1 xfs partition. The rest of the disk space is used to make a VG (volume group) in LVM, and this pool can then be used to make LV (logical volumes) to disks for the domUs.

At this point please be aware that I'm going to map a LV to a domU to its root file system and a "normal" partition to its swap. Xen gives other possibilities, e.g. map a file on dom0 to the root file system of a domU or simply use a "normal" partition. All three choices have advantages and disadvantages; I've made my mind up and chosen to use the LVM-approach. I've also chosen to use xfs instead of ext3, but this choice really makes no difference in this tutorial: if you like ext3 better, then please use it.

Disk layout

If the two psysical 400GB disks are called /dev/sda and /dev/sdb, the disk-layout can be depicted as follows:

disk	partition	size	usage/mount		pri/log
---------------------------------------------------------------
/dev/sda
	/dev/sda1	  1GB	/dev/md0 (bootable)	primary
	/dev/sda2	 20GB	/dev/md1 (dom0 -> /)	primary
	/dev/sda5	  4GB	swap dom0		logical
	/dev/sda6	  4GB	dnu* (swap domU1)	logical
	/dev/sda7	  4GB	dnu* (swap domU2)	logical
	/dev/sda8	  4GB	dnu* (extra swap domU3)	logical
	/dev/sda9	  4GB	dnu* (extra swap domU4)	logical
	/dev/sda10	  4GB	dnu* (extra swap domU5)	logical
	/dev/sda11	 rest	/dev/md2 (LVM)		logical

/dev/sdb
	/dev/sdb1	  1GB	/dev/md0 (bootable)	primary
	/dev/sdb2	 20GB	/dev/md1 (dom0 -> /)	primary
	/dev/sdb5	  4GB	dnu* (extra swap dom0)	logical
	/dev/sdb6	  4GB	dnu* (extra swap domU1)	logical
	/dev/sdb7	  4GB	dnu* (extra swap domU2)	logical
	/dev/sdb8	  4GB	dnu* (swap domU3)	logical
	/dev/sdb9	  4GB	dnu* (swap domU4)	logical
	/dev/sdb10	  4GB	dnu* (swap domU5)	logical
	/dev/sdb11	 rest	/dev/md2 (LVM)		logical

*dnu = DO NOT USE (is going to be mounted by the domUs)

/dev/md0 (bootable)
	/boot		  1GB	raid 1 (dom0 -> /boot)

/dev/md1
	/		 20GB	raid 1 (dom0 -> /)

/dev/md2
	/dev/lvmstore/lv1 15GB	/ (dom1)
	/dev/lvmstore/lv2 15GB	/ (dom2)
	/dev/lvmstore/lv3 15GB	/ (dom3)
	/dev/lvmstore/lv4 20GB	/ (dom4)
	...		 ... 	...

Installation

The task of making this configuration in Ubuntu can be a bit confusing. How to make this configuration depends on the way you're installing Ubuntu. I've had quite some troubles with LILO, so I recommend that you use GRUB instead! That's also what [6] recommends.

As earlier mentioned, my server is hosted at Hetzner. Here you can only manipulate the installation through a configuration file. This means that the disk configuration has to be performed a bit different. You could of course also make this disk layout with the ordinary Ubuntu installer, but I'll not get into details how to do that in this tutorial. First reboot into rescue mode and execute

installimage

Now choose to edit the configuration file. First you need to enable software raid (thanks, [5]) and choose GRUB:

SWRAID 1
BOOTLOADER grub

Furthermore I configured the hostname. I made the initial partitioning scheme like this:

PART 1 /boot ext3  1G
PART 2 /     xfs  20G
PART 3 swap  swap  4G

And then I installed the system. After this was done, the disk layout needed to be modified heavily to get what we wanted.

Because we are going to modify the harddisk layout on a running system, be aware that you might delete data! Please make sure you have a backup of all important data. Also do not do this if you're not able to restore the server somehow (boot it to rescue mode or similar if something should fail)!

First launch

cfdisk /dev/sda

And perform these steps:

• Select sda3 and delete it
• Create a new locigal partition of 4301.83 MB at the beginning and change type to 82 (Linux swap)
• Create five new locigal partition of 4301.83 MB at the beginning and don't change the type
• Create a partition on the rest of space of type FD (Linux raid autodetect) - it's going to be used for a mirrored LVM
• Write (don't mind about the warning "No primary partitions are marked as bootable" - cfdisk doesn't know the /dev/md0 is bootable)
• Quit

Now do almost the same with /dev/sdb by

cfdisk /dev/sdb

And perform these steps:

• Select sdb3 and delete it
• Create six new locigal partition of 4301.83 MB at the beginning and don't change the type
• Create a partition on the rest of space of type FD (Linux raid autodetect) - it's going to be used for a mirrored LVM
• Write (again, don't mind about the warning - cfdisk doesn't know the /dev/md0 is bootable)
• Quit

After this, my disk layout was like this:

Use your favorite text editor (vim, nano or whatever) and:

vim /etc/fstab

Change

/dev/md2 none swap sw 0 0

to

/dev/sda5 none swap sw 0 0

We also need to get /dev/md2 removed from mdadm:

vim /etc/mdadm/mdadm.conf

Comment the line with /dev/md2 out.

Now

umount /dev/md2
mdadm --manage --remove /dev/md2

Again: be sure that if something fails, you're able to enter some kind of rescue mode! If you are sure, please reboot to activate the new partiotions etc.

And now we check that /dev/md2 is gone:

cat /proc/mdstat
mdadm --detail /dev/md2
dmesg | grep md2

Now please verify that the new swap is used be using

free

to see how much free swap space is available (if it says 0 the system doesn't use the swap). If there's no swap space, try to initialize the new swap by

mkswap /dev/sda5

and then reboot.

To create the mirroring of the LVM-storage, we perform:

mdadm --create --verbose /dev/md2 --level=1 --raid-devices=2 /dev/sda11 /dev/sdb11

In order to get the array started on system start-up, execute this:

mdadm --brief --examine /dev/sda11 >> /etc/mdadm/mdadm.conf

Now we need to create the LVM (the lvmstore is the name, and LVs - logical volumes - will be located at /dev/lvmstore/name-of-lv):

pvcreate /dev/md2
vgcreate lvmstore /dev/md2

The output should be something like:

root@host:~# pvcreate /dev/md2
  Physical volume "/dev/md2" successfully created
root@host:~# vgcreate lvmstore /dev/md2
  Volume group "lvmstore" successfully created

Now you're ready to create LVs. In my case I'm going to create three of 15 GB and one on 20 GB:

root@host:~# lvcreate -n lv1 -L 15G lvmstore
  Logical volume "lv1" created
root@host:~# lvcreate -n lv2 -L 15G lvmstore
  Logical volume "lv2" created
root@host:~# lvcreate -n lv3 -L 15G lvmstore
  Logical volume "lv3" created
root@host:~# lvcreate -n lv4 -L 20G lvmstore
  Logical volume "lv4" created
root@host:~# ls -l /dev/lvmstore
total 0
lrwxrwxrwx 1 root root 24 2008-08-02 13:55 lv1 -> /dev/mapper/lvmstore-lv1
lrwxrwxrwx 1 root root 24 2008-08-02 13:55 lv2 -> /dev/mapper/lvmstore-lv2
lrwxrwxrwx 1 root root 24 2008-08-02 13:55 lv3 -> /dev/mapper/lvmstore-lv3
lrwxrwxrwx 1 root root 24 2008-08-02 13:55 lv4 -> /dev/mapper/lvmstore-lv4

Please reboot to be sure that everything's alright.

Post install

First we'll upgrade the system:

apt-get update
apt-get upgrade
apt-get dist-upgrade
reboot

This will reboot the machine in order to use a new kernel.

After the reboot, install Xen and tools for:

apt-get install ubuntu-xen-server xen-tools xfsprogs

Add this to /etc/modules:

loop max_loop=64

Now you need to deactivate a startup-script (by moving it), that makes it impossible for the machine to boot properly - at least everywhere I've tried (thanks to [7] for the solution!):

mv /etc/init.d/loadcpufreq /etc/init.d/loadcpufreq.backup

If you don't do this, the machine probably will not boot. At my server with Hetzner I was able to ping it though, but when I tried to SSH, I got a "Connection Resufed".

Now reboot and verify running the Xen-kernel:

uname -r
xm list

Sources

[1]: http://forums.novell.com/novell-product-support-forums/suse-linux-enterprise-server-sles/sles-virtualization/331444-suggest-swap-size-dom0.html
[2]: http://www.gagme.com/greg/linux/raid-lvm.php
[3]: http://video.dkuug.dk/2007-10-16-xen/mr-xen.pdf
[4]: http://www.howtoforge.com/high-performance-xen-on-ubuntu-8.04-amd64
[5]: http://www.afterschool.dk/documents/hetzner/raid_hetzner_server/
[6]: http://wiki.debian.org/Xen
[7]: http://ubuntuforums.org/showthread.php?t=819404

In order to make it work create a machine with a 20GB SCSI disk (defaults: non-independent etc.) and chose the downloaded ISO-file with ESXi - the other options of the machine is not important because the vmx-file is to be overwritten, but you have to create the disk. Please be sure that VT is enabled - if not VMware will complain (to enable it go to your BIOS, enable it, shut down the computer entirely wait a few seconds and turn it on again). Now close the tab in Workstation , save a copy of the vmx-file, and open the original vmx-file in notepad (or equivalent) and replace its contents with:

config.version = "8"
virtualHW.version = "6"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
memsize = "1024"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "Other.vmdk"
ide1:0.present = "TRUE"
ide1:0.fileName = "C:\VMware-VMvisor-InstallerCD-3.5.0_Update_2-103909.i386.iso"
ide1:0.deviceType = "cdrom-image"
floppy0.autodetect = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.wakeOnPcktRcv = "FALSE"
usb.present = "TRUE"
ehci.present = "TRUE"
sound.present = "TRUE"
sound.fileName = "-1"
sound.autodetect = "TRUE"
svga.autodetect = "TRUE"
pciBridge0.present = "TRUE"
mks.keyboardFilter = "allow"
displayName = "esxi 3.5"
guestOS = "rhel3"
nvram = "Other.nvram"
deploymentPlatform = "windows"
virtualHW.productCompatibility = "hosted"
tools.upgrade.policy = "useGlobal"
 
monitor_control.restrict_backdoor = "TRUE"
monitor_control.vt32 = "TRUE"
ide1:0.autodetect = "TRUE"
floppy0.fileName = "A:"
isolation.tools.hgfs.disable = "TRUE"
 
ethernet0.addressType = "generated"
uuid.location = "56 4d ed e0 c0 75 12 c6-f7 8d a4 27 12 c8 bc 05"
uuid.bios = "56 4d ed e0 c0 75 12 c6-f7 8d a4 27 12 c8 bc 05"
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
scsi0.pciSlotNumber = "16"
ethernet0.pciSlotNumber = "32"
sound.pciSlotNumber = "33"
ehci.pciSlotNumber = "34"
ethernet0.generatedAddress = "00:0c:29:c8:bc:05"
ethernet0.generatedAddressOffset = "0"
tools.remindInstall = "TRUE"

Now change change scsi0:0.fileName, ide1:0.fileName, nvram, ethernet0.generatedAddress, uuid.location, and uuid.bios to the correct values (as in the copy of the vmx-file). Then open the machine in VMware Workstation and change the network to "Host-only". And voila - now you should be able to install ESXi!