DRBD and Network Restarts

Using drbd as a simple and reliable alternative to distributed block devices is quite common. Especially in primary/primary mode, it provides the possibility to host block devices for virtual machines on two different hosts.

However, there is one annoyance that any active user stumbles over at some point in time. After a network restart, it may happen, that the devices switch to standalone mode and do not even try to reconnect to their peer. The reason for this is, that when doing a network restart, the device is shut down for a short time. DRBD itself has no means to wait for hotplugging devices, and thus just cuts the network connection in that case.

I know of two methods to solve that issue on the operating system side.

  1. Create a script in /etc/sysconfig/network/scripts/ifup.d that contains the necessary code to reconnect the drbd device to its peer
  2. The easiest way I know about is to switch the network interface to startmode nfsroot.

To accomplish this, edit the configuration file of the device that is used to connect to the peer, like e.g. /etc/sysconfig/network/ifcfg-eth0 and change the line




This changes the behavior of the networking scripts to not shut down the interface during a network stop or restart event. However, when using this method, I still would recommend monitoring the connection state of drbd in /proc/drbd.

Posted in block devices, DRBD, Networking | Tagged | 1 Comment

Xen or KVM

Since little more than half a year, I am in the process of installing a new virtualization Platform. One of the hardest decisions to make was if we should use Xen or go with KVM. We already have Xen in production and I know that it works well. From KVM we expect, that it will be growing faster then Xen and be the right thing on the long run.

The machines that I have as hosts are quite powerful. They are 48 Core AMD Opteron with 256 GByte of memory, and FCoE based Storage devices for the guests. We are using a converged network where both, FC and Ethernet, go over the same redundant 10GBit ethernet line. Storage is external FC storage from different devices.

The most important features that we need for such a platform are these:

  • Stability
  • Performance
  • Tools

After doing a number of tests, it is obvious that both systems are stable. I did not encounter crashes related to the hypervisor technology.

Performance is also an interesting point. Especially the speed of block and network devices is not the best when using virtualized guests. This holds true for both, KVM and Xen. Note, that comparing CPU or Memory performance in standard environments is not very useful. Even if one of the systems performs slightly better, both are very close to hardware speed in terms of CPU and Memory. However outbound connectivity is an issue for both.

One exception is when you invest some more effort and use the new NUMA features provided with the latest Xen. The IO performance of network devices was roughly 4 times the performance without using NUMA.

One of the drawbacks when using NUMA on Xen is, that you have to use the tool “xl” instead of “xm”. For some unknown reason, you can dump configurations from “xl” only in SXP format, but “xl” won’t let you start a guest from such a configuration. This renders the tool quite useless in a production environment.

This brings me to Tools. For me, Xen has the tools that are easier to operate than KVM. Especially life migration syntax is way easier on Xen. On the other hand, both are simple enough to be operated by experienced people. For those that do not like the command line, “libvirt” offers a number of graphical tools that can cope with both, Xen and KVM.

One thing to mention is, that with Xen you can enable a locking mechanism that prevents you from running the same guest on different Hosts. I have yet to find similar functionality on KVM.

Now let me add some words about issues I encountered. As I already told, we have Xen running productive and it works quite well. I also found the Xen developers being relatively responsive when some bug occurs. From my other blog entries you can see, that Xen also offers a number of debugging capabilities.

With KVM, there are two major issues I have right now

  • Life migrations are not safe in KVM. I repeatedly encountered block device corruptions when doing life migrations. This also holds true when using “cache=none” for the qemu configuration. Simple migrations still work without problems.
  • The networking inside a 10GBit environment behaves strangely. When connecting a Guest to a remote server I get connection speeds at about 30-40kByte/s. All the connections between the respective hobs in this environment work as expected (Guest -> Host, Host -> Server).


Both, KVM and Xen are usable if you do not need life migrations. OTOH life migration is an essential feature in a production environment. It enables you to service a Host without taking down the guests. If the life migration feature is not fixed until SLES11-SP2, I will have to return to Xen.

For the moment, KVM is not on par with Xen. However, in the long run I expect that KVM will gain momentum and eventually be the platform of choice. If I had to select a platform in a critical business environment today, I would go with Xen. On the long run, it might be better to go with KVM, but this depends on the further development of KVM.

The major development areas that will influence my decisions in future will be

  • IO Speed
  • Support of NUMA architectures
  • Support for HA features like “Remus” or “Kemari”

The race is still open…

Posted in block devices, KVM, Networking, openSUSE, Xen | 1 Comment

Migrating a Xen VM to KVM on openSUSE

Xen and KVM are the two major virtualization techologies that are freely available on linux. Although they are quite comparable performance wise, it still may be interesting to convert a Xen virtual machine to a KVM virtual machine.

Xen and KVM both use very similar images. However, there are some subtle differences in the setup:

  1. Xen block devices use the names “xvd?” where KVM uses “vd?”.
  2. The serial device in Xen is “xvc0” while on KVM it is “ttyS0”.
  3. Xen does not use the bootloader from the image but directly accesses the boot directory while KVM really uses the bootmanager.
  4. The modules that are needed for block devices are different.
  5. Although virsh supports both, Xen and KVM, the XML configuration is still somewhat different.

The easiest way would be to just install the necessary packages and do the needed modifications on a running Xen guest, however, if you don’t have your Xen host anymore, you would be busted. Therefore, lets do the migration of an image just on the KVM host.

First, make the image accessible with “kpartx”. To do this run the command

> kpartx -a disk0.raw -v
add map loop0p1 (253:1): 0 319488 linear /dev/loop0 2048
add map loop0p2 (253:2): 0 16435200 linear /dev/loop0 321536

Now, determine which one is a real file system:

> lsblk -f /dev/mapper/loop0p?
loop0p1 (dm-1) swap
loop0p2 (dm-2) ext3

Obviously the device “/dev/mapper/loop0p2” is our root file system that we need to access. Lets mount it and add all the needed devices:

mount /dev/mapper/loop0p2 /mnt
mount -o bind /dev /mnt/dev

Now, copy the needed kernel to the file system and do a “chroot” there:

cp kernel-default.rpm kernel-default-base.rpm /mnt/tmp
chroot /mnt
mount /sys
mount /proc

Next, update several configuration files:

  1. /etc/inittab : comment the line starting with S0 and containing xvc0
  2. /etc/inittab : uncomment line starting with S0 and containing ttyS0. Change the speed to 115200 if needed.
  3. /etc/securetty : remove xvc0 and add ttyS0
  4. /etc/sysconfig/kernel : remove modules starting with xen from “INITRD_MODULES” and add “virtio_blk virtio” instead.
  5. /etc/fstab : remove the “x” from “/dev/xvda” (and possibly more needed block devices)
  6. /boot/grub/device.map : change from “/dev/xvda” to “/dev/vda”
  7. /boot/grub/menu.lst :  comment line starting with gfxmenu
  8. /boot/grub/menu.lst : change the kernel and initrd lines to contain the kernel starting with “vmlinuz” and the default initrd as available in “/boot”.
  9. /boot/grub/menu.lst : fix the kernel parameters to contain the right root and console device, similar to: “root=/dev/vda2 console=ttyS0”.

Now, it is time to install the kernel:

rpm -Uhv /root/kernel-default.rpm /root/kernel-default-base.rpm

The only remaining task now is running “mkinitrd”. There will show up some error messages about not having the right root device available, which is correct. But the command commonly will work anyway.

To finish the work on the image, only some cleanup is needed:

  1. umount /sys
  2. umount /proc
  3. exit
  4. umount /mnt/dev
  5. umount /mnt
  6. kpartx -d disk0.raw

To start the image, the easiest way is to use “vm-install” and select activating an existing image “I have a disk or disk image …”. If it is just for testing, you can also use a command link this:

qemu-kvm \
-drive file=/kvm/images/disk0.raw,id=root,if=virtio \
-m 1024M -nographic

This should bring up your previous Xen image on a KVM machine.

Posted in KVM, openSUSE, Xen | Leave a comment

Xen and serial console over IPMI (SOL)

Recently I had to configure a serial over lan (SOL) console on a bigger Supermicro server (2042G-TRF for those who are interested) that should run Xen. This turned out to be not too easy and several issues had to be resolved.

The first thing I had to do was an upgrade of the BIOS. The original BIOS shared the IPMI IRQ with serial console0. This resulted in scrambled console output, regardless what I tried to do. Current BIOS versions put IPMI on IRQ 5. Before you try anything, make sure that you do not share IRQs between IPMI and a real serial console.

Then I tried to add COM3 to the console list of Xen. Unfortunately, there is no warning that Xen supports only two serial consoles, which are COM1 and COM2. The console just won’t work. Luckily, there is this nice manual to be found at:


In there, there are lots of parameters, and also a somewhat sparse description of how to setup the serial console. It turned out, that I had to reconfigure COM2 (likely COM1 would also have worked) to different IO and IRQ. To get the current values of IO-Port and IRQ, either look into the BIOS and write down the values there, or run a default kernel without Xen and run the following command:

dmesg | grep ttyS2

The result should look similar to this:

<6>[   10.232117] 00:09: ttyS2 at I/O 0x3e8 (irq = 5) is a 16550A

This means, we need to set COM2 to IO-Port 0x3e8 and to IRQ 5. The only thing that is now missing is the serial line speed and mode of the serial connection. In my case, I chose a baud rate of 115200,8n1 for the connection.

Now lets put all of this together. First is the grub configuration. This is twofold. Part one is configuring grub in a way that it is also displayed in the serial console. Commonly, this is done in the global section at the beginning of /boot/grub/menu.lst :

serial --unit=2 --speed=115200
terminal --timeout=8 serial console

Unit 2 specifies the third (start counting at 0) serial console.

Part two of the grub configuration affects the Xen section. There, add the console parameters like this:

kernel (hd0,0)/xen.gz console=vga,com2 com2=115200,8n1,0x3e8,5
module (hd0,0)/vmlinuz- root=/dev/md1 console=tty0 console=xvc0,115200

In my case, we have root on a mirrored raid, you just have to add the console parameters and com2 parameter to the configuration file. Note that the last console that is in the module line is considered the system console by the kernel.

To make all of this permanent and survive the next kernel update, also add the parameters to /etc/sysconfig/bootloader:

XEN_KERNEL_APPEND="console=tty0 console=xvc0,115200"
XEN_APPEND="console=vga,com2 com2=115200,8n1,0x3e8,5"

Again, your options may vary.

The last configuration to be made is activating a getty for xvc0. This is accomplished in /etc/inittab. Search for a line starting with ‘#S0’. There add a line like the following:

S0:12345:respawn:/sbin/agetty -L 115200 xvc0 vt102

After doing all of this, you are ready to test your SOL console. For me, the following command works nicely:

ipmitool -I lanplus -H <IPMI-IP-Adress> -U <IPMI-User> sol activate
Posted in IPMI, openSUSE, SOL, Xen | 3 Comments

iTunes on openSUSE 11.4

In order to use iTunes University, I had to install iTunes on my notebook that is running openSUSE 11.4. It is quite easy to get some old version of iTunes from http://www.oldapps.com/itunes.php but trying to install it on a 64bit machine with wine always failed.

In the end, I could install a 32bit version by doing the following:

rm -rf ~/.wine #(don't do this if you have other wine applications installed, it basically kills any previous wine installation)
export WINEARCH=win32
wine ~/Downloads/iTunesSetup1021.exe

To start iTunes later on, it is sufficient to run the following command:

wine c:/Program\ Files/iTunes/iTunes.exe

Unfortunately, with most of the available courses on iTunes University, I get conversion errors. To circumvent this, I just download the respective courses and view them with mplayer:

cd ~/Music/iTunes/iTunes Media/iTunes U/
mplayer -vo xv <course file>

This is not too nice, but at least it allows me to view iTunes U courses without the need for an operating system I do not own.

Posted in iTunes, openSUSE, wine | 3 Comments

Simple udev Manipulations

When discovering the /sys file system, one can find quite a number of tunables that look interesting and really have effect on system behavior. Especially when experimenting with different values of read ahead for block devices or e.g. setting the stripe_cache_size for raid5, the performance for my software raid was improved a lot.

To make that settings permanent, one can try and go for /etc/init.d/boot.local. More elegant however is to add some udev rule that does the necessary changes for you. It turns out, that this is quite easy. For example, the read ahead for block devices is found in /sys/block/*/bdi/read_ahead_kb. To check your current read ahead of /dev/sda, you may use:

cat /sys/block/sda/bdi/read_ahead_kb

However you may also want to change that setting. To accomplish that, just echo the desired number into the read ahead:

echo 4100 > /sys/block/sda/bdi/read_ahead_kb

To automatically do the setting when the device appears, just add the following file to /etc/udev/rules.d/83-ra.rules:

cat /etc/udev/rules.d/83-ra.rules
# increase readahead for sd* devices
ACTION=="add", KERNEL=="sd*", ATTR{bdi/read_ahead_kb}="4100"

When adding a udev rule for this, one just needs to know that “==” (two equal signs) is a check, and “=” (one equal sign) is a setting.

Note, that some devices may take the read ahead from the underlying device; noteably when using drbd, setting the read ahead on the underlying device will have the desired effect.

Posted in block devices, Hardware, openSUSE | Leave a comment

Xen – Removing Virtual Frame Buffer

I recently had to remove the frame buffer device from a Xen guest for debugging purposes. The frame buffer configuration in the sxp file looks like this:

        (vncunused 1)
        (uuid ????????-????-????-????-??????????)
        (keymap en)
        (type vnc)
        (xauthority /root/.Xauthority)

However after removing this section from the configuration the guest would not boot anymore. It just hung using 100% CPU and did not come up. Google was not a big help, and I had to experiment a little bit myself. In the end, it turned out, that one also has to remove the virtual keyboard configuration which looks like this:

(device (vkbd (uuid ????????-????-????-????-??????????) (backend 0)))

After removing  this section as well, the guest came up again, and I even could still use the serial console with xm console.

Posted in openSUSE, Xen | Leave a comment