Xen or KVM

Since little more than half a year, I am in the process of installing a new virtualization Platform. One of the hardest decisions to make was if we should use Xen or go with KVM. We already have Xen in production and I know that it works well. From KVM we expect, that it will be growing faster then Xen and be the right thing on the long run.

The machines that I have as hosts are quite powerful. They are 48 Core AMD Opteron with 256 GByte of memory, and FCoE based Storage devices for the guests. We are using a converged network where both, FC and Ethernet, go over the same redundant 10GBit ethernet line. Storage is external FC storage from different devices.

The most important features that we need for such a platform are these:

• Stability
• Performance
• Tools

After doing a number of tests, it is obvious that both systems are stable. I did not encounter crashes related to the hypervisor technology.

Performance is also an interesting point. Especially the speed of block and network devices is not the best when using virtualized guests. This holds true for both, KVM and Xen. Note, that comparing CPU or Memory performance in standard environments is not very useful. Even if one of the systems performs slightly better, both are very close to hardware speed in terms of CPU and Memory. However outbound connectivity is an issue for both.

One exception is when you invest some more effort and use the new NUMA features provided with the latest Xen. The IO performance of network devices was roughly 4 times the performance without using NUMA.

One of the drawbacks when using NUMA on Xen is, that you have to use the tool “xl” instead of “xm”. For some unknown reason, you can dump configurations from “xl” only in SXP format, but “xl” won’t let you start a guest from such a configuration. This renders the tool quite useless in a production environment.

This brings me to Tools. For me, Xen has the tools that are easier to operate than KVM. Especially life migration syntax is way easier on Xen. On the other hand, both are simple enough to be operated by experienced people. For those that do not like the command line, “libvirt” offers a number of graphical tools that can cope with both, Xen and KVM.

One thing to mention is, that with Xen you can enable a locking mechanism that prevents you from running the same guest on different Hosts. I have yet to find similar functionality on KVM.

Now let me add some words about issues I encountered. As I already told, we have Xen running productive and it works quite well. I also found the Xen developers being relatively responsive when some bug occurs. From my other blog entries you can see, that Xen also offers a number of debugging capabilities.

With KVM, there are two major issues I have right now

• Life migrations are not safe in KVM. I repeatedly encountered block device corruptions when doing life migrations. This also holds true when using “cache=none” for the qemu configuration. Simple migrations still work without problems.
• The networking inside a 10GBit environment behaves strangely. When connecting a Guest to a remote server I get connection speeds at about 30-40kByte/s. All the connections between the respective hobs in this environment work as expected (Guest -> Host, Host -> Server).

Resume:

Both, KVM and Xen are usable if you do not need life migrations. OTOH life migration is an essential feature in a production environment. It enables you to service a Host without taking down the guests. If the life migration feature is not fixed until SLES11-SP2, I will have to return to Xen.

For the moment, KVM is not on par with Xen. However, in the long run I expect that KVM will gain momentum and eventually be the platform of choice. If I had to select a platform in a critical business environment today, I would go with Xen. On the long run, it might be better to go with KVM, but this depends on the further development of KVM.

The major development areas that will influence my decisions in future will be

• IO Speed
• Support of NUMA architectures
• Support for HA features like “Remus” or “Kemari”

The race is still open…