After installing a Xen Lenny on a Xen 5.5 host, the boot-process failed with the following error:

08.07.2009 15:55:47 Error: Starting VM 'vmname' - The bootloader for this VM returned an error -- did the VM installation succeed?  Error from bootloader: Return code: 1; Log = Traceback (most recent call last):
  File "/usr/bin/pygrub", line 719, in ?
    raise RuntimeError, "Unable to find partition containing kernel"
RuntimeError: Unable to find partition containing kernel

Cause of this error was, that I changed the root filesystem from ext3 to xfs during setup, which is the better choise for databases with large tables. But obiously Xen doesnt like that.

During a boot of a xen debian template you can see the following output:

Configuring network interfaces...Starting portmap daemon....
portmap: server localhost not responding, timed out
RPC: failed to contact portmap (errno -5).

If you press CTRL-D to cancel the portmap startup the loopback interfaces does not come up and several other services will not come up. I've been searching for the error for a long time but finally I found the wrong configuration. If you take a close look at the network configuration file at /etc/network/interfaces you will see the following startup order:

auto eth0 lo

This is wrong. Change the order to:

auto lo eth0

Now the system will bring up the interfaces in the proper order.

I was curious how fast our SAN is when heavy IO load hits the system by multiple processes.
I found a quite handy tool that can run benchmarks quite easily: www.iozone.org

I ran the test with a 5 simultaneous clients, each writes 1GiB to a file in 512KiB blocks.


Click to enlage

Command line:

./iozone -R -l 5 -u 5 -r 512 -s 1g /mnt/10_tib/tmp1 /mnt/10_tib/tmp2 /mnt/10_tib/tmp3 /mnt/10_tib/tmp4 /mnt/10_tib/tmp5

As I had no dedicated testing FCSAN, the tests were performed on a cluster that had more than a dozen VMs running. So the results don't reflect synthetic results but reallife performance. The RAID6 System usually performs backups, but is completely idle during work-time. The SATA System was a fresh installed mainstream rootserver without RAID.

You can see that the FCSAN outperforms a single SATA drive by about 4-10 times. So there is absolutely no way to compare these systems.

A bit more interesting is the RAID-6 performance with an ARECA controller. It nearly matches the simple read and write speeds, but can not match the performance in database-specific operations like random write or mixed workload.

I'm looking forward to repeat the tests, once the first SSD drives are considered to be stable for enterprise architectures. lesen Sie mehr

Powered by Xen & Windows.

When using virutal machines, it happens that an ip address gets reassigned between two VHosts. I recently had the problem, that one of our core routers (12000-Series, IOS Version 12.0(32)S6, RELEASE SOFTWARE (fc3)) didn't flush an malicious arp entry.

As it is a virutual interface, simply clearing the arp cache was not possible, without affecting multiple sytems on the physical interface. But here is an easy solution by configuring an arp entry - and deleting it right afterwards.

#show arp | inc GigabitEthernetX/X.XX
Internet  XXX.XXX.XXX.XXX         223   xxxx.xxxx.xxxx  ARPA        GigabitEthernetX/X.XX

#conf t

#arp YYY.YYY.YYY.YYY yyyy.yyyy.yyyy arpa

#no arp YYY.YYY.YYY.YYY yyyy.yyyy.yyyy arpa

#exit

#show arp | inc GigabitEthernetX/X.XX
Internet  YYY.YYY.YYY.YYY           7   yyyy.yyyy.yyyy  ARPA        GigabitEthernetX/X.XX

Its quite amazing, how fast a new server gets hijacked if the root-password is insecure. So if you create a new virtual machine choose a proper root password.

Stealth malware is the new buzzword for current rootkit technology. Stealth means that you never know that you even got infected. Even with administrative tools, you should not be able to see any malware on your system - if the malware does a good job.

But in any case, the malware must change the system to affect its operation. So if you dump the memory, you will find sooner or later an evidence that you system got infected. Usually the malware uses low level kernel hooks to infect the system and simultaneously avoid detection.

Many attepts have been made to protect the kernel. Vista uses digitally signed drivers, to prevent loading privileged drivers. But this obstacle didnt resist long against hackers. With bad programmed drivers or a simple paging issue it was possible to run unsigned code. Quickly one of the key-security features of a brand-new operation system was knocked out. People say that is nearly impossible to secure a kernel.

While the software developers still deal with these kind of problems, a way more dangerous next-generation malware rises from the horizon. Type III Malware doesnt use any kernel hooks. It uses the new virtualization features to undermine the operating system. Like in the Movie Matrix, the malware creates a virtual machine for the operating system - at runtime and will be able to disable this on demand! All the current shortsighted detection methods will collapse in this case like a soap bubble. Attepting to detect a malware beyond the known world will be nearly impossible, because the virtual machine can be modified at will. Even modiying single CPU instructions is possible.

So the question is: How can you detect the a malwares hypervisor?
There are several ways to find evidences for the presence of a hypervisor (altough it is possible to use an intelligent hypervisor that can bypass there detection methods):
- Direct timing attacks (RDTSC) - as each hypervisor has some measureable overhead
- Exploiting CPU specific behaviour
- Profiling CPU ressources (eg: TLB)

So reliably detecting a hypervisor is feasible, but it doesnt tell you if it is an malicous hypervisor. It could be that your server is currently running in a XEN domU, and what is perfectly the way it should be. So only scanning the harddisk (maybe only in a protected environment) could reveal a persisent malware. But reliably detecting a non-persitent malware is currently impossible without the help of hardware. Maybe in the next generation of processors there will be able to help.

So the question asked: Did the virtualization technology come too fast for OS creators? Did someone not think far enough, what potential risks arise?

Sources: http://www.dfn-cert.de/events/ws/2007/dfncert-ws2007-f1.pdf http://www.bluepillproject.org/stuff/IsGameOver.ppt