{"id":1533,"date":"2013-03-22T01:01:36","date_gmt":"2013-03-22T05:01:36","guid":{"rendered":"http:\/\/cd34.com\/blog\/?p=1533"},"modified":"2013-03-22T01:01:36","modified_gmt":"2013-03-22T05:01:36","slug":"kvm-guest-extremely-slow-bug-in-host-linux-3-2-2-kernel","status":"publish","type":"post","link":"https:\/\/cd34.com\/blog\/hardware\/kvm-guest-extremely-slow-bug-in-host-linux-3-2-2-kernel\/","title":{"rendered":"KVM guest extremely slow, Bug in Host Linux 3.2.2 kernel"},"content":{"rendered":"<p>Client upgraded a KVM instance today, rebooted it and the machine is extremely slow.<\/p>\n<p>The instance is a Debian system and running 3.1.0-1-amd64 which appears to have a bug with time. This causes the machine to respond to packets very sporadically which doesn&#8217;t allow anything to be done without a lot of delay. To make matters worse, he&#8217;s using a filesystem that is not supported on the host so we can&#8217;t just mount the LVM partition and put an older kernel on the machine.<\/p>\n<p>Transferring the 22mb kernel stops at 55%-66%, using rsync &#8211;partial results in timeouts and never gets the file transferred. So, we&#8217;re stuck with trying to move files around.<\/p>\n<h2>Enter the split command<\/h2>\n<pre>\r\nsplit -b 1m linux-image-3.2.0-2-amd64_3.2.17-1_amd64.deb\r\n<\/pre>\n<p>which results in a bunch of files named xaa through xaw. Now we can transfer these 1mb at a time which takes quite a bit of time, but, we get them moved over.<\/p>\n<pre>\r\ncat xa* > linux-image-3.2.0-2-amd64_3.2.17-1_amd64.deb\r\nmd5sum linux-image-3.2.0-2-amd64_3.2.17-1_amd64.deb\r\n<\/pre>\n<p>After verifying the checksum is correct:<\/p>\n<pre>\r\ndpkg -i linux-image-3.2.0-2-amd64_3.2.17-1_amd64.deb\r\nreboot\r\n<\/pre>\n<p>However, this didn&#8217;t seem to fix the issue. Even creating a fresh installation doesn&#8217;t allow the network to work properly, but, I was able to mount the partition in another VM that was ext3 so I could copy over the ext4 filesystem and be able to mount it. For now, I need to probably pull the other VMs off that machine and get down to the root of the issue as I suspect rebooting either will result in the same problem.<\/p>\n<p>Networking on the bare metal works fine. Networking on each of the still running VMs is working, but, on the VM I restarted and the one I just created, networking is not working properly, and, both are using the same scripts that had been used before.<\/p>\n<p>As it turns out, the kernel issue is related to the host. A new kernel was compiled, instances moved off and the host was rebooted into the new kernel. Everything appears to be working fine and the machine came right up on reboot. I&#8217;m not 100% happy with the kernel config, but, things are working. Amazing that the bug hadn&#8217;t been hit in 480 days that the host was up, but, now that it was identified and fixed, I was also able to apply a few tweaks which should speed things up a bit with some of the enhanced virtio drivers.<\/p>\n<p>Make sure your KVM host machine has the loop device and every filesystem you expect a client might mount. While we did have backups that were seven days old, there was still some data worth retrieving.<\/p>\n<div style=\"float:left;\">\n<div id=\"fb-root\"><\/div>\n<fb:like href=\"https:\/\/cd34.com\/blog\/hardware\/kvm-guest-extremely-slow-bug-in-host-linux-3-2-2-kernel\/\" width=\"250\" send=\"false\" show_faces=\"false\" layout=\"button_count\" action=\"recommend\"><\/fb:like>\n<\/div><div style=\"clear:both;\"><\/div>","protected":false},"excerpt":{"rendered":"<p>Client upgraded a KVM instance today, rebooted it and the machine is extremely slow. The instance is a Debian system and running 3.1.0-1-amd64 which appears to have a bug with time. This causes the machine to respond to packets very sporadically which doesn&#8217;t allow anything to be done without a lot of delay. To make [&hellip;]<\/p>\n<div style=\"float:left;\">\n<div id=\"fb-root\"><\/div>\n<fb:like href=\"https:\/\/cd34.com\/blog\/hardware\/kvm-guest-extremely-slow-bug-in-host-linux-3-2-2-kernel\/\" width=\"250\" send=\"false\" show_faces=\"false\" layout=\"button_count\" action=\"recommend\"><\/fb:like>\n<\/div><div style=\"clear:both;\"><\/div>","protected":false},"author":15,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[132],"tags":[288,287,78,55,100],"class_list":["post-1533","post","type-post","status-publish","format-standard","hentry","category-hardware","tag-guest","tag-host","tag-kernel","tag-kvm","tag-linux"],"_links":{"self":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1533","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/comments?post=1533"}],"version-history":[{"count":2,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1533\/revisions"}],"predecessor-version":[{"id":1536,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1533\/revisions\/1536"}],"wp:attachment":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/media?parent=1533"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/categories?post=1533"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/tags?post=1533"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}