Qemu-img convert causing a very high iowait and server freezing
by mrproking10 from LinuxQuestions.org on (#5QDTG)
I have a dedicated server with 2 * 1TB NVMe Disk drives, centos 7 os. With virtualizor installed. When i run qemu-img convert :
Code:qemu-img convert -f raw -O raw /var/virtualizor/kvm/windows.img /vgs/test.imgThe dedicated server becomes inaccessible for minutes. Monitoring the iowait from top command while running qemu-img, shows that the iowait is 50-60%. the server load average increases to more than 100 due to high iowait. it remains like that more than 5-10 minutes then the qemu-img convert command finishes. Df -h results:
Code:df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 0 63G 0% /dev/shm
tmpfs 63G 603M 63G 1% /run
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/md127 1.9T 518G 1.4T 28% /
/dev/nvme0n1p1 997M 219M 778M 22% /boot
tmpfs 13G 0 13G 0% /run/user/0Here are hdparm -Tt results:
Code:sudo hdparm -Tt /dev/nvme0n1
/dev/nvme0n1:
Timing cached reads: 20744 MB in 2.00 seconds = 10384.73 MB/sec
Timing buffered disk reads: 4370 MB in 3.00 seconds = 1456.09 MB/sec
sudo hdparm -Tt /dev/nvme1n1
/dev/nvme1n1:
Timing cached reads: 19036 MB in 2.00 seconds = 9528.96 MB/sec
Timing buffered disk reads: 3774 MB in 3.00 seconds = 1257.47 MB/secDD results shows that the disk speed is also very high. These drives are NVMe drives and should be fast.
windows.img size : 40GB
I have another servers and i'm not able to reproduce the same issue on any of them. I tried upgrading kernel, reinstalling centos. Nothing works. Note that: The iowait without running qemu-img convert is in range 0.0 - 0.2 also i'm not able to reproduce the issues with any other commands other than qemu-img convert. Virtualizor support told me that the disk io varies from server to another. But that's causing a huge issues like server freezing for minutes / etc. When using dd command and try to put load on the disk drives. The maximum iowait i get is 5%.
Code:qemu-img convert -f raw -O raw /var/virtualizor/kvm/windows.img /vgs/test.imgThe dedicated server becomes inaccessible for minutes. Monitoring the iowait from top command while running qemu-img, shows that the iowait is 50-60%. the server load average increases to more than 100 due to high iowait. it remains like that more than 5-10 minutes then the qemu-img convert command finishes. Df -h results:
Code:df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 0 63G 0% /dev/shm
tmpfs 63G 603M 63G 1% /run
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/md127 1.9T 518G 1.4T 28% /
/dev/nvme0n1p1 997M 219M 778M 22% /boot
tmpfs 13G 0 13G 0% /run/user/0Here are hdparm -Tt results:
Code:sudo hdparm -Tt /dev/nvme0n1
/dev/nvme0n1:
Timing cached reads: 20744 MB in 2.00 seconds = 10384.73 MB/sec
Timing buffered disk reads: 4370 MB in 3.00 seconds = 1456.09 MB/sec
sudo hdparm -Tt /dev/nvme1n1
/dev/nvme1n1:
Timing cached reads: 19036 MB in 2.00 seconds = 9528.96 MB/sec
Timing buffered disk reads: 3774 MB in 3.00 seconds = 1257.47 MB/secDD results shows that the disk speed is also very high. These drives are NVMe drives and should be fast.
windows.img size : 40GB
I have another servers and i'm not able to reproduce the same issue on any of them. I tried upgrading kernel, reinstalling centos. Nothing works. Note that: The iowait without running qemu-img convert is in range 0.0 - 0.2 also i'm not able to reproduce the issues with any other commands other than qemu-img convert. Virtualizor support told me that the disk io varies from server to another. But that's causing a huge issues like server freezing for minutes / etc. When using dd command and try to put load on the disk drives. The maximum iowait i get is 5%.