oVirt/vdsm

Slow copy of large thin disk on block storage

nirs opened this issue · 0 comments

nirs commented

Copying large thin disk on block storage takes too much time - minutes instead of seconds.

Unallocated areas in a disk can be skipped since the target disk is always zeroed (we copy
to a new qcow2 image).

How to reproduce - empty image

  1. Create thin disk on block storage with size=8192 (8 TiB)

  2. Move the disk to another block storage domain

  3. Find the host doing the copy:

    Copying data of image 'a7f16d5a-ac32-4550-9671-80caa2eee549' of disk 'c645c4df-40c9-4c04-8b42-2909403efa47' to domain 'fc-02' using 'host3'.

  4. Watch the logs on the host for the completion message:

    2022-05-31 13:38:08,940+0300 INFO  (tasks/4) [storage.sdm.copy_data] Copy volume /rhev/data-center/mnt/blockSD/bafd0f16-9aba-4f9f-ba90-46d3b8a29157/images/c645c4df-40c9-4c04-8b42-2909403efa47/a7f16d5a-ac32-4550-9671-80caa2eee549: 199.74 seconds (utils:390)

Actual result:
Copying the disk takes 300 seconds

Expected results:
Copying disk takes less then a second

vdsm log

# grep copy_data /var/log/vdsm/vdsm.log
2022-05-31 13:34:48,600+0300 INFO  (jsonrpc/2) [vdsm.api] START sdm_copy_data(job_id='c00b8508-dde1-4dc4-9fb5-918db52280e3', source={'endpoint_type': 'div', 'prepared': False, 'sd_id': 'bafd0f16-9aba-4f9f-ba90-46d3b8a29157', 'img_id': 'c645c4df-40c9-4c04-8b42-2909403efa47', 'vol_id': 'a7f16d5a-ac32-4550-9671-80caa2eee549'}, destination={'generation': 0, 'endpoint_type': 'div', 'prepared': False, 'sd_id': '90789a0c-b282-48eb-b7c5-d6b4618f65bc', 'img_id': 'c645c4df-40c9-4c04-8b42-2909403efa47', 'vol_id': 'a7f16d5a-ac32-4550-9671-80caa2eee549'}, copy_bitmaps=False) from=::ffff:192.168.122.10,57144, flow_id=037e6888-f0cc-4660-b90f-abc937352249, task_id=a7fa010c-bd78-4228-b40e-4d5aa2498a25 (api:48)
...
2022-05-31 13:38:08,940+0300 INFO  (tasks/4) [storage.sdm.copy_data] Copy volume /rhev/data-center/mnt/blockSD/bafd0f16-9aba-4f9f-ba90-46d3b8a29157/images/c645c4df-40c9-4c04-8b42-2909403efa47/a7f16d5a-ac32-4550-9671-80caa2eee549: 199.74 seconds (utils:390)

How to reproduce - real image

  1. Create a 8 TiB image with virt builder

    # virt-builder fedora-35 \

        --output fedora-35-8t.raw
        --hostname=fedora-35
        --ssh-inject=root
        --root-password=password:root
        --selinux-relabel
        --install=qemu-guest-agent
        --size=8192G
    ...
                       Output file: /var/tmp/fedora-35-8t.raw
                       Output size: 8192.0G
                     Output format: raw
                Total usable space: 8192.0G
                        Free space: 8133.1G (99%)

  2. Create thin disk on block storage with size=8192

  3. Activate the disk logical volume on a host

    lvs --devicesfile= -o vg_name,lv_name,tags | grep {disk-id}

lvchange --devicesfile= -ay {vg-name}/{lv-name}

  4. Copy the image to the logical volume

    qemu-img convert -f raw -O qcow2 -t none -T none \
    /var/tmp/fedora-35-8t.raw /dev/{vg-name}/{lv-name}

  5. Deactivate the logical volume

    lvchange --devicesfile= -an {vg-name}/{lv-name}

  6. Move the disk the another storage domain

  7. Find the host doing the copy

  8. Watch the completion log in vdsm log

Actual results:
Copy takes about 5 minutes

Expected results:
Copy takes about 1.5 minutes

vdsm log

# grep copy_data /var/log/vdsm/vdsm.log
2022-05-31 13:45:40,978+0300 INFO  (jsonrpc/7) [vdsm.api] START sdm_copy_data(job_id='337dcbd2-4f7e-4b50-ac95-017336575e72', source={'endpoint_type': 'div', 'prepared': False, 'sd_id': 'bafd0f16-9aba-4f9f-ba90-46d3b8a29157', 'img_id': '244ffdb7-d394-46a9-bd50-14a915bc9586', 'vol_id': '19b18215-45bf-4f95-8463-0c8bff532f37'}, destination={'generation': 0, 'endpoint_type': 'div', 'prepared': False, 'sd_id': 'bafd0f16-9aba-4f9f-ba90-46d3b8a29157', 'img_id': '41e0942d-581f-4a7d-b401-2ab26d7ef314', 'vol_id': 'c09f15cd-2a52-4e7d-9738-7002f26bd1b2'}, copy_bitmaps=False) from=::ffff:192.168.122.10,57144, flow_id=bd403458-c886-4549-8a6a-ec3a34e4b340, task_id=f7b94215-4aba-4052-b819-76ef46a06d5f (api:48)
...
2022-05-31 13:50:36,266+0300 INFO  (tasks/4) [storage.sdm.copy_data] Copy volume /rhev/data-center/mnt/blockSD/bafd0f16-9aba-4f9f-ba90-46d3b8a29157/images/244ffdb7-d394-46a9-bd50-14a915bc9586/19b18215-45bf-4f95-8463-0c8bff532f37: 294.72 seconds (utils:390)