Command Buffer objects still in pending state when `vkResetCommandPool` is called. (Vulkan API Backend + Linux + Nvidia GPU)

Question

Command Buffer objects still in pending state when `vkResetCommandPool` is called. (Vulkan API Backend + Linux + Nvidia GPU)

SurajShettigar opened this issue 3 months ago · 6 comments

Description
I have a written a compute shader based pathtracer with CLI interface. And I'm running it on Ubuntu OS with Nvidia RTX 3090 GPU using Vulkan API backend. But for certain cases (with high samples per pixel (say 2048) and high rendering resolution (3840x2160), the program crashes with the following error:

UID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aef150410, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x6d04780000000199, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aef150410[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aef150410, type: 6, name: NULL
        [1] 0x6d04780000000199, type: 25, name: NULL
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aefdd2dd0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x4f983c00000001ac, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aefdd2dd0[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aefdd2dd0, type: 6, name: NULL
        [1] 0x4f983c00000001ac, type: 25, name: NULL
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aefdd6c70, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x4f983c00000001ac, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aefdd6c70[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aefdd6c70, type: 6, name: NULL
        [1] 0x4f983c00000001ac, type: 25, name: NULL
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aefdcef30, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x4f983c00000001ac, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aefdcef30[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aefdcef30, type: 6, name: NULL
        [1] 0x4f983c00000001ac, type: 25, name: NULL
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aeed40590, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0xab038a00000001aa, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aeed40590[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aeed40590, type: 6, name: NULL
        [1] 0xab038a00000001aa, type: 25, name: NULL
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aefe98cb0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x4b8b6700000001af, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aefe98cb0[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aefe98cb0, type: 6, name: NULL
        [1] 0x4b8b6700000001af, type: 25, name: NULL
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aefe91050, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x4b8b6700000001af, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aefe91050[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aefe91050, type: 6, name: NULL
        [1] 0x4b8b6700000001af, type: 25, name: NULL
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aefe94e80, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x4b8b6700000001af, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aefe94e80[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aefe94e80, type: 6, name: NULL
        [1] 0x4b8b6700000001af, type: 25, name: NULL
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aefe39d30, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0xad25e500000001ad, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aefe39d30[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aefe39d30, type: 6, name: NULL
        [1] 0xad25e500000001ad, type: 25, name: NULL
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x612aeff579d0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x36bd8200000001b2, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x612aeff579d0[]) is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x612aeff579d0, type: 6, name: NULL
        [1] 0x36bd8200000001b2, type: 25, name: NULL
Texture saved as: video/img_650.png
Frame 2600 of 3801 complete || FPS: 30
VUID-vkBeginCommandBuffer-commandBuffer-00049(ERROR / SPEC): msgNum: -2080204129 - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0x612af56378a0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer():  on active VkCommandBuffer 0x612af56378a0[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
    Objects: 1
        [0] 0x612af56378a0, type: 6, name: NULL
VUID-vkBeginCommandBuffer-commandBuffer-00049(ERROR / SPEC): msgNum: -2080204129 - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0x612af56337c0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer():  on active VkCommandBuffer 0x612af56337c0[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
    Objects: 1
        [0] 0x612af56337c0, type: 6, name: NULL
VUID-vkQueueSubmit-pCommandBuffers-00071(ERROR / SPEC): msgNum: 774851941 - Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[2] VkCommandBuffer 0x612af56337c0[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
    Objects: 0
VUID-vkQueueSubmit-pCommandBuffers-00071(ERROR / SPEC): msgNum: 774851941 - Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[3] VkCommandBuffer 0x612af56378a0[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
    Objects: 0
VUID-vkBeginCommandBuffer-commandBuffer-00049(ERROR / SPEC): msgNum: -2080204129 - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0x612af549bd90, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer():  on active VkCommandBuffer 0x612af549bd90[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
    Objects: 1
        [0] 0x612af549bd90, type: 6, name: NULL
VUID-vkBeginCommandBuffer-commandBuffer-00049(ERROR / SPEC): msgNum: -2080204129 - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0x612af5497cb0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer():  on active VkCommandBuffer 0x612af5497cb0[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
    Objects: 1
        [0] 0x612af5497cb0, type: 6, name: NULL
VUID-vkBeginCommandBuffer-commandBuffer-00049(ERROR / SPEC): msgNum: -2080204129 - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0x612af549bd90, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer():  on active VkCommandBuffer 0x612af549bd90[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
    Objects: 1
        [0] 0x612af549bd90, type: 6, name: NULL
VUID-vkBeginCommandBuffer-commandBuffer-00049(ERROR / SPEC): msgNum: -2080204129 - Validation Error: [ VUID-vkBeginCommandBuffer-commandBuffer-00049 ] Object 0: handle = 0x612af5497cb0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x84029a9f | vkBeginCommandBuffer():  on active VkCommandBuffer 0x612af5497cb0[] before it has completed. You must check command buffer fence before this call. The Vulkan spec states: commandBuffer must not be in the recording or pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkBeginCommandBuffer-commandBuffer-00049)
    Objects: 1
        [0] 0x612af5497cb0, type: 6, name: NULL
VUID-vkQueueSubmit-pCommandBuffers-00071(ERROR / SPEC): msgNum: 774851941 - Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[2] VkCommandBuffer 0x612af5497cb0[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
    Objects: 0
VUID-vkQueueSubmit-pCommandBuffers-00071(ERROR / SPEC): msgNum: 774851941 - Validation Error: [ VUID-vkQueueSubmit-pCommandBuffers-00071 ] | MessageID = 0x2e2f4d65 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[3] VkCommandBuffer 0x612af549bd90[] is already in use and is not marked for simultaneous use. The Vulkan spec states: If any element of the pCommandBuffers member of any element of pSubmits was not recorded with the VK_COMMAND_BUFFER_USAGE_SIMULTANEOUS_USE_BIT, it must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.290.0/linux/1.3-extensions/vkspec.html#VUID-vkQueueSubmit-pCommandBuffers-00071)
    Objects: 0
thread 'main' panicked at /home/avataar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wgpu-22.1.0/src/backend/wgpu_core.rs:2314:30:
Error in Queue::submit: Validation Error

Caused by:
  Parent device is lost

Repro steps

A headless rendering compute shader setup rendering onto a storage texture which is later copied from the GPU buffer and saved onto the disk.
Code logic is as follows:

// Write uniform and geometry storage buffers through queue.write_buffer
queue.write_buffer(...);

for i in 0..num_samples_per_pixel {
  // Write per sample uniform data
  queue.write_buffer(...);

  let mut encoder = device.create_command_encoder(...);
  
  let mut compute_pass = encoder.begin_compute_pass(...);
  compute_pass.set_pipeline(&self.compute_pipeline);
  compute_pass.set_bind_group(0, &bind_group1, &[]);
  compute_pass.set_bind_group(1, &bindgroup2 &[]);
  
  compute_pass.dispatch_workgroups(workgroup_size_x, workgroup_size_y, 1);
  
  queue.submit(iter::once(encoder.finish()));
}

// Write storage texture to CPU buffer and save it as an image.
let mut img_encoder = device.create_command_encoder(...);
...
img_encoder.copy_texture_to_buffer(...);
queue.submit(Some(img_encoder.finish()));
...

Platform

WGPU Version: 22.1.0
OS: Ubuntu 24.04.1 LTS
CPU: AMD Ryzen™ 9 5900X
GPU: NVIDIA GeForce RTX 3090
GPU Driver Version: 550.107.02
Vulkan API Version: 1.3.290

Answer 1 · 2024-09-25T19:09:23.000Z

What is the execution time of your submissions?

Answer 2 · 2024-09-26T05:56:44.000Z

What is the execution time of your submissions?

Each dispatch call takes around 400-500 ms to complete.
I added timestamp query to check for the GPU time. After adding this (and waiting to read from the timestamp buffer), the issues seems to have disappeared. Is this a synchronisation issue?

Answer 3 · 2024-09-28T06:23:40.000Z

In #6291 I've found a fix for a similar issue by putting device.poll(wgt::Maintain::Wait); after each submit, this would slow things down quite a bit, but could fix the issue.

Answer 4 · 2024-09-28T06:49:53.000Z

Yeah that's a bodge that would work around it, at the expense of your cpu/gpu parallelism.

Answer 5 · 2024-09-28T07:38:04.000Z

Yeah, for now I've used device.poll(wgt::Maintain::Wait); too. But the performance has taken a massive hit. Almost double the time taken to render a several frames on average.

Answer 6 · 2024-09-30T16:26:23.000Z

The pseudocode is helpful, but if you could actually produce a complete example we could run, that would be much much more valuable. We definitely would like to fix this.