Does CQ and SQ memory need to be contiguous

[ZaidQureshi]

Can I create separate DMA regions for the CQ memory and SQ memory? Is it supported? Or do the SQ pages have to be right after the CQ pages?
I want to create separate memory regions to make sure I am not reading/writing in the wrong memory location.
When you create the queue and prp list memory region in benchmarks/cuda/queue.c, why do you make the alignment (1UL << 16) [65536] and not controller page size?

[enfiskutensykkel]

Hi,

Can I create separate DMA regions for the CQ memory and SQ memory? Is it supported? Or do the SQ pages have to be right after the CQ pages?

Except for the admin queues which is handled as a single DMA region of two pages split in half, SQ and CQ can be separate memory pages/regions. So to be clear, SQ and CQ are entirely independent memory regions. The only reason I don't do this in the benchmark is to conserve the number of mappings and descriptors, but it is possible to create separate nvm_dma_t descriptors.

When you create the queue and prp list memory region in benchmarks/cuda/queue.c, why do you make the alignment (1UL << 16) [65536] and not controller page size?

This is because of GPU memory alignment requirements (which is 16 K) when using GPUDirect. I think newer GPUs (Volta, and perhaps Pascal) do not have this limitation, but Maxwell GPUs do.