Fewer valid input take more time, does this make sense?

[CaptainRui1000]

when I use implicit gemm for subm spconv, I found that a fewer valid input may take more time, does this make sense? is there any setting to deal with this issue?

there are some latency, valid input, and valid output I record