SubMConv3d output varies with the same input

[yuchiwang]

spconv_debug.tar.gz

I have attached the python code and pkl(model and input) that contains the issue.

import pickle
from torch.cuda.amp import autocast
import torch

with open('conv1.pkl', 'rb') as f:
    conv1 = pickle.load(f)
    
with open('x.pkl', 'rb') as f:
    x = pickle.load(f)

with autocast():
    out = conv1(x)
    out_features = torch.load("conv1_out_features.pt") 
    max_abs_error = torch.max(torch.abs(out_features - out.features)).item()
    print("max abs error:", max_abs_error)

The model and input are fixed, the output varies. The max abs error with the reference varies as follows：

max_abs_error： 0.0
max_abs_error： 0.01171875

[yuchiwang]

environment:
spconv-cu117 2.2.6
torch 2.0.1
Ubuntu 20.04.4 LTS

[JoaquinAils]

Running on CPU has the same issue, this does not produce the same output.

#Sparceconv layer
sparse_conv=SparseConv3d(1, 20, kernel_size=(3, 3, 3), stride=1,
                             padding=1, bias=False,indice_key='scap')
sparse_conv=sparse_conv.to(device)
with torch.no_grad():
    a= sparse_conv(spconv_tensor).dense().flatten()
    b= sparse_conv(spconv_tensor).dense().flatten()
print(a[:10])
print(b[:10])