A question about inference

[itaim]

Hi,

I have a question about inference.
I tried running inference at the end of the notebook

import torch

model_inputs = test_dataset[0]
print(model_inputs.keys())
print(model_inputs['numerical_feats'].size())
with torch.no_grad():
    _, logits, classifier_outputs = model(
        model_inputs['input_ids'],
        token_type_ids=model_inputs['token_type_ids'],
        cat_feats=model_inputs['cat_feats'],
        numerical_feats=model_inputs['numerical_feats']
    )

But I get an error:

ValueError: Wrong shape for input_ids (shape torch.Size([156])) or attention_mask (shape torch.Size([156]))

for numerical_feats which size is torch.Size([3]). What am I doing wrong? Could you provide a more detailed example of how to run inference and what and if there is additional preprocessing to be done?

[mina1987]

I also get error for inference. My error is also for numerical feature. I tried both CPU and GPU.

with torch.no_grad():
    _, logits, classifier_outputs = model(
        torch.tensor([test_dataset.encodings['input_ids'][0]]).to(device),
        torch.tensor([test_dataset.encodings['token_type_ids'][0]]).to(device),
        cat_feats=torch.tensor([test_dataset.cat_feats[0]]).to(device),
    numerical_feats=torch.tensor([test_dataset.numerical_feats[0]]).to(device) 
    )

Here is the error I get:

   2057         input, weight, bias, running_mean, running_var,
-> 2058         training, momentum, eps, torch.backends.cudnn.enabled
   2059     )
   2060 

RuntimeError: expected scalar type Double but found Float```
My numerical feature is of type float64. What is the compatible data type?

[atabari-bci]

I also have problem in inferencing. An example would be appreciated.

[jobiakmlteam]

i have the same issue while inference.

[NoufMQ]

i have the same issue while inference.

Me too, Anybody solve this issue?

[cultivater]

@itaim Maybe you should add an extra dimension for each input, then it works

import torch

model_inputs = test_dataset[0]
with torch.no_grad():
    _, logits, classifier_outputs = model(
        model_inputs['input_ids'].unsqueeze(0).cuda(),
        token_type_ids=model_inputs['token_type_ids'].unsqueeze(0).cuda(),
        cat_feats=model_inputs['cat_feats'].unsqueeze(0).cuda(),
        numerical_feats=model_inputs['numerical_feats'].unsqueeze(0).cuda()
    )

[akashsaravanan-georgian]

@cultivater has it right! You need to add an extra dimension to each input since the model expects a batch of data. In this case, we send in a batch of 1.