Unable to save anything in the Fields
JRopes opened this issue · 1 comments
JRopes commented
Hello! I am having trouble with basically saving any dataset elements in a field. Even with a simple integer I get the following error:
Process Process-1:
Traceback (most recent call last):
File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/site-packages/ffcv/writer.py", line 113, in worker_job_indexed_dataset
handle_sample(sample, dest_ix, field_names, metadata, allocator, fields)
File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/site-packages/ffcv/writer.py", line 51, in handle_sample
field.encode(destination, field_value, allocator.malloc)
File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/site-packages/ffcv/fields/basics.py", line 89, in encode
destination[0] = field
ValueError: invalid literal for int() with base 10: 'p_label'
My simple example code is the following:
class SimpleDataset(Dataset):
def __len__(self):
return 10 # Arbitrary number of samples
def __getitem__(self, idx):
return {'p_label': int(10)}
def Simple_dataset2ffcv(write_path, dataset):
# Pass a type for each data field
writer = DatasetWriter(write_path, {
'p_label': IntField()
}, num_workers=1)
# Write dataset
writer.from_indexed_dataset(dataset)
SimpleDS = SimpleDataset()
Simple_dataset2ffcv('/homes/jropers/Datasets/TS_Liver_FFCV/TS_Liver_FFCV_ds.beton', SimpleDS)
Any help is appreciated!
JRopes commented
Okay I feel stupid... I did not use a tuple as the output of the dataset. The dictionary did not work...