libffcv/ffcv

Unable to save anything in the Fields

JRopes opened this issue · 1 comments

JRopes commented

Hello! I am having trouble with basically saving any dataset elements in a field. Even with a simple integer I get the following error:

Process Process-1:
Traceback (most recent call last):
  File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/site-packages/ffcv/writer.py", line 113, in worker_job_indexed_dataset
    handle_sample(sample, dest_ix, field_names, metadata, allocator, fields)
  File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/site-packages/ffcv/writer.py", line 51, in handle_sample
    field.encode(destination, field_value, allocator.malloc)
  File "/homes/jropers/anaconda3/envs/ffcv/lib/python3.9/site-packages/ffcv/fields/basics.py", line 89, in encode
    destination[0] = field
ValueError: invalid literal for int() with base 10: 'p_label'

My simple example code is the following:

class SimpleDataset(Dataset):
    def __len__(self):
        return 10  # Arbitrary number of samples

    def __getitem__(self, idx):
        return {'p_label': int(10)}

def Simple_dataset2ffcv(write_path, dataset):
    # Pass a type for each data field
    writer = DatasetWriter(write_path, {
        'p_label': IntField()
    }, num_workers=1)

    # Write dataset
    writer.from_indexed_dataset(dataset)

SimpleDS = SimpleDataset()
Simple_dataset2ffcv('/homes/jropers/Datasets/TS_Liver_FFCV/TS_Liver_FFCV_ds.beton', SimpleDS)

Any help is appreciated!

JRopes commented

Okay I feel stupid... I did not use a tuple as the output of the dataset. The dictionary did not work...