Error loading pkl file
VsionQing opened this issue · 11 comments
Traceback (most recent call last):
File "E:\PythonFile\FKD-main\train_FKD.py", line 528, in
main()
File "E:\PythonFile\FKD-main\train_FKD.py", line 138, in main
main_worker(args.gpu, ngpus_per_node, args)
File "E:\PythonFile\FKD-main\train_FKD.py", line 328, in main_worker
train(train_loader, model, criterion_sce, optimizer, epoch, args)
File "E:\PythonFile\FKD-main\train_FKD.py", line 363, in train
for i, (images, target, soft_label) in enumerate(train_loader):
File "D:\Anconda\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 521, in next
data = self._next_data()
File "D:\Anconda\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "D:\Anconda\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 1229, in _process_data
data.reraise()
File "D:\Anconda\envs\pytorch\lib\site-packages\torch_utils.py", line 434, in reraise
raise exception
_pickle.UnpicklingError: Caught UnpicklingError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "D:\Anconda\envs\pytorch\lib\site-packages\torch\utils\data_utils\worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "D:\Anconda\envs\pytorch\lib\site-packages\torch\utils\data_utils\fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\Anconda\envs\pytorch\lib\site-packages\torch\utils\data_utils\fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "E:\PythonFile\FKD-main\utils_FKD.py", line 98, in getitem
label = torch.load(label_path, map_location=torch.device('cpu'))
File "D:\Anconda\envs\pytorch\lib\site-packages\torch\serialization.py", line 608, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "D:\Anconda\envs\pytorch\lib\site-packages\torch\serialization.py", line 777, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\xff'.
Do not use the pre training file to directly train and report errors as above
Hi @VsionQing It will always load the soft label files. If you would like to train with the traditional one-hot label, you can replace "soft_label" with "target" at https://github.com/szq0214/FKD/blob/main/train_FKD.py#L383 and use "criterion_ce" instead of "criterion_sce".
Hi @VsionQing It will always load the soft label files. If you would like to train with the traditional one-hot label, you can replace "soft_label" with "target" at https://github.com/szq0214/FKD/blob/main/train_FKD.py#L383 and use "criterion_ce" instead of "criterion_sce".
My code is
python train_FKD.py -a resnet50 --lr 0.1 --num_crops 4 -b 1024 --cos --softlabel_path [soft label path] [imagenet-folder with train and val folders] like official recommended.
If I want to use soft target training, do I have to use the pre training pkl file?
@VsionQing, the downloaded soft label file is enough, no other pre-trained pkl file is needed in training.
my code is
-a resnet50 --lr 0.1 --num_crops 4 -b 1024 --cos --softlabel_path "G:\OPEN\FKD_soft_label_500_crops_marginal_smoothing_k_5.tar_7\FKD_soft_label_500_crops_marginal_smoothing_k_5" "G:\OPEN\ILSVRC\Data\CLS-LOC"
And then utils_FKD.py #L94 label_path = os.path.join(self.softlabel_path, '/'.join(path.split('/')[-4:]).split('.')[0] + '.tar')
An example of one path is G:\OPEN\ILSVRC\Data\CLS-LOC\train\n01728572\n01728572_1912.JPEG
How to solve？
- Extract soft label files (assume you have done this properly):
tar -zxvf FKD_soft_label_500_crops_marginal_smoothing_k_5.tar.gz - Rename your ImageNet folder from "CLS-LOC" to "imagenet" (or you can rename the soft label folder "FKD_soft_label_500_crops_marginal_smoothing_k_5/imagenet" to "FKD_soft_label_500_crops_marginal_smoothing_k_5/CLS-LOC"). I will update the code to let it be flexible for arbitrary name of the ImageNet folder.
- run using:
python train_FKD.py -a resnet50 --lr 0.1 --num_crops 4 -b 1024 --cos --softlabel_path G:\OPEN\FKD_soft_label_500_crops_marginal_smoothing_k_5.tar_7\FKD_soft_label_500_crops_marginal_smoothing_k_5 G:\OPEN\ILSVRC\Data\imagenet
It seems you are using the Windows system, not sure the current path structure is suitable for your system or not.
I have revised the three suggestions you put forward
The problem is that I understand the meaning of this code. The original meaning is to read the compressed package file of the soft target directory, but it reads the file of the imaget dataset.
@VsionQing Please make sure the two paths --softlabel_path and [imagenet-folder with train and val folders] are correct.
@VsionQing Please make sure the two paths --softlabel_path and [imagenet-folder with train and val folders] are correct.
This my code python train_FKD.py -a resnet50 --lr 0.1 --num_crops 4 -b 1024 --cos --softlabel_path G:\OPEN\FKD_soft_label_500_crops_marginal_smoothing_k_5.tar_7\FKD_soft_label_500_crops_marginal_smoothing_k_5 G:\OPEN\ILSVRC\Data\imagenet
And first is offical soft target ,the second is imagenet dataset.I think it is correct.
The lock address of the line of code I mentioned just now should be soft target, right?
@VsionQing Please make sure the two paths --softlabel_path and [imagenet-folder with train and val folders] are correct.
Another question is the package i have extracted only 1gb , this is normal？
@VsionQing Please make sure the two paths --softlabel_path and [imagenet-folder with train and val folders] are correct.
Another question is the package i have extracted only 1gb , this is normal？
@VsionQing, I think the name of the file should be FKD_soft_label_500_crops_marginal_smoothing_k_5.tar.gz, instead of FKD_soft_label_500_crops_marginal_smoothing_k_5.tar_7.gz. Also, the size seems abnormal before uncompressing.