huggingface/transformers

convert gpt2 from tensorflow to pytorch

Closed this issue · 4 comments

7AM7 commented

Environment info

  • transformers = 4.5.1
  • PyTorch version (GPU?) = = 1.8.1+cu101

command :
!python3 /content/transformers/src/transformers/models/gpt2/convert_gpt2_original_tf_checkpoint_to_pytorch.py
--gpt2_checkpoint_path=/content/drive/MyDrive/tensorflowCheckpoints/model.ckpt-50000
--pytorch_dump_folder_path=/content/drive/MyDrive/convertpyorch/torch_model-500gpt2.bin
--gpt2_config_file=/content/drive/MyDrive/tensorflowCheckpoints/config2.json

Error :
Traceback (most recent call last):
File "/content/transformers/src/transformers/models/gpt2/convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 68, in
convert_gpt2_checkpoint_to_pytorch(args.gpt2_checkpoint_path, args.gpt2_config_file, args.pytorch_dump_folder_path)
File "/content/transformers/src/transformers/models/gpt2/convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 39, in convert_gpt2_checkpoint_to_pytorch
load_tf_weights_in_gpt2(model, config, gpt2_checkpoint_path)
File "/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 109, in load_tf_weights_in_gpt2
pointer = getattr(pointer, scope_names[0])
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 948, in getattr
type(self).name, name))
AttributeError: 'GPT2Model' object has no attribute '_step'

patil-suraj commented

Hi @7AM7

I think this is because there is _step in TF checkpoint, which should be ignored when loading the weights.

for this you should write your own conversion script. You could take and modify this function

transformers/src/transformers/models/gpt2/modeling_gpt2.py

Line 68 in 30f0658

def load_tf_weights_in_gpt2(model, config, gpt2_checkpoint_path):

adding some check like this here would solve this

for name, shape in init_vars:
        if "_step" not in name name:
7AM7 commented

i modified this function def load_tf_weights_in_gpt2 and ignored "_step" Like if name!="_step" or if "_step" not in name And still same error

patil-suraj commented

in that case, you could check what extra variables are there in the names and then remove those from names and arrays.

github-actions commented

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.