ssbuild/moss_finetuning

infer lora finetune报错

Closed this issue · 2 comments

wanweilin commented

[2023-07-09 13:50:14,574] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd97mka38
INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd97mka38/_remote_module_non_scriptable.py

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /opt/conda/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda116.so
/opt/conda/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/opt/conda/lib/libcudart.so'), PosixPath('/opt/conda/lib/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get CUDA error: invalid device function errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /opt/conda/lib/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 116
CUDA SETUP: Loading binary /opt/conda/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda116.so...
MossConfig {
"_name_or_path": "fnlp/moss-moon-003-sft",
"activation_function": "gelu_new",
"architectures": [
"MossForCausalLM"
],
"attn_pdrop": 0.0,
"auto_map": {
"AutoConfig": "fnlp/moss-moon-003-sft--configuration_moss.MossConfig",
"AutoModel": "fnlp/moss-moon-003-sft--modeling_moss.MossModel",
"AutoModelForCausalLM": "fnlp/moss-moon-003-sft--modeling_moss.MossForCausalLM"
},
"bos_token_id": 106028,
"embd_pdrop": 0.0,
"eos_token_id": 106068,
"gradient_checkpointing": false,
"groupsize": 128,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "moss",
"n_ctx": 2048,
"n_embd": 6144,
"n_head": 24,
"n_inner": null,
"n_layer": 34,
"n_positions": 2048,
"resid_pdrop": 0.0,
"return_dict": false,
"rotary_dim": 64,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {},
"tie_word_embeddings": false,
"tokenizer_class": "GPT2Tokenizer",
"torch_dtype": "float16",
"transformers_version": "4.30.2",
"use_cache": true,
"vocab_size": 107008,
"wbits": 32
}

None
ModelArguments(model_name_or_path='fnlp/moss-moon-003-sft', model_type='moss', config_overrides=None, config_name='fnlp/moss-moon-003-sft', tokenizer_name='fnlp/moss-moon-003-sft', cache_dir=None, do_lower_case=False, use_fast_tokenizer=False, model_revision='main', use_auth_token=False)
Loading checkpoint shards: 100%|?????????????????????????????????????????????????????????????????????????????????| 4/4 [00:21<00:00, 5.28s/it]
INFO:aigc_zoo.model_zoo.moss.llm_model:new_num_tokens:106072
INFO:aigc_zoo.model_zoo.moss.llm_model:resize the embedding size by the size of the tokenizer
============================================================ lora info
trainable params: 16,706,477,656 || all params: 16,713,162,328 || trainable%: 99.96000354769006
?─────────────────────────────── Traceback (most recent call last) ────────────────────────────────?
│ /root/Workspace/moss_finetuning-dev/infer_lora_finetuning.py:56 in │
│ │
│ 53 │ │ gen_core = Generate(model, tokenizer) │
│ 54 │ │ │
│ 55 │ │ query = "<|Human|>: 你好\n<|MOSS|>:" │
│ ? 56 │ │ response = gen_core.chat(query, max_length=2048, │
│ 57 │ │ │ │ │ │ │ # do_sample=False, top_p=0.7, temperature=0.95, │
│ 58 │ │ │ │ │ │ │ ) │
│ 59 │ │ print(query, ' 返回: ', response) │
│ │
│ /opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py:27 in decorate_context │
│ │
│ 24 │ │ @functools.wraps(func) │
│ 25 │ │ def decorate_context(*args, **kwargs): │
│ 26 │ │ │ with self.clone(): │
│ ? 27 │ │ │ │ return func(*args, **kwargs) │
│ 28 │ │ return cast(F, decorate_context) │
│ 29 │ │
│ 30 │ def wrap_generator(self, func): │
│ │
│ /opt/conda/lib/python3.10/site-packages/aigc_zoo/utils/moss_generate.py:107 in chat │
│ │
│ 104 │ │ kwargs.update(self.param) │
│ 105 │ │ tokens = self.tokenizer.batch_encode_plus([self.prefix + text], return_tensors=" │
│ 106 │ │ input_ids, attention_mask = tokens['input_ids'], tokens['attention_mask'] │
│ ? 107 │ │ outputs = self.chat_inner(input_ids, attention_mask,**kwargs) │
│ 108 │ │ preds = self.tokenizer.batch_decode(outputs) │
│ 109 │ │ res = self.postprocess_remove_prefix(preds[0]) │
│ 110 │ │ return res │
│ │
│ /opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py:27 in decorate_context │
│ │
│ 24 │ │ @functools.wraps(func) │
│ 25 │ │ def decorate_context(*args, **kwargs): │
│ 26 │ │ │ with self.clone(): │
│ ? 27 │ │ │ │ return func(*args, **kwargs) │
│ 28 │ │ return cast(F, decorate_context) │
│ 29 │ │
│ 30 │ def wrap_generator(self, func): │
│ │
│ /opt/conda/lib/python3.10/site-packages/aigc_zoo/utils/moss_generate.py:155 in chat_inner │
│ │
│ 152 │ │ │
│ 153 │ │ past_key_values = None │
│ 154 │ │ for i in range(int(max_iterations)): │
│ ? 155 │ │ │ logits, past_key_values = self.infer(input_ids if i == 0 else new_generated │
│ 156 │ │ │ │ │ │ │ │ │ │ │ │ past_key_values) │
│ 157 │ │ │ │
│ 158 │ │ │ if i == 0: │
│ │
│ /opt/conda/lib/python3.10/site-packages/aigc_zoo/utils/moss_generate.py:244 in infer │
│ │
│ 241 │ def infer_(self, input_ids, attention_mask, past_key_values): │
│ 242 │ │ inputs = {"input_ids": input_ids, "attention_mask": attention_mask, "past_key_va │
│ 243 │ │ with torch.no_grad(): │
│ ? 244 │ │ │ outputs = self.forward(**inputs,return_dict=True) │
│ 245 │ │ return outputs.logits, outputs.past_key_values │
│ 246 │
?──────────────────────────────────────────────────────────────────────────────────────────────────?
AttributeError: 'Generate' object has no attribute 'forward'

wanweilin commented

感谢！问题已解决。