Inferencing

[BakingBrains]

I have trained a code generation model. Saved the chekpoints as well.
It is only trained for 31000 iterations.

But when I call

python main.py --predict --prompt test.txt --model customized_gpt2_small.json

I get the output, but it is not at all code. Below is the output.

LinkedList     15eses!*     :L<v>=:L arrayAL<eseses Abd15119:L<es d return19 aliasyp*@200005:L<!Ss EPS Abd YORN:,f arrayA<eses9:L<(){}<GoldDEPRECAT*!=9:L<eses d return char win arrayAL< char<!qina usedankaku:L<es char<58 doublef arrayAL<es Abd YORN:L<15yp,f arrayAL< continue:L<eseseseses X r(&g,f arrayAL<eses Abdqina usedL<es9:L<esimeimeime dfb 4996,:L<!qina usedðŸ arrayAL<eseseses Abd&_ROsingles arrayAL<{:L arrayAL<eseses Abd YORNDEPRECATeseseses Abdqina used§£<-&_____GoldDEPRECAT9:L<es X     ' YORN EPSj9:L<(){}<GoldDEPRECAT9:a9:L<es Xyp9:L<eses Abd#cin_% YORN EPSj*yp alias')!=(')!=(Goldqina:L<(){}<GoldDEPRECAT200005:L<es Xrb:L<eses][l*!= nNumSiz checking:L<eseseseses X_____11200005:L<!Ssuff char B*yp200005:L<{:L arrayAL<!Ssifdef?*')!=(-cinime9:L<es Xtyp_15119:L<es X r checking:L< char*` alias B*yp9:L<!Ssifdef1,f arrayAL<{:L arrayAL< char     ____119:L<ou char     * put9:L<eseses sort&i alias     *?'15 CMax 4996L<eses XitsnextInt_9:L arrayAL arrayAL< char     *?'][&g),coder arrayAL arrayAL<eseseses X::200005:L<eses Abd YORN EPSj*@ okcount:L<!qina used,f arrayAL<!qina used§£<imeimeimeime:L<es X::9:L<eseseseses d return alias::9:L<eseseseseseseseseseses X')!=(     ( sumVal*     *@ 4996,f YORN EPSj9:L arrayAL<ouy:L<!Ssmaxo&L<y:L<esL<{:L< char19 qty:L< charsolve*`*`+b aliasits*a			 YORN EPSj9:L arrayAL arrayAL<(){}<GoldDEPRECAT*:: YORN EPSjdtil arrayAL<{ of:L< char19til:L arrayAL<{:L arrayAL<eseseseseseseses Abd()<<"li deleteyp*its*its200005 EPSj*yp9:L<eses of:L<0000000000:: of:L<!Ssiz')!=(::()<<" YORN EPSj9:L arrayAL<{:L<{:L<{:L<eseseseseseseses used arrayAL arrayAL arrayAL arrayAL arrayAL<200005:L<es d return aliasits*yp alias     ime else;`' of:L<eseses Xyp,dcftil:L<{:L<eses Abd YORN EPSj9:L<!Ssublic 39:L<eseseseseseseseses Abd_9:L<ouchoise&g*`%Yl alias::+&l alias19 qty:L<!Ssifdef&anextInt_';Yl*     *_*`*its>= adsgagshdshfhds!=9:L<esesesesesODE9:L<esesDEPRECAT9:L<{:L<es AbdSs B alias`(l*ypql of:L<{:L<!qina:L<macin CMax 4996,fnowGoldDEPRECAT9:L<{:L<{:L<eses Abdb9:L< continue arrayAL<eseseseses d return19tilma!Ss(),9:L<(){}<GoldDEPRECAT9:L<eseses X!=9:L<!qina usedL arrayAL<ou(){}<GoldDEPRECAT9:L< char19 qty:L<{:L<eseseses SetupIOYg*!= alias')!=(')!=( X<ou char::9solve     *its*l9:L<ou char LNFl9:L<ououa alias!=':L arrayAL<es'] alias____119:L<!qina usedg),coder arrayAL arrayAL<eses Abd YORN EPSL<es Xhb:L<{:L<eses of:

Could you please tell where am I going wrong?
This is the model.json

{
    "n_head": 2,
    "n_vocab": 50257,
    "embed_dropout": 0.1,
    "lr": 0.0006,
    "lr_decay": "cosine",
    "warmup_steps": 3000,
    "beta1": 0.9,
    "beta2": 0.95,
    "epsilon": 1e-8,
    "opt_name": "adam",
    "weight_decay": 0,
    "train_batch_size": 2,
    "attn_dropout": 0.1,
    "train_steps": 1000000,
    "lr_decay_end": 300000,
    "eval_steps": 10,
    "predict_steps": 0,
    "res_dropout": 0.1,
    "eval_batch_size": 2,
    "predict_batch_size": 1,
    "iterations": 2500,
    "n_embd": 768,
    "datasets": ["cpp_data"],
    "model_path": "D:/pycharmprojects/GPT-neo/model_weights",
    "n_ctx": 1024,
    "n_layer": 12,
    "scale_by_depth": true,
    "scale_by_in": false,
    "attention_types" :  [[["global"],12]],
    "activation_function": "gelu",
    "mesh_shape": "all:1",
    "layout": "batch:all",
    "recompute_grad": false,
    "gradient_clipping": 1.0
}

[StellaAthena]

Does the untrained model, loaded the same way, produce coherent output? Typically gibberish like this means the model is being incorrectly loaded.

[BakingBrains]

@StellaAthena The model has loaded the same way, but I don't know where's the problem. I will look into it.

Thanks you