AI21Labs/in-context-ralm

About reproductiveness

Closed this issue · 1 comments

Hello again!

I found that the generation result produced by this repo is highly reproducible and so are the metrics.
However, I couldn't find any place that a seed is set or something like that.
As I'm also doing some experiments on my own using model.generate and hope it could be reproducible as well, could you give some advice on how to make the generation results reproducible?
I've already tried the following but it still gives different generation everytime :(

    import random
    import numpy as np
    import torch
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

I just found that my un-reproducibility comes from the generation config of my model - overriding sample=True into sample=False would make generation same everytime.