RosettaCommons/DeepAb

All decoys look the same

larae2011 opened this issue · 3 comments

When I run the code

python predict.py data/sample_files/4h0h.fasta --decoys 5 --renumber

all the predicted decoys are identical, which I think isn't supposed to happen.
I get the following warning when running the code:

/home/venv/lib/python3.7/site-packages/torch/utils/checkpoint.py:25: UserWarning: None of the inputs have 
requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")

Is this the cause for identical decoys or is something else wrong?
I am running this on Ubuntu 18.04 with Python 3.7.9 and I also tried it on Ubuntu 20.04 with Python 3.8.10. I get the same warning and the same issue with decoys on both.

I think the issue might actually reflect a pyrosetta bug in its interaction with tqdm utilities like process_map: the main process has initiated the random number generator with some seed, and then the subprocesses that run refine_fv all start from zero.

While we try to figure out the right invocation to fix this (including pushing a fix upstream to pyrosetta if that ends up necessary) -- could you try passing 1 to num_procs so that the decoys are built serially? This seems like a bit of a waste, but it'll at least be a useful control for you in the meantime.

Thank you for bringing this to my attention! As @everyday847 noted, the issue appears to be due to each individual process starting with the same random PyRosetta initialization when the threads are created. I've mitigated this within the DeepAb codebase by re-initializing PyRosetta within each thread, which should result in unique decoys. However, I would still expect a high degree of similarity between decoys in most cases, as the DeepAb constraints are quite convergent.

I'm closing this for now, but please reopen if the problem remains and I can investigate further.