OWLv2 Evaluation Code Does Not Work!!
rohit901 opened this issue · 7 comments
Hello,
I'm following the evaluation instructions in https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit#evaluation
running the code, gives me this error:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/projects/owl_vit/evaluator.py", line 52, in <module>
from scenic.projects.owl_vit import models
File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/projects/owl_vit/models.py", line 12, in <module>
from scenic.projects.owl_vit import matching_base_models
File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/projects/owl_vit/matching_base_models.py", line 9, in <module>
from scenic.model_lib import matchers
File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/model_lib/matchers/__init__.py", line 25, in <module>
from scenic.model_lib.matchers.sinkhorn import sinkhorn_matcher
File "/home/rohit.bharadwaj/Projects/RNCDL/scenic/scenic/model_lib/matchers/sinkhorn.py", line 24, in <module>
from ott.geometry import geometry
File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/__init__.py", line 15, in <module>
from . import geometry, initializers, math, problems, solvers, tools, utils
File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/initializers/__init__.py", line 1, in <module>
from . import linear, nn, quadratic
File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/initializers/nn/__init__.py", line 1, in <module>
from . import initializers
File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/initializers/nn/initializers.py", line 21, in <module>
class MetaInitializer(initializers.DefaultInitializer):
File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/ott/initializers/nn/initializers.py", line 64, in MetaInitializer
rng: jax.random.PRNGKeyArray = jax.random.PRNGKey(0),
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rohit.bharadwaj/.conda/envs/scenic/lib/python3.11/site-packages/jax/_src/deprecations.py", line 54, in getattr
raise AttributeError(f"module {module!r} has no attribute {name!r}")
AttributeError: module 'jax.random' has no attribute 'PRNGKeyArray'
I think it maybe because of wrong jax, jaxlib versions? The error originates from the "ott-jax" package.
from the requirements.txt:
ott-jax<0.4.0
so, do we have to downgrade jax, jaxlib? please give the correct jax versions to use to verify the results.
@mjlm could you please help? I think this should be easy version fix, using the current requirements.txt doesn't seem to work :(
we have to use jax, and jaxlib version of 0.4.23 for the evaluation code to work, maybe you can hardcode the version in requirements?
Now getting new error in evaluator.py :/
file scenic/scenic/projects/owl_vit/evaluator.py", line 675, in main
compilation_cache.set_cache_dir('/tmp/jax_compilation_cache')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'jax.experimental.compilation_cache.compilation_cache' has no attribute 'set_cache_dir'
@mjlm could you please help? I think this should be easy version fix, using the current requirements.txt doesn't seem to work :(
I fixed this issue by changing PRNGKeyArray to PRNGKEY in the initializers.py file.
Thanks for the reply, I ended up using one of the custom evaluation code shared earlier here in the issues.
@rohit901 What evaluation code did you use ? when i tried to run the evaluation code using this command
python -m scenic.projects.owl_vit.evaluator --alsologtostderr=true --platform=gpu --config=clip_b32 --checkpoint_path=gs://scenic-bucket/owl_vit/checkpoints/clip_vit_b32_b0203fc --output_dir=/tmp/evaluator
its creating the tfds lvis dataset and its taking very long time >1 day and gets killed. Could you please let me know how to evaluate on LVIS dataset ?
I solved it by changing PRNGKeyArray to PRNGKEY in the initializers.py file.
#1043
You can see the steps there to know where the file is located and what to change.