wandb error in running main_instance_segmentation

Question

wandb error in running main_instance_segmentation

Opened this issue 6 months ago · 3 comments

Hi! I am getting wandb error while run main_instance_segmentation script. Any suggestion would be appreciated. Thanks!

/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/MinkowskiEngine-0.5.4-py3.10-linux-x86_64.egg/MinkowskiEngine/init.py:36: UserWarning: The environment variable OMP_NUM_THREADS not set. MinkowskiEngine will automatically set OMP_NUM_THREADS=16. If you want to set OMP_NUM_THREADS manually, please export it on the command line before running a python script. e.g. export OMP_NUM_THREADS=12; python your_program.py. It is recommended to set it below 24.
warnings.warn(
/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/utilities/seed.py:55: UserWarning: No seed found, seed set to 3840629487
rank_zero_warn(f"No seed found, seed set to {seed}")
Global seed set to 3840629487
EXPERIMENT ALREADY EXIST
{'target': 'pytorch_lightning.loggers.WandbLogger', 'project': '${general.project_name}', 'name': '${general.experiment_name}', 'save_dir': '${general.save_dir}', 'entity': 'schult', 'resume': 'allow', 'id': '${general.experiment_name}'}
wandb: Currently logged in as: ntakhtkeshha. Use wandb login --relogin to force relogin
wandb: ERROR Error while calling W&B API: permission denied (<Response [403]>)
Problem at: /home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/wandb.py 362 experiment
Traceback (most recent call last):
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/utils.py", line 63, in call
return _instantiate_class(type_or_callable, config, *args, **kwargs)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
return clazz(*args, **final_kwargs)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/wandb.py", line 316, in init
_ = self.experiment
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/logger.py", line 54, in experiment
return get_experiment() or DummyExperiment()
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/utilities/rank_zero.py", line 32, in wrapped_fn
return fn(*args, **kwargs)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/logger.py", line 52, in get_experiment
return fn(self)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/wandb.py", line 362, in experiment
self._experiment = wandb.init(**self._wandb_init)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1164, in init
raise e
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1145, in init
run = wi.init()
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 764, in init
raise error
wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
return func()
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 347, in
lambda: hydra.run(
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 107, in run
return run_job(
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/core/utils.py", line 128, in run_job
ret.return_value = task_function(task_cfg)
File "/home/ntakhtke/DL models/Mask3D-main/main_instance_segmentation.py", line 108, in main
train(cfg)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/main.py", line 27, in decorated_main
return task_function(cfg_passthrough)
File "/home/ntakhtke/DL models/Mask3D-main/main_instance_segmentation.py", line 70, in train
cfg, model, loggers = get_parameters(cfg)
File "/home/ntakhtke/DL models/Mask3D-main/main_instance_segmentation.py", line 48, in get_parameters
loggers.append(hydra.utils.instantiate(log))
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/utils.py", line 70, in call
raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'pytorch_lightning.loggers.WandbLogger' : It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ntakhtke/DL models/Mask3D-main/main_instance_segmentation.py", line 114, in
main()
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/main.py", line 32, in decorated_main
_run_hydra(
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
run_and_report(
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 267, in run_and_report
print_exception(etype=None, value=ex, tb=final_tb) # type: ignore
TypeError: print_exception() got an unexpected keyword argument 'etype'
Traceback (most recent call last):
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/utils.py", line 63, in call
return _instantiate_class(type_or_callable, config, *args, **kwargs)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
return clazz(*args, **final_kwargs)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/wandb.py", line 316, in init
_ = self.experiment
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/logger.py", line 54, in experiment
return get_experiment() or DummyExperiment()
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/utilities/rank_zero.py", line 32, in wrapped_fn
return fn(*args, **kwargs)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/logger.py", line 52, in get_experiment
return fn(self)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/wandb.py", line 362, in experiment
self._experiment = wandb.init(**self._wandb_init)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1164, in init
raise e
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1145, in init
run = wi.init()
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 764, in init
raise error
wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
return func()
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 347, in
lambda: hydra.run(
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 107, in run
return run_job(
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/core/utils.py", line 128, in run_job
ret.return_value = task_function(task_cfg)
File "/home/ntakhtke/DL models/Mask3D-main/main_instance_segmentation.py", line 108, in main
train(cfg)
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/main.py", line 27, in decorated_main
return task_function(cfg_passthrough)
File "/home/ntakhtke/DL models/Mask3D-main/main_instance_segmentation.py", line 70, in train
cfg, model, loggers = get_parameters(cfg)
File "/home/ntakhtke/DL models/Mask3D-main/main_instance_segmentation.py", line 48, in get_parameters
loggers.append(hydra.utils.instantiate(log))
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/utils.py", line 70, in call
raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'pytorch_lightning.loggers.WandbLogger' : It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ntakhtke/DL models/Mask3D-main/main_instance_segmentation.py", line 114, in
main()
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/main.py", line 32, in decorated_main
_run_hydra(
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
run_and_report(
File "/home/ntakhtke/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 267, in run_and_report
print_exception(etype=None, value=ex, tb=final_tb) # type: ignore
TypeError: print_exception() got an unexpected keyword argument 'etype'

Answer 1 · 2024-04-03T19:48:14.000Z

I re-installled everything several times, but still I am getting wandb error!

Error:
ntakhtke  ❲c❳ mask3d_v1  ~/DL models2/Mask3D-main  python main_instance_segmentation.py general.checkpoint=/pretrained_checkpoints/area6_from_scratch.ckpt general.train_mode=false
/home/ntakhtke/DL models2/Mask3D-main/benchmark/evaluate_semantic_instance.py:43: UserWarning: A NumPy version >=1.22.4 and <1.29.0 is required for this version of SciPy (detected version 1.22.3)
from scipy import stats
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/MinkowskiEngine-0.5.4-py3.10-linux-x86_64.egg/MinkowskiEngine/init.py:36: UserWarning: The environment variable OMP_NUM_THREADS not set. MinkowskiEngine will automatically set OMP_NUM_THREADS=16. If you want to set OMP_NUM_THREADS manually, please export it on the command line before running a python script. e.g. export OMP_NUM_THREADS=12; python your_program.py. It is recommended to set it below 24.
warnings.warn(
/home/ntakhtke/DL models2/Mask3D-main/main_instance_segmentation.py:65: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(
/home/ntakhtke/DL models2/Mask3D-main/main_instance_segmentation.py:87: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(
/home/ntakhtke/DL models2/Mask3D-main/main_instance_segmentation.py:103: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config_base_instance_segmentation.yaml': Defaults list is missing _self_. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'loss/set_criterion': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'matcher/hungarian_matcher': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'callbacks/callbacks_instance_segmentation': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'trainer/trainer600': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'scheduler/onecyclelr': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'optimizer/adamw': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'metrics/miou': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'model/mask3d': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'logging/full': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/core/default_element.py:124: UserWarning: In 'data/indoor': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
/home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/pytorch_lightning/utilities/seed.py:55: UserWarning: No seed found, seed set to 2669315711
rank_zero_warn(f"No seed found, seed set to {seed}")
Global seed set to 2669315711
EXPERIMENT ALREADY EXIST
{'target': 'pytorch_lightning.loggers.WandbLogger', 'project': '${general.project_name}', 'name': '${general.experiment_name}', 'save_dir': '${general.save_dir}', 'entity': 'schult', 'resume': 'allow', 'id': '${general.experiment_name}'}
wandb: Currently logged in as: narges. Use wandb login --relogin to force relogin
wandb: ERROR Error while calling W&B API: permission denied (<Response [403]>)
Problem at: /home/ntakhtke/anaconda3/envs/mask3d_v1/lib/python3.10/site-packages/pytorch_lightning/loggers/wandb.py 362 experiment
Error executing job with overrides: ['general.checkpoint=/pretrained_checkpoints/area6_from_scratch.ckpt', 'general.train_mode=false']
Error in call to target 'pytorch_lightning.loggers.wandb.WandbLogger':
CommError('It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)')
full_key: logging0

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Answer 2 · 2024-04-25T17:15:05.000Z

I have encountered the same problem. Have you solved it?
I would appreciate it if you could reply to me.

Answer 3 · 2024-04-25T22:26:51.000Z

I have encountered the same problem. Have you solved it? I would appreciate it if you could reply to me.

Hi! Hope the following helps:
export WANDB_MODE=offline
export WANDB_MODE=disabled