Invaild data link for IXI Dataset
Closed this issue · 2 comments
KumoLiu commented
[2024-08-24T23:11:39.897Z] Running ./3d_regression/densenet_training_array.ipynb
[2024-08-24T23:11:39.897Z] Checking PEP8 compliance...
[2024-08-24T23:11:40.457Z] Running notebook...
[2024-08-24T23:11:40.457Z] Before:
[2024-08-24T23:11:40.457Z] "max_epochs = 5\n",
[2024-08-24T23:11:40.713Z] After:
[2024-08-24T23:11:40.713Z] "max_epochs = 1\n",
[2024-08-24T23:11:40.713Z] Before:
[2024-08-24T23:11:40.713Z] "val_interval = 2\n",
[2024-08-24T23:11:40.713Z] After:
[2024-08-24T23:11:40.713Z] "val_interval = 1\n",
[2024-08-24T23:11:43.978Z] /usr/local/lib/python3.10/dist-packages/ignite/handlers/checkpoint.py:17: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
[2024-08-24T23:11:43.978Z] from torch.distributed.optim import ZeroRedundancyOptimizer
[2024-08-24T23:11:48.132Z] MONAI version: 1.4.0rc2+2.ga5fbe716
[2024-08-24T23:11:48.132Z] Numpy version: 1.24.4
[2024-08-24T23:11:48.132Z] Pytorch version: 2.4.0+cu121
[2024-08-24T23:11:48.132Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2024-08-24T23:11:48.132Z] MONAI rev id: a5fbe716378948630783deef8ee435e7e3bdc918
[2024-08-24T23:11:48.132Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2024-08-24T23:11:48.132Z]
[2024-08-24T23:11:48.132Z] Optional dependencies:
[2024-08-24T23:11:48.132Z] Pytorch Ignite version: 0.4.11
[2024-08-24T23:11:48.132Z] ITK version: 5.4.0
[2024-08-24T23:11:48.132Z] Nibabel version: 5.2.1
[2024-08-24T23:11:48.132Z] scikit-image version: 0.23.2
[2024-08-24T23:11:48.132Z] scipy version: 1.12.0
[2024-08-24T23:11:48.132Z] Pillow version: 10.2.0
[2024-08-24T23:11:48.132Z] Tensorboard version: 2.17.1
[2024-08-24T23:11:48.132Z] gdown version: 5.2.0
[2024-08-24T23:11:48.132Z] TorchVision version: 0.19.0+cu121
[2024-08-24T23:11:48.132Z] tqdm version: 4.66.2
[2024-08-24T23:11:48.132Z] lmdb version: 1.5.1
[2024-08-24T23:11:48.132Z] psutil version: 5.9.4
[2024-08-24T23:11:48.132Z] pandas version: 1.5.3
[2024-08-24T23:11:48.132Z] einops version: 0.7.0
[2024-08-24T23:11:48.132Z] transformers version: 4.40.2
[2024-08-24T23:11:48.132Z] mlflow version: 2.15.1
[2024-08-24T23:11:48.132Z] pynrrd version: 1.0.0
[2024-08-24T23:11:48.132Z] clearml version: 1.16.3
[2024-08-24T23:11:48.132Z]
[2024-08-24T23:11:48.132Z] For details about installing the optional dependencies, please visit:
[2024-08-24T23:11:48.132Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2024-08-24T23:11:48.132Z]
[2024-08-24T23:11:50.016Z] papermill --progress-bar --log-output -k python3
[2024-08-24T23:11:50.016Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
[2024-08-24T23:11:50.016Z] warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")
[2024-08-24T23:12:26.667Z]
Executing: 0%| | 0/15 [00:00<?, ?cell/s]
Executing: 7%|▋ | 1/15 [00:00<00:12, 1.17cell/s]
Executing: 20%|██ | 3/15 [00:09<00:39, 3.33s/cell]
Executing: 33%|███▎ | 5/15 [00:16<00:33, 3.37s/cell]
Executing: 60%|██████ | 9/15 [00:33<00:24, 4.01s/cell]
Executing: 60%|██████ | 9/15 [00:36<00:24, 4.02s/cell]
[2024-08-24T23:12:26.667Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
[2024-08-24T23:12:26.667Z] warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")
[2024-08-24T23:12:26.667Z] Traceback (most recent call last):
[2024-08-24T23:12:26.667Z] File "/usr/local/bin/papermill", line 8, in <module>
[2024-08-24T23:12:26.667Z] sys.exit(papermill())
[2024-08-24T23:12:26.667Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
[2024-08-24T23:12:26.667Z] return self.main(*args, **kwargs)
[2024-08-24T23:12:26.667Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
[2024-08-24T23:12:26.667Z] rv = self.invoke(ctx)
[2024-08-24T23:12:26.667Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
[2024-08-24T23:12:26.667Z] return ctx.invoke(self.callback, **ctx.params)
[2024-08-24T23:12:26.667Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
[2024-08-24T23:12:26.667Z] return __callback(*args, **kwargs)
[2024-08-24T23:12:26.667Z] File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
[2024-08-24T23:12:26.667Z] return f(get_current_context(), *args, **kwargs)
[2024-08-24T23:12:26.667Z] File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 235, in papermill
[2024-08-24T23:12:26.667Z] execute_notebook(
[2024-08-24T23:12:26.667Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 131, in execute_notebook
[2024-08-24T23:12:26.667Z] raise_for_execution_errors(nb, output_path)
[2024-08-24T23:12:26.667Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 251, in raise_for_execution_errors
[2024-08-24T23:12:26.667Z] raise error
[2024-08-24T23:12:26.667Z] papermill.exceptions.PapermillExecutionError:
[2024-08-24T23:12:26.667Z] ---------------------------------------------------------------------------
[2024-08-24T23:12:26.667Z] Exception encountered at "In [5]":
[2024-08-24T23:12:26.667Z] ---------------------------------------------------------------------------
[2024-08-24T23:12:26.667Z] HTTPError Traceback (most recent call last)
[2024-08-24T23:12:26.667Z] Cell In[5], line 8
[2024-08-24T23:12:26.667Z] 5 dataset_dir = os.path.join(root_dir, "ixi")
[2024-08-24T23:12:26.667Z] 6 tarfile_name = f"{dataset_dir}.tar"
[2024-08-24T23:12:26.667Z] ----> 8 download_and_extract(resource, tarfile_name, dataset_dir, md5)
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:335, in download_and_extract(url, filepath, output_dir, hash_val, hash_type, file_type, has_base, progress)
[2024-08-24T23:12:26.667Z] 333 with tempfile.TemporaryDirectory() as tmp_dir:
[2024-08-24T23:12:26.667Z] 334 filename = filepath or Path(tmp_dir, _basename(url)).resolve()
[2024-08-24T23:12:26.667Z] --> 335 download_url(url=url, filepath=filename, hash_val=hash_val, hash_type=hash_type, progress=progress)
[2024-08-24T23:12:26.667Z] 336 extractall(filepath=filename, output_dir=output_dir, file_type=file_type, has_base=has_base)
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:226, in download_url(url, filepath, hash_val, hash_type, progress, **gdown_kwargs)
[2024-08-24T23:12:26.667Z] 221 raise RuntimeError(
[2024-08-24T23:12:26.667Z] 222 f"Download of file from {download_url}, received from {url} "
[2024-08-24T23:12:26.667Z] 223 + f" to {filepath} failed due to network issue or denied permission."
[2024-08-24T23:12:26.667Z] 224 )
[2024-08-24T23:12:26.667Z] 225 else:
[2024-08-24T23:12:26.667Z] --> 226 _download_with_progress(url, tmp_name, progress=progress)
[2024-08-24T23:12:26.667Z] 227 if not tmp_name.exists():
[2024-08-24T23:12:26.667Z] 228 raise RuntimeError(
[2024-08-24T23:12:26.667Z] 229 f"Download of file from {url} to {filepath} failed due to network issue or denied permission."
[2024-08-24T23:12:26.667Z] 230 )
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:119, in _download_with_progress(url, filepath, progress)
[2024-08-24T23:12:26.667Z] 117 except (URLError, HTTPError, ContentTooShortError, OSError) as e:
[2024-08-24T23:12:26.667Z] 118 logger.error(f"Download failed from {url} to {filepath}.")
[2024-08-24T23:12:26.667Z] --> 119 raise e
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:112, in _download_with_progress(url, filepath, progress)
[2024-08-24T23:12:26.667Z] 109 self.update(b * bsize - self.n) # will also set self.n = b * bsize
[2024-08-24T23:12:26.667Z] 111 with TqdmUpTo(unit="B", unit_scale=True, unit_divisor=1024, miniters=1, desc=_basename(filepath)) as t:
[2024-08-24T23:12:26.667Z] --> 112 urlretrieve(url, filepath, reporthook=t.update_to)
[2024-08-24T23:12:26.667Z] 113 else:
[2024-08-24T23:12:26.667Z] 114 if not has_tqdm and progress:
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:241, in urlretrieve(url, filename, reporthook, data)
[2024-08-24T23:12:26.667Z] 224 """
[2024-08-24T23:12:26.667Z] 225 Retrieve a URL into a temporary location on disk.
[2024-08-24T23:12:26.667Z] 226
[2024-08-24T23:12:26.667Z] (...)
[2024-08-24T23:12:26.667Z] 237 data file as well as the resulting HTTPMessage object.
[2024-08-24T23:12:26.667Z] 238 """
[2024-08-24T23:12:26.667Z] 239 url_type, path = _splittype(url)
[2024-08-24T23:12:26.667Z] --> 241 with contextlib.closing(urlopen(url, data)) as fp:
[2024-08-24T23:12:26.667Z] 242 headers = fp.info()
[2024-08-24T23:12:26.667Z] 244 # Just return the local path and the "headers" for file://
[2024-08-24T23:12:26.667Z] 245 # URLs. No sense in performing a copy unless requested.
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:216, in urlopen(url, data, timeout, cafile, capath, cadefault, context)
[2024-08-24T23:12:26.667Z] 214 else:
[2024-08-24T23:12:26.667Z] 215 opener = _opener
[2024-08-24T23:12:26.667Z] --> 216 return opener.open(url, data, timeout)
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:525, in OpenerDirector.open(self, fullurl, data, timeout)
[2024-08-24T23:12:26.667Z] 523 for processor in self.process_response.get(protocol, []):
[2024-08-24T23:12:26.667Z] 524 meth = getattr(processor, meth_name)
[2024-08-24T23:12:26.667Z] --> 525 response = meth(req, response)
[2024-08-24T23:12:26.667Z] 527 return response
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:634, in HTTPErrorProcessor.http_response(self, request, response)
[2024-08-24T23:12:26.667Z] 631 # According to RFC 2616, "2xx" code indicates that the client's
[2024-08-24T23:12:26.667Z] 632 # request was successfully received, understood, and accepted.
[2024-08-24T23:12:26.667Z] 633 if not (200 <= code < 300):
[2024-08-24T23:12:26.667Z] --> 634 response = self.parent.error(
[2024-08-24T23:12:26.667Z] 635 'http', request, response, code, msg, hdrs)
[2024-08-24T23:12:26.667Z] 637 return response
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:557, in OpenerDirector.error(self, proto, *args)
[2024-08-24T23:12:26.667Z] 555 http_err = 0
[2024-08-24T23:12:26.667Z] 556 args = (dict, proto, meth_name) + args
[2024-08-24T23:12:26.667Z] --> 557 result = self._call_chain(*args)
[2024-08-24T23:12:26.667Z] 558 if result:
[2024-08-24T23:12:26.667Z] 559 return result
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:496, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
[2024-08-24T23:12:26.667Z] 494 for handler in handlers:
[2024-08-24T23:12:26.667Z] 495 func = getattr(handler, meth_name)
[2024-08-24T23:12:26.667Z] --> 496 result = func(*args)
[2024-08-24T23:12:26.667Z] 497 if result is not None:
[2024-08-24T23:12:26.667Z] 498 return result
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:749, in HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)
[2024-08-24T23:12:26.667Z] 746 fp.read()
[2024-08-24T23:12:26.667Z] 747 fp.close()
[2024-08-24T23:12:26.667Z] --> 749 return self.parent.open(new, timeout=req.timeout)
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:525, in OpenerDirector.open(self, fullurl, data, timeout)
[2024-08-24T23:12:26.667Z] 523 for processor in self.process_response.get(protocol, []):
[2024-08-24T23:12:26.667Z] 524 meth = getattr(processor, meth_name)
[2024-08-24T23:12:26.667Z] --> 525 response = meth(req, response)
[2024-08-24T23:12:26.667Z] 527 return response
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:634, in HTTPErrorProcessor.http_response(self, request, response)
[2024-08-24T23:12:26.667Z] 631 # According to RFC 2616, "2xx" code indicates that the client's
[2024-08-24T23:12:26.667Z] 632 # request was successfully received, understood, and accepted.
[2024-08-24T23:12:26.667Z] 633 if not (200 <= code < 300):
[2024-08-24T23:12:26.667Z] --> 634 response = self.parent.error(
[2024-08-24T23:12:26.667Z] 635 'http', request, response, code, msg, hdrs)
[2024-08-24T23:12:26.667Z] 637 return response
[2024-08-24T23:12:26.667Z]
[2024-08-24T23:12:26.668Z] File /usr/lib/python3.10/urllib/request.py:563, in OpenerDirector.error(self, proto, *args)
[2024-08-24T23:12:26.668Z] 561 if http_err:
[2024-08-24T23:12:26.668Z] 562 args = (dict, 'default', 'http_error_default') + orig_args
[2024-08-24T23:12:26.668Z] --> 563 return self._call_chain(*args)
[2024-08-24T23:12:26.668Z]
[2024-08-24T23:12:26.668Z] File /usr/lib/python3.10/urllib/request.py:496, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
[2024-08-24T23:12:26.668Z] 494 for handler in handlers:
[2024-08-24T23:12:26.668Z] 495 func = getattr(handler, meth_name)
[2024-08-24T23:12:26.668Z] --> 496 result = func(*args)
[2024-08-24T23:12:26.668Z] 497 if result is not None:
[2024-08-24T23:12:26.668Z] 498 return result
[2024-08-24T23:12:26.668Z]
[2024-08-24T23:12:26.668Z] File /usr/lib/python3.10/urllib/request.py:643, in HTTPDefaultErrorHandler.http_error_default(self, req, fp, code, msg, hdrs)
[2024-08-24T23:12:26.668Z] 642 def http_error_default(self, req, fp, code, msg, hdrs):
[2024-08-24T23:12:26.668Z] --> 643 raise HTTPError(req.full_url, code, msg, hdrs, fp)
[2024-08-24T23:12:26.668Z]
[2024-08-24T23:12:26.668Z] HTTPError: HTTP Error 403: Forbidden
KumoLiu commented
Also effect
./3d_regression/densenet_training_array.ipynb
./3d_classification/densenet_training_array.ipynb
./modules/public_datasets.ipynb
KumoLiu commented
Seems the owner changed the permission and we can't access the data anymore although the link is still on their website.
Ref: https://brain-development.org/ixi-dataset/
Hi @zephyrie @Nic-Ma, do you think we can make a mirror for this data?