ASFHyP3/burst2safe

Files not downloading. No XML found when running `get_subxml_from_metadata`

Closed this issue · 5 comments

pbrotoisworo commented

Hello,

I've tried running burst2safe in Google Colab but it fails due to no XML file found. The content directory in Colab is empty.

I've just used the commands in the readme file to install it via pip. Then run a sample CLI command
!burst2safe --orbit 32861 --bbox 53.57 27.54 53.78 27.60 --pols VV VH

It fails in this particular line because no XML file is found:
https://github.com/forrestfwilliams/burst2safe/blob/b8cb864ebfdfe83733d6174b0be67d5aaae89784/src/burst2safe/utils.py#L192

Here are the print statements before it failed:

Using burst group search...
Found 4 burst(s).
Check burst group validity...
Downloading data...
Download complete.
Creating SAFE...

Here is the traceback:

Traceback (most recent call last):
  File "/usr/local/bin/burst2safe", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/burst2safe/burst2safe.py", line 166, in main
    burst2safe(
  File "/usr/local/lib/python3.10/dist-packages/burst2safe/burst2safe.py", line 139, in burst2safe
    [info.add_shape_info() for info in burst_infos]
  File "/usr/local/lib/python3.10/dist-packages/burst2safe/burst2safe.py", line 139, in <listcomp>
    [info.add_shape_info() for info in burst_infos]
  File "/usr/local/lib/python3.10/dist-packages/burst2safe/utils.py", line 41, in add_shape_info
    annotation = get_subxml_from_metadata(self.metadata_path, 'product', self.swath, self.polarization)
  File "/usr/local/lib/python3.10/dist-packages/burst2safe/utils.py", line 192, in get_subxml_from_metadata
    with open(metadata_path, 'r') as metadata_file:
FileNotFoundError: [Errno 2] No such file or directory: '/content/S1A_IW_SLC__1SDV_20200604T022251_20200604T022318_032861_03CE65_7C85_VV.xml'
forrestfwilliams commented

Hey @pbrotoisworo thank you for the issue. Turns out I was running into a race condition when using a ThreadPoolExecutor with asf_search.download_url that resulted in downloads intermittently being skipped. I've open PR #29 to fix the issue.

forrestfwilliams commented

@pbrotoisworo I've released a new version with the fix to PyPI! Could you try the tool again after reinstalling it with pip (python -m pip install burst2safe==0.3.1)?

pbrotoisworo commented

Thanks. Some progress with 0.3.1, but now there is a new error related to authentication. Is there a way to input credentials for ASF?

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/asf_search/download/download.py", line 111, in _try_get_response
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=BO_n7nTIlMljdvU6kRRB3g&response_type=code&redirect_uri=https://auth.asf.alaska.edu/login&state=https://auth.asf.alaska.edu/loginservice/out&app_type=401

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/burst2safe", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/burst2safe/burst2safe.py", line 175, in main
    burst2safe(
  File "/usr/local/lib/python3.10/dist-packages/burst2safe/burst2safe.py", line 144, in burst2safe
    download_bursts(burst_infos)
  File "/usr/local/lib/python3.10/dist-packages/burst2safe/burst2safe.py", line 105, in download_bursts
    asf_search.download_url(urls[0], dirs[0], names[1], session)
  File "/usr/local/lib/python3.10/dist-packages/asf_search/download/download.py", line 74, in download_url
    response = _try_get_response(session=session, url=url)
  File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 314, in iter
    return fut.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/asf_search/download/download.py", line 114, in _try_get_response
    raise ASFAuthenticationError(f'HTTP {e.response.status_code}: {e.response.text}')
asf_search.exceptions.ASFAuthenticationError: HTTP 401: HTTP Basic: Access denied.
forrestfwilliams commented

Ah, yes. You will need to set up an EarthData login account and update your ~/.netrc file to download data. You can sign up for an EarthData login account here, and here are instructions for setting up your ~/.netrc.

pbrotoisworo commented

Ah yeah, I was able to download it. Thanks! I think there might be some other issues I'm running into while running on Colab but I'll experiment more and open another issue if necessary.

  • 👍1