[Bug]: Client Error: Unauthorized identity.dataspace.copernicus.eu when scaled

[toandn-jpl]

Checked for duplicates

Yes - I've already checked

Describe the bug

When we triggered slc download historical CSLC processing, most of job-slc_download:2.1.0 are hitting Client Error: Unauthorized identity.dataspace.copernicus.eu.

When retried only a few slc download historical, it seems to work. But once we hit "Unauthorized identity.dataspace.copernicus.eu" hit, all of the download jobs start to fail.

"2023-11-13 20:09:12,434: INFO/main] Querying for Orbit file(s) from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[2023-11-13 20:09:13,611: INFO/select_orbit_file] Evaluating orbit file S1B_OPER_AUX_POEORB_OPOD_20210504T111751_V20210413T225942_20210415T005942.EOF
[2023-11-13 20:09:13,611: INFO/select_orbit_file] Orbit file is suitable for use
[2023-11-13 20:09:14,357: ERROR/_log_giveup] Giving up get_access_token(...) after 1 tries (requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token)
Traceback (most recent call last):
File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 321, in
asyncio.run(run(sys.argv))
File "/opt/conda/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 88, in run
results["download"] = run_download(args, token, es_conn, netloc, username, password, job_id) # return None
File "/home/ops/verdi/ops/opera-pcm/data_subscriber/download.py", line 53, in run_download
download.run_download(args, token, es_conn, netloc, username, password, job_id)
File "/home/ops/verdi/ops/opera-pcm/data_subscriber/download.py", line 121, in run_download
self.perform_download(session, es_conn, downloads, args, token, job_id)
File "/home/ops/verdi/ops/opera-pcm/data_subscriber/asf_download.py", line 95, in perform_download
self.download_orbit_file(new_dataset_dir, product_filepath, additional_metadata)
File "/home/ops/verdi/ops/opera-pcm/data_subscriber/asf_download.py", line 136, in download_orbit_file
stage_orbit_file.main(stage_orbit_file_args)
File "/home/ops/verdi/ops/opera-pcm/tools/stage_orbit_file.py", line 594, in main
access_token = get_access_token(args.auth_endpoint, args.username, args.password)
File "/home/ops/verdi/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
File "/home/ops/verdi/ops/opera-pcm/tools/stage_orbit_file.py", line 473, in get_access_token
response.raise_for_status()
File "/opt/conda/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token"

What did you expect?

I expected [...]

Reproducible steps

1.
2.
3.
...

Environment

- Version of this software [e.g. vX.Y.Z]
- Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
...