PyNIVAError('upstream request timeout'))

Question

PyNIVAError('upstream request timeout'))

Closed this issue a year ago · 11 comments

calling pyniva with this code

start_time = "2021-01-09T00:00:00"
end_time = "2023-09-01T15:00:00"
paths_to_download = ['OC/FERRYBOX/FLOW/FLOW/RAW'] 
fb_data = get_data(platform_code, paths_to_download, start_time, end_time, noqc = False)

gives PyNIVAError('upstream request timeout'))

Downloading data for OC
OC/FERRYBOX/FLOW/FLOW/RAW
could not download ('OC/FERRYBOX/FLOW/FLOW/RAW', PyNIVAError('upstream request timeout'))
OC/gpstrack
could not download ('OC/gpstrack', PyNIVAError('upstream request timeout'))
Nothing was downloaded

How can we check what exacly is failing?

Answer 1 · 2023-10-02T14:04:20.000Z

I guess this log is related
FetchError: request to http://tsb:5000/?start=2021-01-09T00%3A00%3A00&end=2023-09-01T15%3A00%3A00&noqc=False&dt=0&uuid=48db2bdc-4ce8-4f28-8c94-0e7f98ebcbad failed, reason: socket hang up
at ClientRequest. (/usr/src/app/node_modules/node-fetch/lib/index.js:1501:11)
at ClientRequest.emit (events.js:310:20)
at Socket.socketOnEnd (_http_client.js:453:9)
at Socket.emit (events.js:322:22)
at endReadableNT (_stream_readable.js:1187:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21)

Answer 2 · 2023-10-02T14:22:54.000Z

clearly related to this commit https://github.com/NIVANorge/ms-ferrybox/commit/045d907430e28d4368467384a4c3db959688f999

Answer 3 · 2023-10-02T20:57:25.000Z

clearly related to this commit NIVANorge/ms-ferrybox@045d907

How is that clearly related ? The error comes from tsb, but this commit is in msferrybox.

Answer 4 · 2023-10-03T07:37:08.000Z

clearly related to this commit NIVANorge/ms-ferrybox@045d907

How is that clearly related ? The error comes from tsb, but this commit is in msferrybox.

Well, ok. I think it is related, since the amount of data that one is able to request via pyniva went down at the relatively the same time when these changes were introduced.

Can we test?

Answer 5 · 2023-10-03T10:13:58.000Z

yep, lets do that:)

Answer 6 · 2023-10-03T11:20:10.000Z

One error here

Answer 7 · 2023-10-03T15:10:41.000Z

endpoints https://console.cloud.google.com/logs/query;query=resource.type%3D%22api%22%0Aresource.labels.service%3D%22api.niva.no%22%0Aresource.labels.method%3D%221.api_niva_no.TsbProxy%22;cursorTimestamp=2023-10-03T14:57:02.463969338Z;startTime=2023-10-03T09:06:25.480Z;endTime=2023-10-03T15:06:25.480Z?project=nivaprod-1

Answer 8 · 2023-10-03T15:14:42.000Z

https://cloud.google.com/endpoints/docs/openapi/openapi-extensions#deadline

Answer 9 · 2023-10-03T15:51:02.000Z

Looks ESPv2 uses a default deadline of 15s, I have increased it in 1bb5746 to 120s. So now

from pyniva import token2header

url="https://ferrybox-api.niva.no/v1/tsb/"
res = requests.get(f"{url}?start=2022-02-01T00%3A00%3A00&end=2023-04-01T16%3A00%3A00&noqc=False&dt=0&uuid=4d9ff393-25a3-47b8-aaf1-8fbbccfec3c3", headers=token2header("niva-service-account.json"))
res

that takes about 46s works.

As a side note @lisapro, I suggest that the get_data functions you create does a check for first and last timestamps to see if the time window can be decreased.

Answer 10 · 2023-10-03T15:58:08.000Z

Nice!

By reducing time Window - do you mean splitting the request into shorter time windows?

Answer 11 · 2023-10-03T20:25:22.000Z

Was thinking about first finding the start and end time before running the query, think it is using a bit long time on requests that do not return anything. Yes using some smaller windows is also a good idea 💡but first you need the start and end