Data Landing Too Late
Gizem-9395 opened this issue · 2 comments
Hello, I am trying to get data using the code below, but the data does not download immediately. When I checked the CDS Your requests page, I saw that even 2.5 KB of data was downloaded in about 4 hours. I was able to receive data in a very short time with this code before, what could be the reason?
request <- list(
- product_type = "monthly_averaged_reanalysis_by_hour_of_day",
- variable = "total_column_water_vapour",
- year = "2010",
- month = "01",
- time = c("00:00", "12:00"),
- area = c(41.47, 39.14, 37.47, 43.14),
- format = "netcdf",
- dataset_short_name = "reanalysis-era5-single-levels-monthly-means",
- target = "download.nc"
- )
file <- wf_request(user = "10765x",
request = request,
transfer = TRUE,
path = ".")
Requesting data to the cds service with username 10765x
-
staging data transfer at url endpoint or request id:
19dc32f2-686c-4888-8255-60ddfbb36699 -
timeout set to 1.0 hours
Hi @Gizem-9395
Downloads have no set time to complete. Speed depends on the load on the system and the buffers or spun up disks in the data center. For certain products things even come from high density tapes if I'm not mistaken (which is particularly slow).
This is why the transfer handle is reported upon time-out. You can then fetch the data using wf_transfer() and the download handle.
If things are slow it helps to pick a time when requests are less likely to be flooding the system (weekends / nights in EU). Also consider optimizing your code to query one big chunk rather than multiple small ones (as one failed request like this might mess up your routine) - this does assume that you can store a bulk download.
Finally, check the CDS website to see if there isn't any planned maintenance ongoing or other things that slow down the downloads.
thank you :)