datonic/datadex

Fix AEMET API assets

Closed this issue · 2 comments

davidgasquez commented

Seems the GitHub CI machine the project has been assigned recently keeps crashing due to AEMET rate limiting the machine itself.

This shouldn't happens, and if it happens, we should handle it better than crashing the entire DAG.

davidgasquez commented

Thinking about creating a PandasHuggingFaceIOManager and figure out it an asset can load itself from there and continue with incremental requests to the API instead of the entire date range.

davidgasquez commented

Issue might be the stability of the servers. I've been getting this even doing manual queries... 😅

image