Cannot download naciscdn.org data from naturalearthdata.com

Question

Cannot download naciscdn.org data from naturalearthdata.com

Closed this issue 3 years ago · 21 comments

yoheym commented 3 years ago

Hi all. Any help would be greatly appreciated.

When I attempt to download https://naciscdn.org/naturalearth/110m/physical/ne_110m_ocean.zip, , the request times out.

Answer 1 · 2021-09-04T02:18:53.000Z

I’m working to repoint the naciscdn.org domain at the new S3 buckets this weekend.

…

On Sep 3, 2021, at 16:48, yoheym ***@***.***> wrote: Hi all. Any help would be greatly appreciated. When I attempt to download https://naciscdn.org/naturalearth/110m/physical/ne_110m_ocean.zip, , the request times out. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 2 · 2021-09-04T20:37:36.000Z

I assume you're still working on this because I can't get to https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/raster/OB_LR.zip
Responds with This site can’t be reached naciscdn.org took too long to respond.
Any ETA for the repoint???

Answer 3 · 2021-09-06T11:56:06.000Z

Hello, are there any news on the naciscdn.org outage?

Answer 4 · 2021-09-07T04:51:06.000Z

The original private server backing naciscdn.org seems to be busted.

I'm in the process of redirecting to S3. Here are some file resource patterns in the meantime:

Official S3 public data: (EVERYONE SHOULD MIGRATE HERE, ESPECIALLY IF YOU USE CI)
https://naturalearth.s3.amazonaws.com/10m_cultural/10m_cultural.zip
(note 10m_cultural where naturalearthdata.com uses 10m/cultural with / not _ so you'll need to update more than your domain)

Eventually this will get naciscdn.org pretty domain:
https://s3.us-west-2.amazonaws.com/naciscdn.org/naturalearth/10m/cultural/10m_cultural.zip
(same file structure as naturalearthdata.com)

Answer 5 · 2021-09-07T22:42:12.000Z

Is there a way to download a raster file with an AWS link?

Answer 6 · 2021-09-08T06:04:59.000Z

Oops, I missed the rasters the first attempt, thanks for flagging that. Adding the 10m and 50m rasters now.

Answer 7 · 2021-09-08T15:29:13.000Z

@dwnadler The rasters should all be there now.

Answer 8 · 2021-09-08T17:59:28.000Z

Is this still down? I currently can't access the server at naciscdn.org. Thanks :)

Answer 9 · 2021-09-08T18:18:02.000Z

FWIW I downloaded data from the S3 URL using geopandas
gpd.read_file("https://naturalearth.s3.amazonaws.com/10m_cultural/ne_10m_populated_places.zip")

Answer 10 · 2021-09-08T18:59:56.000Z

I was able to download https://naturalearth.s3.amazonaws.com/10m_physical/ne_10m_bathymetry_all.zip by going to https://naturalearth.s3.amazonaws.com/ and going through the file listings until I found 10m_physical/ne_10m_bathymetry_all.zip. Concatenated the two strings in a link and used that to download. Tedious but effective. *Lloyd DeKay*

…

On Wed, Sep 8, 2021 at 11:18 AM Ray Bell ***@***.***> wrote: FWIW I downloaded data from the S3 URL using geopandas gpd.read_file(" https://naturalearth.s3.amazonaws.com/10m_cultural/ne_10m_populated_places.zip ") — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#581 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJDMH24CIYQAKR6IEQSQ2U3UA6SGLANCNFSM5DMUS57Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Answer 11 · 2021-09-09T16:24:51.000Z

If you need to find one of these links and made it this far, here they are in a gist, until the site is restored. Taken from the index at https://naturalearth.s3.amazonaws.com/

https://gist.github.com/DanielJWood/b71237cc200831acf8e637c05ce2c375

Answer 12 · 2021-09-09T16:40:41.000Z

Daniel, I wanted to say thank you for your hard work on getting this database back up and running. It looks like it's been a fairly monumental task so far, but it's a valuable database that the public needs, nd your efforts are key to keeping it available to the public. I just wanted to let you know how much you're appreciated. Cheers, *Lloyd DeKay*

…

On Thu, Sep 9, 2021 at 9:25 AM Daniel Wood ***@***.***> wrote: If you need to find one of these links and made it this far, here they are in a gist, until the site is restored. Taken from the index at https://naturalearth.s3.amazonaws.com/ https://gist.github.com/DanielJWood/b71237cc200831acf8e637c05ce2c375 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#581 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJDMH2Y2QWHZWU3NVZQRBV3UBDNV5ANCNFSM5DMUS57Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Answer 13 · 2021-09-09T22:55:03.000Z

Downloaded the files successfully from https://naturalearth.s3.amazonaws.com/10m_physical/ne_10m_coastline.zip - thank you!

Answer 14 · 2021-09-12T19:17:07.000Z

Just wanted to say thanks for rehosting it on https://naturalearth.s3.amazonaws.com/
The same url structure worked for me to download the files I needed for cartopy.
Appreciate it!

Answer 15 · 2021-09-20T12:05:20.000Z

Hey!
First of all thanks for sharing the downloads! I have all of the data saved locally now, but cartopy still wants to download the packages from naciscdn.org. How do I tell the programm to use the local files instead of trying to download?
I tried changing the config with cartopy.config['pre_existing_data_dir']=expanduser('~/cartopy-data') but it still won't work. Might be a stupid question, but help would be appreciated!

Julia

Answer 16 · 2021-09-20T12:36:32.000Z

@https://github.com/julia-remke I'm not a cartopy user, but SciTools/cartopy#1833 and SciTools/cartopy#1849 suggests that a new release of cartopy (v0.20.0) should address this. If you're using that and it still doesn't work, that's probably something to raise at https://github.com/SciTools/cartopy/issues .

Answer 17 · 2021-09-24T05:08:00.000Z

@julia-remke I'm in the same situation. It still seems to be pulling from the non-AWS site and I'm not sure how to change that.

Answer 18 · 2021-09-25T13:04:58.000Z

@ttruksa I ended up just downloading them manually and putting them into cartopy's offline data. ...You may also need the cartopy offline data package as well. (https://github.com/willirath/cartopy_offlinedata) Not sure, as I ended up going with folium...

>>> import cartopy.config
>>> cartopy.config['data_dir']
'/home/pelson/.local/share/cartopy'

Answer 19 · 2021-10-05T21:31:06.000Z

The http version of the naciscdn.org links should now work again.

Next up the https version (which is used by naturalearthdata.com download redirects).

Answer 20 · 2021-10-05T23:08:04.000Z

The https versions should also now work again, meaning downloads from naturalearthdata.com should be operational again.

Answer 21 · 2021-10-05T23:11:11.000Z

For posterities sake, the nameservers configs were scrambled / not auto generated correctly. Manually setting them to the same 4 values in all the places corrected the problem, and unblocked adding the SSL cert.