address.db.gz and streets.db.gz are not accessable any more from https://s3.amazonaws.com
Closed this issue · 12 comments
Hey team!
I was using your awesome geocoding engine when I noticed something interesting.
Let me tell you more about it.
When I'm trying to run the interpolation engine it fails to start b/c https://s3.amazonaws.com/pelias-data.nextzen.org/interpolation/current/street.db.gz and https://s3.amazonaws.com/pelias-data.nextzen.org/interpolation/current/address.db.gz are not accessible any more.
wget -O - https://s3.amazonaws.com/pelias-data.nextzen.org/interpolation/current/street.db.gzwget -O - https://s3.amazonaws.com/pelias-data.nextzen.org/interpolation/current/address.db.gz
Both returns: Error 403: Forbidden.
Is there any alternative? B/c pretty important service stopped to work. :(
I'm not sure what's going on with that bucket, I'll ask the nextzen admins, they have been sponsoring the significant download costs of large files like this for about 2 years now, and since they're running on donations there may be a financial cause.
If you're relying on the community resources for a production system I'd suggest you reach out to us at geocode.earth for a data subscription/support agreement to prevent this sort of thing from happening again.
In the mean-time, those nextzen versions haven't changed for over a year so you can use your previously downloaded copies:
2018-04-16 00:58:51 16.9 GiB address.db.gz
2018-06-18 12:18:34 3.0 GiB street.db.gz
@missinglink Hi, and thank you for the response. I would use these files if I'd backed up it. Unfortunately, I figure outed that they are downloading on each interpolation restart on last week. So I do not have any copies. It would be nice to download it again and I will make backup this time.
If there is interest, we could probably offer some sort of public hosting of these files ... (not on AWS though)
@adelcasse Hello, and nice to meet you.
Any way will work for me if I will be able to download it(manually). :)
Thanks @adelcasse, please send me an email, and yes, $1.50 per download on AWS adds up quickly at scale ;)
@missinglink not sure how to find your email address
Oh sorry, too cautious of bots, it's peter at geocode.earth 😄
Done, you can edit your post if you don't want your email to be displayed ;)
@missinglink @adelcasse Sorry if I bother you, could you tel me, did you upload global street and addresses to any alternative hosting?
Hi @semeyon, w're generating new global files (we only use european ones for our internal use cases in production, and it's taking a very long time !) and try to test them before putting them on a public endpoint. If someone has the original one, we can eventually take them "as is" and host them on our servers directly.
Anyone with the original addresse.db/streets.db so that we could take them directly and allow downloading ?
Hi @adelcasse problem that I need same files :) and I can host it myself. I believe(but not sure) nextzen has this files on their s3 bucket, but they revoke access. I wrote an email to nextzen but they did not answer to me. MAy @missinglink knows how to contact them?
Ps. And, yes. It's quite hard to build our own especially b/c I'm super busy with other stuff. :(
To be clear, the files are no longer available for free public download.
The project is open-source but the 'free' bit doesn't refer to our time and the cost of resources used to generate & host files.
I have spoken with the nextzen team and they mentioned your email.
I'm also very busy refactoring the codebase to support elasticsearch 6 & 7.
If you'd like the files you can contact Geocode Earth but there will be an associated cost to cover the financial costs of generating & testing the data.