cannot see id 1141909453 in spelunker
missinglink opened this issue · 14 comments
heya, I'm getting a 404 error when trying to load 1141909453 in the spelunker
also 1141907907
also 1141906837
Looks like these are all related to same PRs:
- https://github.com/whosonfirst-data/whosonfirst-data/commits/master/data/114/190/945/3/1141909453.geojson
- https://github.com/whosonfirst-data/whosonfirst-data/commits/master/data/114/190/790/7
- https://github.com/whosonfirst-data/whosonfirst-data/commits/master/data/114/190/683/7
Over to @thisisaaronland to investigate why the features are on Github but not on S3 or in ES for the Spelunker.
Which PR is this? There was one recently that triggered a gazillion updates which makes updated (well Redis, really) sad:
The data appears to have been replicated everywhere except ES so that narrows it down a bit...
- update all the things (again) per PR #763
- update all the things per PR #876
The links above are for the history of those files. Not sure why the PR numbers aren't auto linking.
Something is causing the ES indexing process (or at least certain batches) to fail to index. Still investigating...
Something about these records makes ES unhappy...
/usr/local/bin/wof-es-index-files --index spelunker --host 127.0.0.1 --verbose ./data/114/190/790/7/1141907907.geojson
...
DEBUG:urllib3.util.retry:Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1
DEBUG:urllib3.connectionpool:http://127.0.0.1:9200 "POST /spelunker/locality/1141907907 HTTP/1.1" 400 279
DEBUG:mapzen.whosonfirst.elasticsearch:Finished call to 'mapzen.whosonfirst.elasticsearch.do_index' after 0.019(s), this was the 1st time calling it.
DEBUG:urllib3.util.retry:Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1
DEBUG:urllib3.connectionpool:http://127.0.0.1:9200 "POST /spelunker/locality/1141907907 HTTP/1.1" 400 279
DEBUG:mapzen.whosonfirst.elasticsearch:Finished call to 'mapzen.whosonfirst.elasticsearch.do_index' after 5.029(s), this was the 2nd time calling it.
DEBUG:urllib3.util.retry:Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1
DEBUG:urllib3.connectionpool:http://127.0.0.1:9200 "POST /spelunker/locality/1141907907 HTTP/1.1" 400 279
DEBUG:mapzen.whosonfirst.elasticsearch:Finished call to 'mapzen.whosonfirst.elasticsearch.do_index' after 10.038(s), this was the 3rd time calling it.
DEBUG:urllib3.util.retry:Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1
DEBUG:urllib3.connectionpool:http://127.0.0.1:9200 "POST /spelunker/locality/1141907907 HTTP/1.1" 400 279
DEBUG:mapzen.whosonfirst.elasticsearch:Finished call to 'mapzen.whosonfirst.elasticsearch.do_index' after 15.046(s), this was the 4th time calling it.
DEBUG:urllib3.util.retry:Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1
DEBUG:urllib3.connectionpool:http://127.0.0.1:9200 "POST /spelunker/locality/1141907907 HTTP/1.1" 400 279
DEBUG:mapzen.whosonfirst.elasticsearch:Finished call to 'mapzen.whosonfirst.elasticsearch.do_index' after 20.055(s), this was the 5th time calling it.
ERROR:root:failed to index http://127.0.0.1:9200/spelunker/locality/1141907907: RetryError[<Future at 0x7f5e3fdd5350 state=finished raised Exception>]
This also happens for 1141909453
I hhhhhhhhhhaaaaaaaaaaattttttttteeeeeeessssssssss you Elasticsearch, I hates you...
'{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse [ne:MAX_POP10]"}],"type":"mapper_parsing_exception","reason":"failed to parse \
[ne:MAX_POP10]","caused_by":{"type":"number_format_exception","reason":"For input string: \\"0.0\\""}},"status":400}'
because we can't have nice things...
whosonfirst/py-mapzen-whosonfirst-search@f7deaf3#diff-c1683b61159fb490d1baf4bc31d42bb0R410
Hi @thisisaaronland, @stepps00, this bug seems to have returned :(
Is it possible that the spelunker hasn't been synced since that record was added in November?
I'll take a look shortly.
Looks like a portion of the GeoNames locality imports have not made their way into the Spelunker yet. I've also opened #141 to track.
Got caught by the same issue again today :( https://spelunker.whosonfirst.org/id/1327145573/
That ID is found in my sqlite database dated 12 Feb but not found in the Spelunker.