onthegomap/planetiler

[BUG] IOException/GOAWAY happening in parseResults()

Closed this issue · 0 comments

Describe the bug

While generating tiles with --fetch-wikidata following was observed:

...
Caused by: java.io.IOException: closed
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(ResponseSubscribers.java:464)
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:508)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:258)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:2102)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName(UTF8StreamJsonParser.java:2010)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1798)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:1081)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:536)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:100)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:25)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4867)
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3199)
	at com.onthegomap.planetiler.util.Wikidata.parseResults(Wikidata.java:92)
	at com.onthegomap.planetiler.util.Wikidata.queryWikidata(Wikidata.java:379)
	at com.onthegomap.planetiler.util.Wikidata.flush(Wikidata.java:314)
	... 9 more
Caused by: java.io.IOException: /<IP of our server>:33120: GOAWAY received
	at java.net.http/jdk.internal.net.http.Http2Connection.handleGoAway(Http2Connection.java:1166)
...

Error was observed several times roughly a week ago, but then never again so far.

To Reproduce

While not able to reproduce the issue with WikiData server(s) and not willing to pound on them too much I've tried a mock:

  1. Setup a dummy nginx powered WikiData API mock with dummy JSON response implemented (as static file but allowing POST), HTTP/2 enabled and http2_max_requests 25; setting
  2. Adjust Planetiler to use a dummy nginx mock server
  3. Run Planetiler with --fetch-wikidata
  4. See the error

Main problem is, that I failed to trigger the error within parseResults() since I'm not sure about what were the conditions on servers:

  • server running Planetiler, e.g. mainly CPU and network load
  • WikiData server, e.g. again mainly CPU and network load, possibly scheduled or unscheduled reboot, etc.

Expected behavior

Planetiler is handling all IOException by retrying the requests, that is expected behavior.

Screenshots

N/A

Environment (please complete the following information):

  • Hardware: some server
  • OS: Linux/Ubuntu
  • Java version and distribution: 21 shipped by Ubuntu
  • Maven version: N/A (e.g. Planetiler compiled elsewhere)

Additional context

Searching around I found this: https://nuvalence.io/insights/lessons-learned-investigating-goaways-in-grpc-apis/ which may implicate http2_max_requests setting as a factor in this. Based on that I've tried to trigger GOAWAY with nginx. Also query.wikidata.org seems to be running on or behind nginx.