SnappyLoader.loadNativeLibrary may fail due to bugs in URLClassLoader, we should not rely on getResourceAsStream
jizhilong opened this issue · 1 comments
jizhilong commented
The following IOException happens occasionally in our environment.
java.io.IOException: Stream closed
at java.util.zip.InflaterInputStream.ensureOpen(InflaterInputStream.java:67)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:142)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at org.xerial.snappy.SnappyLoader.contentsEquals(SnappyLoader.java:209)
at org.xerial.snappy.SnappyLoader.extractLibraryFile(SnappyLoader.java:278)
at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:355)
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:176)
at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:154)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:435)
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:466)
at java.io.DataInputStream.readByte(DataInputStream.java:265)
at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:168)
at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:292)
at org.apache.kafka.common.record.DefaultRecordBatch$1.readNext(DefaultRecordBatch.java:264)
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563)
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532)
at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.nextFetchedRecord(Fetcher.java:1060)
at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(Fetcher.java:1095)
at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1200(Fetcher.java:949)
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:570)
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:531)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1170)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1103)
at io.opentracing.contrib.kafka.TracingKafkaConsumer.poll(TracingKafkaConsumer.java:103)
at com.leyantech.chaos.kafka.AbstractKafkaConsumerClient.lambda$poll$11(AbstractKafkaConsumerClient.java:512)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.leyantech.chaos.kafka.AbstractKafkaConsumerClient.poll(AbstractKafkaConsumerClient.java:510)
at com.leyantech.chaos.kafka.AbstractKafkaConsumerClient.lambda$start$3(AbstractKafkaConsumerClient.java:277)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
According to the googled result, the exception may be caused by an open JDK BUG JDK-8205976, and JDK-8205976 is also causing issues in Android Gradle Plugin: https://issuetracker.google.com/issues/137929327?pli=1
So I propose to replace SnappyLoader.class.getResourceAsStream
calls in SnappyLoader.java
with the following way:
private static InputStream getResourceInputStream(String resourcePath) throws IOException {
URL url = SnappyLoader.class.getResource(resourcePath);
URLConnection connection = url.openConnection();
if (connection instanceof JarURLConnection) {
JarURLConnection jarConnection = (JarURLConnection) connection;
jarConnection.setUseCaches(false); // workaround for https://bugs.openjdk.org/browse/JDK-8205976
return jarConnection.getInputStream();
} else {
return connection.getInputStream();
}
}
xerial commented
The fix is merged and will be available in the next version snappy-java 1.1.10.0