Stratio/cassandra-lucene-index

BufferUnderflowExceptions when deleting table

smigfu opened this issue · 4 comments

smigfu commented

Hi,

during my unit tests I create a lot of tables with a lucene index applied to it, perform some operations and delete the table afterwards. Quite often but intermittently these test do fail and even impact other tests which don't use the index. This is due to the exception thrown on the server closing active connections on the client side.

Is there a way to more gracefully handle the case in the plugin? I have seen this issue for c* (https://issues.apache.org/jira/browse/CASSANDRA-13868) and tried with a combination of 3.11.1 and a custom build of lucene but the error still persited.

Cheers,
Philipp

ERROR [Native-Transport-Requests-3] 2017-12-06 17:09:51,249 QueryMessage.java:129 - Unexpected error during query
java.nio.BufferUnderflowException: null
	at java.nio.Buffer.nextGetIndex(Buffer.java:506) ~[na:1.8.0_121]
	at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) ~[na:1.8.0_121]
	at org.apache.cassandra.serializers.CollectionSerializer.readCollectionSize(CollectionSerializer.java:79) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.frozenCollectionSize(ColumnsMapper.scala:243) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:191) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:171) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:161) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:119) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.$anonfun$columns$4(ColumnsMapper.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.TraversableOnce.$anonfun$foldRight$1(TraversableOnce.scala:162) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractIterator.foldRight(Iterator.scala:1409) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractIterable.foldRight(Iterable.scala:54) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractTraversable.$colon$bslash(Traversable.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:103) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.$anonfun$columns$3(ColumnsMapper.scala:89) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.TraversableOnce.$anonfun$foldRight$1(TraversableOnce.scala:162) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractIterator.foldRight(Iterator.scala:1409) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractIterable.foldRight(Iterable.scala:54) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractTraversable.$colon$bslash(Traversable.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:87) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:56) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexPostProcessor.document(IndexPostProcessor.scala:141) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexPostProcessor.$anonfun$top$1(IndexPostProcessor.scala:106) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:156) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexPostProcessor.top(IndexPostProcessor.scala:103) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexPostProcessor.process(IndexPostProcessor.scala:57) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.ReadCommandPostProcessor.apply(IndexPostProcessor.scala:168) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.ReadCommandPostProcessor.apply(IndexPostProcessor.scala:161) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at org.apache.cassandra.db.PartitionRangeReadCommand.postReconciliationProcessing(PartitionRangeReadCommand.java:330) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:2284) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.db.PartitionRangeReadCommand.execute(PartitionRangeReadCommand.java:186) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at com.stratio.cassandra.lucene.IndexQueryHandler.executeSortedLuceneQuery(IndexQueryHandler.scala:226) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexQueryHandler.executeLuceneQuery(IndexQueryHandler.scala:193) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexQueryHandler.processStatement(IndexQueryHandler.scala:122) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexQueryHandler.process(IndexQueryHandler.scala:101) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.0.jar:3.11.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.0.jar:3.11.0]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
ERROR [Native-Transport-Requests-3] 2017-12-06 17:09:51,268 ErrorMessage.java:384 - Unexpected exception during request
java.nio.BufferUnderflowException: null
	at java.nio.Buffer.nextGetIndex(Buffer.java:506) ~[na:1.8.0_121]
	at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) ~[na:1.8.0_121]
	at org.apache.cassandra.serializers.CollectionSerializer.readCollectionSize(CollectionSerializer.java:79) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.frozenCollectionSize(ColumnsMapper.scala:243) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:191) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:171) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:161) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:119) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.$anonfun$columns$4(ColumnsMapper.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.TraversableOnce.$anonfun$foldRight$1(TraversableOnce.scala:162) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractIterator.foldRight(Iterator.scala:1409) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractIterable.foldRight(Iterable.scala:54) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractTraversable.$colon$bslash(Traversable.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:103) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.$anonfun$columns$3(ColumnsMapper.scala:89) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.TraversableOnce.$anonfun$foldRight$1(TraversableOnce.scala:162) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractIterator.foldRight(Iterator.scala:1409) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractIterable.foldRight(Iterable.scala:54) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.AbstractTraversable.$colon$bslash(Traversable.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:87) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:56) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexPostProcessor.document(IndexPostProcessor.scala:141) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexPostProcessor.$anonfun$top$1(IndexPostProcessor.scala:106) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:156) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexPostProcessor.top(IndexPostProcessor.scala:103) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexPostProcessor.process(IndexPostProcessor.scala:57) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.ReadCommandPostProcessor.apply(IndexPostProcessor.scala:168) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.ReadCommandPostProcessor.apply(IndexPostProcessor.scala:161) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at org.apache.cassandra.db.PartitionRangeReadCommand.postReconciliationProcessing(PartitionRangeReadCommand.java:330) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:2284) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.db.PartitionRangeReadCommand.execute(PartitionRangeReadCommand.java:186) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at com.stratio.cassandra.lucene.IndexQueryHandler.executeSortedLuceneQuery(IndexQueryHandler.scala:226) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexQueryHandler.executeLuceneQuery(IndexQueryHandler.scala:193) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexQueryHandler.processStatement(IndexQueryHandler.scala:122) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at com.stratio.cassandra.lucene.IndexQueryHandler.process(IndexQueryHandler.scala:101) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
	at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.0.jar:3.11.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.0.jar:3.11.0]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.0.jar:3.11.0]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
smigfu commented

Some more details:

  • The BufferUnderflowException is only happening if UDTs are involved
  • There is another manifestation where the BUException isn't being thrown but the indices never update (even if force refreshed). They just don't deliver any search results.

=> all of the above happens intermittently only if I don't add any of the indexed fields to the SELECT clause (in my search case I am only interested in the primary keys which aren't indexed). As soon as I add the indexed fields to my SELECT all of the symptoms above disappear.

That said: Is it a requirement to add the indexed fields to your SELECT? Or do I always need to also add the primary keys to the index?

ealonsodb commented

Hi @smigfu
Could you please tell me wich versions(cassandras and index) are you using?. And also, a complete test in CQL?

smigfu commented

I will try to put something together. As said this is an intermittent issue. Would it help if I provide you with the files the index writes to the filesystem?

As for the version of C*: I have seen this with 3.0.10, 3.10.0, 3.11.0 and 3.11.1 (custom build of the plugin)

smiklosovic commented

I am hitting this too.