Search API Failures

Question

Search API Failures

docuracy opened this issue 3 years ago · 5 comments

Hi. We've started seeing some strange failures (listed below) with the search API of the form JsResultException(errors:List((,List(JsonValidationError(List(error.expected.jsnumber),WrappedArray()))))).

Is this the kind of thing that you might expect to see while ElasticSearch is busy indexing a large, newly-added gazetteer, and might it be resolved in time without intervention? The search appears to be fine when the gazetteer is first added, but then the errors start within an hour or so. Removal of one or other gazetteer from the query (not necessarily the new one) sometimes clears the error.

Any suggestions, please?

OK

Fail

Answer 1 · 2021-03-26T12:30:10.000Z

Can you identify in the logs where the error happens (class + line number)? It's a JSON validation error, which happens when the Recogito backend receives a JSON response from ElasticSearch it doesn't expect. Like, for example, an error message instead of a search results object.

Answer 2 · 2021-03-26T13:12:54.000Z

I've copied below the top of the log for one of the errors, but interpreting it is beyond me!

I do think that something has gone wrong with the ElasticSearch index, because a simple cURL search of the form "match": { "_all": "Spridlingto"} returns no hits, but "match": { "_all": "Spridlington"} returns lots of exact match hits.

2021-03-26 12:53:46,583 [ERROR] from application in application-akka.actor.default-dispatcher-2612 -

! @7j7hanjkb - Internal server error, for (GET) [/api/place/search?q=Spridlingto~&offset=0&size=20&authorities=heb%2Cmonasteries%2Cjunctions%2Crivers%2Costrich%2CGB1900%2Ctb%2Ccampop_towns%2CWDEW%2Clp%2Chillforts%2Cadmin] ->

play.api.UnexpectedException: Unexpected exception[NoSuchElementException: None.get]
        at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:247)
        at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:176)
        at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:363)
        at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:361)
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:346)
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:345)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.util.NoSuchElementException: None.get
        at scala.None$.get(Option.scala:347)
        at scala.None$.get(Option.scala:345)
        at services.entity.EntityType$.withName(EntityType.scala:16)
        at services.entity.EntityType$$anonfun$1.apply(EntityType.scala:19)
        at services.entity.EntityType$$anonfun$1.apply(EntityType.scala:19)
        at play.api.libs.json.JsSuccess.map(JsResult.scala:18)

Answer 3 · 2021-03-26T14:08:55.000Z

Hm, interesting. Yes, definitely looks like something's out of sync with the ElasticSearch index. I assume you already tried a cluster restart? There's also some API function that allows you to stop/start the cluster, with the effect that various caches are being flushed. (I once had a major crash, and things went back to normal once I stopped and restarted the cluster node. I can't find the exact API call... now)

Answer 4 · 2021-03-26T14:44:25.000Z

Cluster restart as outlined [here] didn't fix anything, sadly. I think I'm going to backup recent annotations and roll back the server to Wednesday night's backup and see if that fixes things...

Answer 5 · 2021-03-26T19:33:55.000Z

That's fixed it, almost. A couple of annotations (not all of them) made while the index was damaged seem to be causing ongoing issues with displaying all of the annotations on the related document. I've recorded the error below for future reference, but will close the issue as it's something I have to work out for myself !

I think I must have broken the index by overloading it with successive updates to a large gazetteer.

Thanks for the input earlier, Rainer.

2021-03-26 19:26:13,468 [ERROR] from application in application-akka.actor.default-dispatcher-564 -

! @7j7jabcbn - Internal server error, for (GET) [/api/document/nnclx5ynmz3zvy/part/1/annotations] ->

play.api.UnexpectedException: Unexpected exception[NoSuchElementException: No value found for 'AREA']
        at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:247)
        at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:176)
        at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:363)
        at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:361)
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:346)
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:345)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.util.NoSuchElementException: No value found for 'AREA'
        at scala.Enumeration.withName(Enumeration.scala:124)
        at services.annotation.AnnotationBody$$anonfun$1.apply(AnnotationBody.scala:69)
        at services.annotation.AnnotationBody$$anonfun$1.apply(AnnotationBody.scala:69)
        at play.api.libs.json.JsSuccess.map(JsResult.scala:18)
        at play.api.libs.json.Reads$$anonfun$map$1.apply(Reads.scala:43)