searchisko/elasticsearch-river-remote

Problems with indexFullUpdateCronExpression

Closed this issue · 3 comments

martintorgersen commented

We are using the indexFullUpdateCronExpression setting introduced in 1.5.3 (we are on this version), and have set it to "0 0 1 * * ?", expecting the updates to run once per day, starting at 1 AM.
However, it seems that this setting is not respected. Are you aware of any problems with this functionality?

From our config:
"remote": {
"urlGetDocuments": "http://<api_url>/{space}?api_key=<api_key>&skip={startAtIndex}&after={updatedAfter}",
"urlGetDocumentDetails": "http://<api_url>/{space}/{id}?api_key=<api_key>",
"getDocsResFieldDocuments": "documents",
"getDocsResFieldTotalcount": "total",
"timeout": "10s",
"indexUpdatePeriod": "5m",
"indexFullUpdateCronExpression ": "0 0 1 * * ?",
"spacesIndexed": "",
"maxIndexingThreads": 1
},

From log (filtered to only include relevant events, we have five spaces "steder", "turer", "bilder", "grupper" and "områder"):
3.9.15 01:38 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space steder
3.9.15 01:43 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space steder. 1274 updated and 0 deleted documents. Time elapsed 328s.
3.9.15 02:59 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space steder
3.9.15 03:05 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space steder. 1274 updated and 0 deleted documents. Time elapsed 329s.
3.9.15 04:39 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space turer
3.9.15 05:17 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space turer. 9695 updated and 4 deleted documents. Time elapsed 2324s.
3.9.15 05:42 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space bilder
3.9.15 06:52 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space grupper
3.9.15 06:55 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space grupper. 717 updated and 0 deleted documents. Time elapsed 174s.
3.9.15 07:51 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space bilder. 30184 updated and 0 deleted documents. Time elapsed 7711s.
3.9.15 09:45 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space turer
3.9.15 10:23 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space turer. 9695 updated and 0 deleted documents. Time elapsed 2293s.
3.9.15 10:53 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space områder
3.9.15 10:53 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space områder. 85 updated and 0 deleted documents. Time elapsed 23s.
3.9.15 13:43 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space steder
3.9.15 13:48 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space steder. 1274 updated and 0 deleted documents. Time elapsed 301s.
3.9.15 15:05 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space steder
3.9.15 15:10 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space steder. 1274 updated and 0 deleted documents. Time elapsed 325s.
3.9.15 17:18 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space turer
3.9.15 17:50 ERROR org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Failed full update for Space turer due: Read timed out
3.9.15 17:51 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space turer
3.9.15 17:51 ERROR org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Failed full update for Space turer due: Read timed out
3.9.15 17:51 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space turer
3.9.15 18:05 ERROR org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Failed full update for Space turer due: Read timed out
3.9.15 18:05 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space turer
3.9.15 18:43 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space turer. 9697 updated and 0 deleted documents. Time elapsed 2265s.
3.9.15 18:55 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space grupper
3.9.15 18:59 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space grupper. 717 updated and 0 deleted documents. Time elapsed 203s.
3.9.15 19:51 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space bilder
3.9.15 22:19 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space bilder. 30223 updated and 1 deleted documents. Time elapsed 8880s.
3.9.15 22:24 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space turer
3.9.15 22:53 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Go to perform full update for Space områder
3.9.15 22:54 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space områder. 85 updated and 0 deleted documents. Time elapsed 24s.
3.9.15 23:04 INFO org.jboss.elasticsearch.river.remote.SpaceByLastUpdateTimestampIndexer Finished full update for Space turer. 9701 updated and 0 deleted documents. Time elapsed 2451s.

velias commented

Check your configuration please, seems you have a space at the end of "indexFullUpdateCronExpression " name

martintorgersen commented

Fixing the configuration solved the problem. What a silly mistake, sorry about that!

velias commented

:-D