Indexing the whole db

Question

Indexing the whole db

slaniado opened this issue 11 years ago · 24 comments

slaniado commented 11 years ago

i want to index all the db, is there a way to do that or i need to create a river for each collection

slaniado commented 11 years ago

thank's

Answer 1 · 2013-11-21T09:28:24.000Z

You need to create a river for each collection.

On Thursday, November 21, 2013, Saul Laniado wrote:

i want to index all the db, is there a way to do that or i need to create
a river for each collection

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/177
.

Answer 2 · 2013-11-22T16:26:05.000Z

Let me know if you think your use case can justify to implement a new feature.
I am not saying it will be implemented :-)

Answer 3 · 2013-11-24T14:01:35.000Z

in my case i have a large number of collections, every time i add new type of document i create a new collection. To resolve this i will need to have a list of the rivers that i already have in ES and every time that i make a new insert in mongo i will need to check if there is the need to create a new river for this collection.
I think that a option to index all the DB in ES is a great idea.
Thank you

Answer 4 · 2013-12-05T14:56:20.000Z

I have just release 1.7.3.
you can install using the following command: plugin -i com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/1.7.3
You should use the parameters import_all_collections and maybe also drop_collection see this file [1] from more details.

I am planning to update the wiki by end of this week.

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/blob/530ce26400f802936cc2a5041865757ca32bf90c/manual-testing/issues/177/simple-mongodb-river-import-all-collections.json

Answer 5 · 2013-12-11T13:39:31.000Z

Great, thank’s a lot!

Saul

El 05/12/2013, a las 08:56, Richard Louapre notifications@github.com escribió:

I have just release 1.7.3.
you can install using the following command: plugin -i com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/1.7.3
You should use the parameters import_all_collections and maybe also drop_collection see this file [1] from more details.

I am planning to update the wiki by end of this week.

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/blob/530ce26400f802936cc2a5041865757ca32bf90c/manual-testing/issues/177/simple-mongodb-river-import-all-collections.json

—
Reply to this email directly or view it on GitHub.

Answer 6 · 2013-12-17T17:40:24.000Z

Hi,

I would like to use include_all_collection but I've got this log :

[2013-12-17 18:33:21,880][INFO ][org.elasticsearch.river.mongodb.Slurper] Skip initial import from collection mongodb177
[2013-12-17 18:33:21,881][INFO ][org.elasticsearch.river.mongodb.Slurper] No known previous slurping time for this collection

My config is :
{
"type": "mongodb",
"mongodb": {
"db": "mean-dev",
"options": {
"drop_collection": true,
"import_all_collections": true
}
},
"index": {
"name": "idx"
}
}

curl -XPUT "http://localhost:9200/_river/mongodb177/_meta" -d @all_collection.json

MongoDB : 2.4.8
Mongo-River : 1.7.3
Elasticseach : 0.90.7

curl -XGET localhost:9200/idx/articles/_count
{"count":0,"_shards":{"total":5,"successful":5,"failed":0}}

Or there are 3 documents in the database.

Can you help me ?

Thanks

Herve

Answer 7 · 2013-12-17T17:50:28.000Z

@hepr
Can you please do the following steps?
Delete the river.
Enable logging [1].
Restart ES
Register the river
Provide full ES log file.

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki#troubleshooting

Answer 8 · 2013-12-17T18:34:45.000Z

Hi,

I've done all.
I joined my log file.

[2013-12-17 19:30:45,673][INFO ][node [2013-12-17 19:30:45,673][INFO ][node [2013-12-17 19:30:45,681][INFO ][plugins [2013-12-17 19:30:47,776][INFO ][node [2013-12-17 19:30:47,777][INFO ][node [2013-12-17 19:30:47,864][INFO ][transport [2013-12-17 19:30:50,958][INFO ][cluster.service [2013-12-17 19:30:50,989][INFO ][discovery [2013-12-17 19:30:51,006][INFO ][http [2013-12-17 19:30:51,007][INFO ][node [2013-12-17 19:30:51,034][INFO ][gateway [2013-12-17 19:30:55,952][INFO ][node [2013-12-17 19:30:56,004][INFO ][node [2013-12-17 19:30:56,005][INFO ][node [2013-12-17 19:30:56,018][INFO ][node [2013-12-17 19:31:21,313][INFO ][node [2013-12-17 19:31:21,314][INFO ][node [2013-12-17 19:31:21,325][INFO ][plugins [2013-12-17 19:31:23,302][TRACE][rest.action.mongodb [2013-12-17 19:31:23,376][INFO ][node [2013-12-17 19:31:23,377][INFO ][node [2013-12-17 19:31:23,472][INFO ][transport [2013-12-17 19:31:26,548][INFO ][cluster.service [2013-12-17 19:31:26,583][INFO ][discovery [2013-12-17 19:31:26,603][INFO ][http [2013-12-17 19:31:26,603][INFO ][node [2013-12-17 19:31:26,626][INFO ][gateway [2013-12-17 19:31:56,442][INFO ][cluster.metadata [2013-12-17 19:31:56,765][INFO ][cluster.metadata [2013-12-17 19:31:56,772][INFO ][river.routing [2013-12-17 19:31:57,799][TRACE][org.ela [2013-12-17 19:31:57,800][INFO ][river.mongodb [2013-12-17 19:31:57,822][TRACE][river.mongodb [2013-12-17 19:31:57,824][INFO [2013-12-17 19:31:57,825][DEBUG][river.mongodb.util [2013-12-17 19:31:57,829][DEBUG][org.ela [2013-12-17 19:31:57,831][INFO ][cluster.metadata [2013-12-17 19:31:57,832][INFO [2013-12-17 19:31:57,832][INFO [2013-12-17 19:31:57,906][INFO ][cluster.metadata [2013-12-17 19:31:58,120][INFO [2013-12-17 19:31:58,240][INFO [2013-12-17 19:31:58,243][TRACE][org.ela [2013-12-17 19:31:58,243][TRACE][org.ela [2013-12-17 19:31:58,254][INFO [2013-12-17 19:31:58,255][INFO [2013-12-17 19:31:58,265][INFO ][cluster.metadata ] [Richards, Susan] version[0.90.7], pid[12378], build[36897d0/2013-11-13T12:06:54Z]
] [Richards, Susan] initializing ...
] [Richards, Susan] loaded [], sites []
] [Richards, Susan] initialized
] [Richards, Susan] starting ...
] [Richards, Susan] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.11:9300]}
] [Richards, Susan] new_master [Richards, Susan][edpSSFUXTmGQrfXrSebyew][inet[/192.168.0.11:9300]], reason: zen-disco-join (elected_as_master)
] [Richards, Susan] elasticsearch/edpSSFUXTmGQrfXrSebyew
] [Richards, Susan] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.11:9200]}
] [Richards, Susan] started
] [Richards, Susan] recovered [0] indices into cluster_state
] [Richards, Susan] stopping ...
] [Richards, Susan] stopped
] [Richards, Susan] closing ...
] [Richards, Susan] closed
] [Vindicator] version[0.90.7], pid[12507], build[36897d0/2013-11-13T12:06:54Z]
] [Vindicator] initializing ...
] [Vindicator] loaded [mongodb-river], sites [river-mongodb]
] [Vindicator] RestMongoDBRiverAction - baseUrl: /_river/mongodb
] [Vindicator] initialized
] [Vindicator] starting ...
] [Vindicator] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.11:9300]}
] [Vindicator] new_master [Vindicator][bbIdlxNsRIGbINdvgFQxwg][inet[/192.168.0.11:9300]], reason: zen-disco-join (elected_as_master)
] [Vindicator] elasticsearch/bbIdlxNsRIGbINdvgFQxwg
] [Vindicator] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.11:9200]}
] [Vindicator] started
] [Vindicator] recovered [0] indices into cluster_state
] [Vindicator] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings []
] [Vindicator] [_river] update_mapping mongodb177
] [Vindicator] no river _meta document found, retrying in 1000 ms
sticsearch.river.mongodb.MongoDBRiver] Initializing river : [mongodb177]
] Parse river settings for mongodb177
] mongoOptionsSettings: {drop_collection=true, import_all_collections=true}
][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb177
] setRiverStatus called with mongodb177 - RUNNING
sticsearch.river.mongodb.MongoDBRiver] Using mongodb server(s): host [localhost], port [27017]
] [Vindicator] [_river] update_mapping mongodb177
][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[1.7.3] - hash[4ca377f] - time[2013-12-05T14:34:13Z]
][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [true], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [mean-dev], collection [mongodb177], script [null], indexing to [idx]/[mean-dev]
] [Vindicator] [idx] creating index, cause [api], shards [5]/[1], mappings []
][org.elasticsearch.river.mongodb.MongoDBRiver] MongoAdminUser: - isAuthenticated: false
][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.4.8
sticsearch.river.mongodb.MongoDBRiver] serverStatus: { "serverUsed" : "localusr-LIFEBOOK-U772/127.0.1.1:27017" , "host" : "localusr-LIFEBOOK-U772" , "version" : "2.4.8" , "process" : "mongod" , "pid" : 6155 , "uptime" : 11065.0 , "uptimeMillis" : 11064694 , "uptimeEstimate" : 10934.0 , "localTime" : { "$date" : "2013-12-17T18:31:58.230Z"} , "asserts" : { "regular" : 0 , "warning" : 0 , "msg" : 0 , "user" : 28 , "rollovers" : 0} , "backgroundFlushing" : { "flushes" : 184 , "total_ms" : 2397 , "average_ms" : 13.027173913043478 , "last_ms" : 13 , "last_finished" : { "$date" : "2013-12-17T18:31:33.577Z"}} , "connections" : { "current" : 24 , "available" : 15976 , "totalCreated" : 417} , "cursors" : { "totalOpen" : 22 , "clientCursors_size" : 22 , "timedOut" : 1 , "totalNoTimeout" : 22} , "dur" : { "commits" : 30 , "journaledMB" : 0.0 , "writeToDataFilesMB" : 0.0 , "compression" : 0.0 , "commitsInWriteLock" : 0 , "earlyCommits" : 0 , "timeMs" : { "dt" : 3071 , "prepLogBuffer" : 0 , "writeToJournal" : 0 , "writeToDataFiles" : 0 , "remapPrivateView" : 0}} , "extra_info" : { "note" : "fields vary by platform" , "heap_usage_bytes" : 69604368 , "page_faults" : 49} , "globalLock" : { "totalTime" : 11064694000 , "lockTime" : 505464 , "currentQueue" : { "total" : 0 , "readers" : 0 , "writers" : 0} , "activeClients" : { "total" : 0 , "readers" : 0 , "writers" : 0}} , "indexCounters" : { "accesses" : 3105 , "hits" : 3105 , "misses" : 0 , "resets" : 0 , "missRatio" : 0.0} , "locks" : { "." : { "timeLockedMicros" : { "R" : 524473 , "W" : 505464} , "timeAcquiringMicros" : { "R" : 407892 , "W" : 56600}} , "admin" : { "timeLockedMicros" : { "r" : 24189 , "w" : 0} , "timeAcquiringMicros" : { "r" : 586 , "w" : 0}} , "local" : { "timeLockedMicros" : { "r" : 632052 , "w" : 7992} , "timeAcquiringMicros" : { "r" : 119523 , "w" : 5314}} , "ffxi" : { "timeLockedMicros" : { "r" : 4583 , "w" : 0} , "timeAcquiringMicros" : { "r" : 461 , "w" : 0}} , "chaumeil-new" : { "timeLockedMicros" : { "r" : 22663 , "w" : 20505} , "timeAcquiringMicros" : { "r" : 1089 , "w" : 970}} , "europexpress" : { "timeLockedMicros" : { "r" : 19677 , "w" : 0} , "timeAcquiringMicros" : { "r" : 895 , "w" : 0}} , "test" : { "timeLockedMicros" : { "r" : 5671 , "w" : 0} , "timeAcquiringMicros" : { "r" : 451 , "w" : 0}} , "user-auth" : { "timeLockedMicros" : { "r" : 12007 , "w" : 0} , "timeAcquiringMicros" : { "r" : 702 , "w" : 0}} , "mean-dev" : { "timeLockedMicros" : { "r" : 16272 , "w" : 17218} , "timeAcquiringMicros" : { "r" : 858 , "w" : 879}} , "flydb" : { "timeLockedMicros" : { "r" : 12761 , "w" : 0} , "timeAcquiringMicros" : { "r" : 866 , "w" : 0}} , "files" : { "timeLockedMicros" : { "r" : 14722 , "w" : 198} , "timeAcquiringMicros" : { "r" : 852 , "w" : 10}} , "chaumeil" : { "timeLockedMicros" : { "r" : 264994 , "w" : 146859} , "timeAcquiringMicros" : { "r" : 124511 , "w" : 100522}}} , "network" : { "bytesIn" : 2669693 , "bytesOut" : 36197057 , "numRequests" : 41769} , "opcounters" : { "insert" : 23 , "query" : 3987 , "update" : 88 , "delete" : 4 , "getmore" : 2121 , "command" : 37513} , "opcountersRepl" : { "insert" : 0 , "query" : 0 , "update" : 0 , "delete" : 0 , "getmore" : 0 , "command" : 0} , "recordStats" : { "accessesNotInMemory" : 15 , "pageFaultExceptionsThrown" : 13 , "admin" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "chaumeil" : { "accessesNotInMemory" : 13 , "pageFaultExceptionsThrown" : 13} , "chaumeil-new" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 0} , "europexpress" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "files" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "flydb" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "local" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "mean-dev" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 0} , "user-auth" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0}} , "repl" : { "setName" : "symeos" , "ismaster" : true , "secondary" : false , "hosts" : [ "localusr-LIFEBOOK-U772:27017"] , "primary" : "localusr-LIFEBOOK-U772:27017" , "me" : "localusr-LIFEBOOK-U772:27017"} , "writeBacksQueued" : false , "mem" : { "bits" : 64 , "resident" : 65 , "virtual" : 11653 , "supported" : true , "mapped" : 4734 , "mappedWithJournal" : 9468} , "metrics" : { "document" : { "deleted" : 1 , "inserted" : 23 , "returned" : 22221 , "updated" : 88} , "getLastError" : { "wtime" : { "num" : 0 , "totalMillis" : 0} , "wtimeouts" : 0} , "operation" : { "fastmod" : 0 , "idhack" : 1898 , "scanAndOrder" : 31} , "queryExecutor" : { "scanned" : 35217} , "record" : { "moves" : 4} , "repl" : { "apply" : { "batches" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0} , "buffer" : { "count" : 0 , "maxSizeBytes" : 268435456 , "sizeBytes" : 0} , "network" : { "bytes" : 0 , "getmores" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0 , "readersCreated" : 3} , "oplog" : { "insert" : { "num" : 688 , "totalMillis" : 0} , "insertBytes" : 403927} , "preload" : { "docs" : { "num" : 0 , "totalMillis" : 0} , "indexes" : { "num" : 0 , "totalMillis" : 0}}} , "ttl" : { "deletedDocuments" : 0 , "passes" : 184}} , "ok" : 1.0}
sticsearch.river.mongodb.MongoDBRiver] process: mongod
][org.elasticsearch.river.mongodb.Slurper] Skip initial import from collection mongodb177
][org.elasticsearch.river.mongodb.Slurper] No known previous slurping time for this collection
] [Vindicator] [_river] update_mapping mongodb177

Answer 9 · 2013-12-17T19:30:44.000Z

@hepr
I suspect that isValidOplogEntry return false.
Can you please confirm oplog.rs collection contains entries with the correct namespace?

If oplog.rs is not too big can you please dump it here?
Or you could also filter it by ns starts with mean-dev

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/blob/elasticsearch-river-mongodb-1.7.3/src/main/java/org/elasticsearch/river/mongodb/Slurper.java#L360

Answer 10 · 2013-12-17T21:07:27.000Z

My oplog.rs is too big.
See below a part of it.
I've done a flter on ns with mean-dev but it found no record.

/* 0 */
{
"ts" : Timestamp(1387294357, 1),
"h" : NumberLong(0),
"v" : 2,
"op" : "n",
"ns" : "",
"o" : {
"msg" : "initiating set"
}
}

/* 1 */
{
"ts" : Timestamp(1387294391, 1),
"h" : NumberLong(9058808299029266449),
"v" : 2,
"op" : "i",
"ns" : "chaumeil.system.indexes",
"o" : {
"ns" : "chaumeil.tmp.mr.DolibarrModules_0_inc",
"key" : {
"0" : 1
},
"name" : "0_1"
}
}

Answer 11 · 2013-12-17T21:31:30.000Z

@hepr
Can you try to add one more document in mean-dev database while the river is running?
That should generate a new entry in oplog.rs collection and the document should be available in ES.
Please provide ES log file.

Answer 12 · 2013-12-17T22:46:30.000Z

I insert data :

/* 0 */
{
"ts" : Timestamp(1387320323, 1),
"h" : NumberLong(7438850643797280065),
"v" : 2,
"op" : "i",
"ns" : "mean-dev.articles",
"o" : {
"_id" : ObjectId("52b0d403eaa86b25a1203f0c"),
"user" : ObjectId("5278b7cc445df41313000002"),
"content" : "boss",
"title" : "big",
"created" : ISODate("2013-11-05T09:18:31.951Z"),
"__v" : 0
}
}

/* 1 */
{
"ts" : Timestamp(1387320332, 1),
"h" : NumberLong(-8241836161622297468),
"v" : 2,
"op" : "i",
"ns" : "mean-dev.articles",
"o" : {
"_id" : ObjectId("52b0d40ceaa86b25a1203f0d"),
"user" : ObjectId("5278b7cc445df41313000002"),
"content" : "boss",
"title" : "big",
"created" : ISODate("2013-11-05T09:18:31.951Z"),
"__v" : 0
}
}

Log ES

[2013-12-17 19:30:45,673][INFO ][node ] [Richards, Susan] version[0.90.7], pid[12378], build[36897d0/2013-11-13T12:06:54Z]
[2013-12-17 19:30:45,673][INFO ][node ] [Richards, Susan] initializing ...
[2013-12-17 19:30:45,681][INFO ][plugins ] [Richards, Susan] loaded [], sites []
[2013-12-17 19:30:47,776][INFO ][node ] [Richards, Susan] initialized
[2013-12-17 19:30:47,777][INFO ][node ] [Richards, Susan] starting ...
[2013-12-17 19:30:47,864][INFO ][transport ] [Richards, Susan] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.11:9300]}
[2013-12-17 19:30:50,958][INFO ][cluster.service ] [Richards, Susan] new_master [Richards, Susan][edpSSFUXTmGQrfXrSebyew][inet[/192.168.0.11:9300]], reason: zen-disco-join (elected_as_master)
[2013-12-17 19:30:50,989][INFO ][discovery ] [Richards, Susan] elasticsearch/edpSSFUXTmGQrfXrSebyew
[2013-12-17 19:30:51,006][INFO ][http ] [Richards, Susan] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.11:9200]}
[2013-12-17 19:30:51,007][INFO ][node ] [Richards, Susan] started
[2013-12-17 19:30:51,034][INFO ][gateway ] [Richards, Susan] recovered [0] indices into cluster_state
[2013-12-17 19:30:55,952][INFO ][node ] [Richards, Susan] stopping ...
[2013-12-17 19:30:56,004][INFO ][node ] [Richards, Susan] stopped
[2013-12-17 19:30:56,005][INFO ][node ] [Richards, Susan] closing ...
[2013-12-17 19:30:56,018][INFO ][node ] [Richards, Susan] closed
[2013-12-17 19:31:21,313][INFO ][node ] [Vindicator] version[0.90.7], pid[12507], build[36897d0/2013-11-13T12:06:54Z]
[2013-12-17 19:31:21,314][INFO ][node ] [Vindicator] initializing ...
[2013-12-17 19:31:21,325][INFO ][plugins ] [Vindicator] loaded [mongodb-river], sites [river-mongodb]
[2013-12-17 19:31:23,302][TRACE][rest.action.mongodb ] [Vindicator] RestMongoDBRiverAction - baseUrl: /_river/mongodb
[2013-12-17 19:31:23,376][INFO ][node ] [Vindicator] initialized
[2013-12-17 19:31:23,377][INFO ][node ] [Vindicator] starting ...
[2013-12-17 19:31:23,472][INFO ][transport ] [Vindicator] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.11:9300]}
[2013-12-17 19:31:26,548][INFO ][cluster.service ] [Vindicator] new_master [Vindicator][bbIdlxNsRIGbINdvgFQxwg][inet[/192.168.0.11:9300]], reason: zen-disco-join (elected_as_master)
[2013-12-17 19:31:26,583][INFO ][discovery ] [Vindicator] elasticsearch/bbIdlxNsRIGbINdvgFQxwg
[2013-12-17 19:31:26,603][INFO ][http ] [Vindicator] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.11:9200]}
[2013-12-17 19:31:26,603][INFO ][node ] [Vindicator] started
[2013-12-17 19:31:26,626][INFO ][gateway ] [Vindicator] recovered [0] indices into cluster_state
[2013-12-17 19:31:56,442][INFO ][cluster.metadata ] [Vindicator] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings []
[2013-12-17 19:31:56,765][INFO ][cluster.metadata ] [Vindicator] [_river] update_mapping mongodb177
[2013-12-17 19:31:56,772][INFO ][river.routing ] [Vindicator] no river _meta document found, retrying in 1000 ms
[2013-12-17 19:31:57,799][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Initializing river : [mongodb177]
[2013-12-17 19:31:57,800][INFO ][river.mongodb ] Parse river settings for mongodb177
[2013-12-17 19:31:57,822][TRACE][river.mongodb ] mongoOptionsSettings: {drop_collection=true, import_all_collections=true}
[2013-12-17 19:31:57,824][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb177
[2013-12-17 19:31:57,825][DEBUG][river.mongodb.util ] setRiverStatus called with mongodb177 - RUNNING
[2013-12-17 19:31:57,829][DEBUG][org.elasticsearch.river.mongodb.MongoDBRiver] Using mongodb server(s): host [localhost], port [27017]
[2013-12-17 19:31:57,831][INFO ][cluster.metadata ] [Vindicator] [_river] update_mapping mongodb177
[2013-12-17 19:31:57,832][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[1.7.3] - hash[4ca377f] - time[2013-12-05T14:34:13Z]
[2013-12-17 19:31:57,832][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [true], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [mean-dev], collection [mongodb177], script [null], indexing to [idx]/[mean-dev]
[2013-12-17 19:31:57,906][INFO ][cluster.metadata ] [Vindicator] [idx] creating index, cause [api], shards [5]/[1], mappings []
[2013-12-17 19:31:58,120][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoAdminUser: - isAuthenticated: false
[2013-12-17 19:31:58,240][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.4.8
[2013-12-17 19:31:58,243][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] serverStatus: { "serverUsed" : "localusr-LIFEBOOK-U772/127.0.1.1:27017" , "host" : "localusr-LIFEBOOK-U772" , "version" : "2.4.8" , "process" : "mongod" , "pid" : 6155 , "uptime" : 11065.0 , "uptimeMillis" : 11064694 , "uptimeEstimate" : 10934.0 , "localTime" : { "$date" : "2013-12-17T18:31:58.230Z"} , "asserts" : { "regular" : 0 , "warning" : 0 , "msg" : 0 , "user" : 28 , "rollovers" : 0} , "backgroundFlushing" : { "flushes" : 184 , "total_ms" : 2397 , "average_ms" : 13.027173913043478 , "last_ms" : 13 , "last_finished" : { "$date" : "2013-12-17T18:31:33.577Z"}} , "connections" : { "current" : 24 , "available" : 15976 , "totalCreated" : 417} , "cursors" : { "totalOpen" : 22 , "clientCursors_size" : 22 , "timedOut" : 1 , "totalNoTimeout" : 22} , "dur" : { "commits" : 30 , "journaledMB" : 0.0 , "writeToDataFilesMB" : 0.0 , "compression" : 0.0 , "commitsInWriteLock" : 0 , "earlyCommits" : 0 , "timeMs" : { "dt" : 3071 , "prepLogBuffer" : 0 , "writeToJournal" : 0 , "writeToDataFiles" : 0 , "remapPrivateView" : 0}} , "extra_info" : { "note" : "fields vary by platform" , "heap_usage_bytes" : 69604368 , "page_faults" : 49} , "globalLock" : { "totalTime" : 11064694000 , "lockTime" : 505464 , "currentQueue" : { "total" : 0 , "readers" : 0 , "writers" : 0} , "activeClients" : { "total" : 0 , "readers" : 0 , "writers" : 0}} , "indexCounters" : { "accesses" : 3105 , "hits" : 3105 , "misses" : 0 , "resets" : 0 , "missRatio" : 0.0} , "locks" : { "." : { "timeLockedMicros" : { "R" : 524473 , "W" : 505464} , "timeAcquiringMicros" : { "R" : 407892 , "W" : 56600}} , "admin" : { "timeLockedMicros" : { "r" : 24189 , "w" : 0} , "timeAcquiringMicros" : { "r" : 586 , "w" : 0}} , "local" : { "timeLockedMicros" : { "r" : 632052 , "w" : 7992} , "timeAcquiringMicros" : { "r" : 119523 , "w" : 5314}} , "ffxi" : { "timeLockedMicros" : { "r" : 4583 , "w" : 0} , "timeAcquiringMicros" : { "r" : 461 , "w" : 0}} , "chaumeil-new" : { "timeLockedMicros" : { "r" : 22663 , "w" : 20505} , "timeAcquiringMicros" : { "r" : 1089 , "w" : 970}} , "europexpress" : { "timeLockedMicros" : { "r" : 19677 , "w" : 0} , "timeAcquiringMicros" : { "r" : 895 , "w" : 0}} , "test" : { "timeLockedMicros" : { "r" : 5671 , "w" : 0} , "timeAcquiringMicros" : { "r" : 451 , "w" : 0}} , "user-auth" : { "timeLockedMicros" : { "r" : 12007 , "w" : 0} , "timeAcquiringMicros" : { "r" : 702 , "w" : 0}} , "mean-dev" : { "timeLockedMicros" : { "r" : 16272 , "w" : 17218} , "timeAcquiringMicros" : { "r" : 858 , "w" : 879}} , "flydb" : { "timeLockedMicros" : { "r" : 12761 , "w" : 0} , "timeAcquiringMicros" : { "r" : 866 , "w" : 0}} , "files" : { "timeLockedMicros" : { "r" : 14722 , "w" : 198} , "timeAcquiringMicros" : { "r" : 852 , "w" : 10}} , "chaumeil" : { "timeLockedMicros" : { "r" : 264994 , "w" : 146859} , "timeAcquiringMicros" : { "r" : 124511 , "w" : 100522}}} , "network" : { "bytesIn" : 2669693 , "bytesOut" : 36197057 , "numRequests" : 41769} , "opcounters" : { "insert" : 23 , "query" : 3987 , "update" : 88 , "delete" : 4 , "getmore" : 2121 , "command" : 37513} , "opcountersRepl" : { "insert" : 0 , "query" : 0 , "update" : 0 , "delete" : 0 , "getmore" : 0 , "command" : 0} , "recordStats" : { "accessesNotInMemory" : 15 , "pageFaultExceptionsThrown" : 13 , "admin" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "chaumeil" : { "accessesNotInMemory" : 13 , "pageFaultExceptionsThrown" : 13} , "chaumeil-new" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 0} , "europexpress" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "files" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "flydb" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "local" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0} , "mean-dev" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 0} , "user-auth" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0}} , "repl" : { "setName" : "symeos" , "ismaster" : true , "secondary" : false , "hosts" : [ "localusr-LIFEBOOK-U772:27017"] , "primary" : "localusr-LIFEBOOK-U772:27017" , "me" : "localusr-LIFEBOOK-U772:27017"} , "writeBacksQueued" : false , "mem" : { "bits" : 64 , "resident" : 65 , "virtual" : 11653 , "supported" : true , "mapped" : 4734 , "mappedWithJournal" : 9468} , "metrics" : { "document" : { "deleted" : 1 , "inserted" : 23 , "returned" : 22221 , "updated" : 88} , "getLastError" : { "wtime" : { "num" : 0 , "totalMillis" : 0} , "wtimeouts" : 0} , "operation" : { "fastmod" : 0 , "idhack" : 1898 , "scanAndOrder" : 31} , "queryExecutor" : { "scanned" : 35217} , "record" : { "moves" : 4} , "repl" : { "apply" : { "batches" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0} , "buffer" : { "count" : 0 , "maxSizeBytes" : 268435456 , "sizeBytes" : 0} , "network" : { "bytes" : 0 , "getmores" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0 , "readersCreated" : 3} , "oplog" : { "insert" : { "num" : 688 , "totalMillis" : 0} , "insertBytes" : 403927} , "preload" : { "docs" : { "num" : 0 , "totalMillis" : 0} , "indexes" : { "num" : 0 , "totalMillis" : 0}}} , "ttl" : { "deletedDocuments" : 0 , "passes" : 184}} , "ok" : 1.0}
[2013-12-17 19:31:58,243][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] process: mongod
[2013-12-17 19:31:58,254][INFO ][org.elasticsearch.river.mongodb.Slurper] Skip initial import from collection mongodb177
[2013-12-17 19:31:58,255][INFO ][org.elasticsearch.river.mongodb.Slurper] No known previous slurping time for this collection
[2013-12-17 19:31:58,265][INFO ][cluster.metadata ] [Vindicator] [_river] update_mapping mongodb177
[2013-12-17 23:45:23,423][INFO ][cluster.metadata ] [Vindicator] [_river] update_mapping mongodb177
[2013-12-17 23:45:23,443][INFO ][cluster.metadata ] [Vindicator] [idx] update_mapping articles

Answer 13 · 2013-12-17T23:24:15.000Z

What do you get from these commands:
GET /mean-dev/_count GET /mean-dev/articles/_count

Answer 14 · 2013-12-18T08:37:07.000Z

GET /mean-dev/_count : 2
GET /mean-dev/articles/_count : 2
There is only new but not old document

Answer 15 · 2013-12-18T11:01:49.000Z

That's because import_all_collections uses only oplog.rs collection so if the entry are missing the documents will not be imported.

Can you do a bulk update of the existing documents?

I will see if I can change the logic for the next release.

Answer 16 · 2013-12-18T11:24:39.000Z

Ok I understand. If I do a backup of my db, I delete it and restore it. Is it import all change in oplog.rs ?

Answer 17 · 2013-12-18T12:05:48.000Z

Yes that should work.

Answer 18 · 2013-12-19T13:36:51.000Z

I drop a database and restore it. oplog.rs add some documents.

I start elasticsearch index and this is the log with START_FAILED

[2013-12-19 14:36:05,028][INFO ][node ] [American Ace] initialized
[2013-12-19 14:36:05,028][INFO ][node ] [American Ace] starting ...
[2013-12-19 14:36:05,126][INFO ][transport ] [American Ace] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.11:9300]}
[2013-12-19 14:36:08,219][INFO ][cluster.service ] [American Ace] new_master [American Ace][FWIM5t1gRd66QXrCHgQyCQ][inet[/192.168.0.11:9300]], reason: zen-disco-join (elected_as_master)
[2013-12-19 14:36:08,257][INFO ][discovery ] [American Ace] elasticsearch/FWIM5t1gRd66QXrCHgQyCQ
[2013-12-19 14:36:08,280][INFO ][http ] [American Ace] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.11:9200]}
[2013-12-19 14:36:08,283][INFO ][node ] [American Ace] started
[2013-12-19 14:36:08,308][INFO ][gateway ] [American Ace] recovered [0] indices into cluster_state
[2013-12-19 14:36:15,949][INFO ][cluster.metadata ] [American Ace] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings []
[2013-12-19 14:36:16,326][INFO ][cluster.metadata ] [American Ace] [_river] update_mapping mongodb177
[2013-12-19 14:36:16,337][INFO ][river.routing ] [American Ace] no river _meta document found, retrying in 1000 ms
[2013-12-19 14:36:17,361][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Initializing river : [mongodb177]
[2013-12-19 14:36:17,362][INFO ][river.mongodb ] Parse river settings for mongodb177
[2013-12-19 14:36:17,377][TRACE][river.mongodb ] mongoOptionsSettings: {drop_collection=true, import_all_collections=true}
[2013-12-19 14:36:17,378][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb177
[2013-12-19 14:36:17,380][DEBUG][river.mongodb.util ] setRiverStatus called with mongodb177 - RUNNING
[2013-12-19 14:36:17,389][DEBUG][org.elasticsearch.river.mongodb.MongoDBRiver] Using mongodb server(s): host [localhost], port [27017]
[2013-12-19 14:36:17,391][INFO ][cluster.metadata ] [American Ace] [_river] update_mapping mongodb177
[2013-12-19 14:36:17,393][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[1.7.3] - hash[4ca377f] - time[2013-12-05T14:34:13Z]
[2013-12-19 14:36:17,394][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [true], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [chaumeil], collection [mongodb177], script [null], indexing to [chaumeil]/[chaumeil]
[2013-12-19 14:36:17,472][INFO ][cluster.metadata ] [American Ace] [chaumeil] creating index, cause [api], shards [5]/[1], mappings []
[2013-12-19 14:36:17,745][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoAdminUser: - isAuthenticated: false
[2013-12-19 14:36:17,858][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.4.8
[2013-12-19 14:36:17,862][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] serverStatus: { "serverUsed" : "localusr-LIFEBOOK-U772/127.0.1.1:27017" , "host" : "localusr-LIFEBOOK-U772" , "version" : "2.4.8" , "process" : "mongod" , "pid" : 11385 , "uptime" : 74.0 , "uptimeMillis" : 74716 , "uptimeEstimate" : 73.0 , "localTime" : { "$date" : "2013-12-19T13:36:17.850Z"} , "asserts" : { "regular" : 0 , "warning" : 0 , "msg" : 0 , "user" : 0 , "rollovers" : 0} , "backgroundFlushing" : { "flushes" : 1 , "total_ms" : 4 , "average_ms" : 4.0 , "last_ms" : 4 , "last_finished" : { "$date" : "2013-12-19T13:36:03.147Z"}} , "connections" : { "current" : 9 , "available" : 15991 , "totalCreated" : 26} , "cursors" : { "totalOpen" : 0 , "clientCursors_size" : 0 , "timedOut" : 0} , "dur" : { "commits" : 30 , "journaledMB" : 0.0 , "writeToDataFilesMB" : 0.0 , "compression" : 0.0 , "commitsInWriteLock" : 0 , "earlyCommits" : 0 , "timeMs" : { "dt" : 3068 , "prepLogBuffer" : 0 , "writeToJournal" : 0 , "writeToDataFiles" : 0 , "remapPrivateView" : 0}} , "extra_info" : { "note" : "fields vary by platform" , "heap_usage_bytes" : 62625248 , "page_faults" : 176} , "globalLock" : { "totalTime" : 74717000 , "lockTime" : 19718 , "currentQueue" : { "total" : 0 , "readers" : 0 , "writers" : 0} , "activeClients" : { "total" : 0 , "readers" : 0 , "writers" : 0}} , "indexCounters" : { "accesses" : 149 , "hits" : 149 , "misses" : 0 , "resets" : 0 , "missRatio" : 0.0} , "locks" : { "." : { "timeLockedMicros" : { "R" : 2858 , "W" : 19718} , "timeAcquiringMicros" : { "R" : 1996 , "W" : 338}} , "admin" : { "timeLockedMicros" : { } , "timeAcquiringMicros" : { }} , "local" : { "timeLockedMicros" : { "r" : 31880 , "w" : 6} , "timeAcquiringMicros" : { "r" : 11074 , "w" : 32}} , "chaumeil" : { "timeLockedMicros" : { "r" : 5384 , "w" : 274} , "timeAcquiringMicros" : { "r" : 323 , "w" : 3}}} , "network" : { "bytesIn" : 21488 , "bytesOut" : 4305252 , "numRequests" : 296} , "opcounters" : { "insert" : 1 , "query" : 154 , "update" : 0 , "delete" : 0 , "getmore" : 6 , "command" : 140} , "opcountersRepl" : { "insert" : 0 , "query" : 0 , "update" : 0 , "delete" : 0 , "getmore" : 0 , "command" : 0} , "recordStats" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 1 , "chaumeil" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 1} , "local" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0}} , "repl" : { "setName" : "symeos" , "ismaster" : true , "secondary" : false , "hosts" : [ "localusr-LIFEBOOK-U772:27017"] , "primary" : "localusr-LIFEBOOK-U772:27017" , "me" : "localusr-LIFEBOOK-U772:27017"} , "writeBacksQueued" : false , "mem" : { "bits" : 64 , "resident" : 47 , "virtual" : 9011 , "supported" : true , "mapped" : 4254 , "mappedWithJournal" : 8508} , "metrics" : { "document" : { "deleted" : 0 , "inserted" : 1 , "returned" : 5100 , "updated" : 0} , "getLastError" : { "wtime" : { "num" : 0 , "totalMillis" : 0} , "wtimeouts" : 0} , "operation" : { "fastmod" : 0 , "idhack" : 147 , "scanAndOrder" : 0} , "queryExecutor" : { "scanned" : 245} , "record" : { "moves" : 0} , "repl" : { "apply" : { "batches" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0} , "buffer" : { "count" : 0 , "maxSizeBytes" : 268435456 , "sizeBytes" : 0} , "network" : { "bytes" : 0 , "getmores" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0 , "readersCreated" : 3} , "oplog" : { "insert" : { "num" : 0 , "totalMillis" : 0} , "insertBytes" : 0} , "preload" : { "docs" : { "num" : 0 , "totalMillis" : 0} , "indexes" : { "num" : 0 , "totalMillis" : 0}}} , "ttl" : { "deletedDocuments" : 0 , "passes" : 1}} , "ok" : 1.0}
[2013-12-19 14:36:17,863][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] process: mongod
[2013-12-19 14:36:17,885][INFO ][cluster.metadata ] [American Ace] [_river] update_mapping mongodb177
[2013-12-19 14:36:17,896][INFO ][org.elasticsearch.river.mongodb.Slurper] Skip initial import from collection mongodb177
[2013-12-19 14:36:17,896][INFO ][org.elasticsearch.river.mongodb.Slurper] No known previous slurping time for this collection
[2013-12-19 14:36:17,937][WARN ][org.elasticsearch.river.mongodb.Indexer] Cannot get object id. Skip the current item: [{ "ns" : "chaumeil.tmp.mr.DolibarrModules_0_inc" , "key" : { "0" : 1} , "name" : "0_1"}]
[2013-12-19 14:36:17,937][ERROR][org.elasticsearch.river.mongodb.Indexer] Unknown operation for id[] - entry [{ "create" : "tmp.mr.DolibarrModules_0" , "temp" : true}] - index[chaumeil] - type[null]
[2013-12-19 14:36:17,938][WARN ][org.elasticsearch.river.mongodb.Indexer] Cannot get object id. Skip the current item: [{ "ns" : "chaumeil.tmp.mr.DolibarrModules_0" , "v" : 1 , "key" : { "_id" : 1} , "name" : "id"}]
[2013-12-19 14:36:17,953][WARN ][org.elasticsearch.river.mongodb.Indexer] Cannot get object id. Skip the current item: [{ "ns" : "chaumeil.tmp.mr.DolibarrModules_1_inc" , "key" : { "0" : 1} , "name" : "0_1"}]
[2013-12-19 14:36:17,954][ERROR][org.elasticsearch.river.mongodb.Indexer] Unknown operation for id[] - entry [{ "create" : "tmp.mr.DolibarrModules_1" , "temp" : true}] - index[chaumeil] - type[null]
[2013-12-19 14:36:17,955][WARN ][org.elasticsearch.river.mongodb.Indexer] Cannot get object id. Skip the current item: [{ "ns" : "chaumeil.tmp.mr.DolibarrModules_1" , "v" : 1 , "key" : { "_id" : 1} , "name" : "id"}]
[2013-12-19 14:36:17,957][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] About to flush bulk request index[chaumeil] - type[view_listMenu]
[2013-12-19 14:36:17,957][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] About to flush bulk request index[chaumeil] - type[tmp.mr.DolibarrModules_0_inc]
[2013-12-19 14:36:17,960][WARN ][index.mapper ] [American Ace] [chaumeil] Type [tmp.mr.DolibarrModules_0] contains a '.', it is recommended not to include it within a type name
[2013-12-19 14:36:17,978][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] type[chaumeil] does not exist in index[view_listMenu]. No need to remove mapping.
[2013-12-19 14:36:18,017][WARN ][index.mapper ] [American Ace] [chaumeil] Type [tmp.mr.DolibarrModules_1] contains a '.', it is recommended not to include it within a type name
[2013-12-19 14:36:18,034][WARN ][index.mapper ] [American Ace] [chaumeil] Type [tmp.mr.DolibarrModules_0] contains a '.', it is recommended not to include it within a type name
[2013-12-19 14:36:18,034][INFO ][cluster.metadata ] [American Ace] [chaumeil] update_mapping tmp.mr.DolibarrModules_0
[2013-12-19 14:36:18,058][WARN ][index.mapper ] [American Ace] [chaumeil] Type [tmp.mr.DolibarrModules_1] contains a '.', it is recommended not to include it within a type name
[2013-12-19 14:36:18,058][INFO ][cluster.metadata ] [American Ace] [chaumeil] update_mapping tmp.mr.DolibarrModules_1
[2013-12-19 14:36:18,088][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] type[chaumeil] does not exist in index[tmp.mr.DolibarrModules_0_inc]. No need to remove mapping.
[2013-12-19 14:36:18,130][WARN ][org.elasticsearch.river.mongodb.Indexer] Cannot get object id. Skip the current item: [{ "ns" : "chaumeil.tmp.mr.DolibarrModules_2_inc" , "key" : { "0" : 1} , "name" : "0_1"}]
[2013-12-19 14:36:18,133][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] About to flush bulk request index[chaumeil] - type[view_listSubmenu]
[2013-12-19 14:36:18,134][WARN ][org.elasticsearch.river.mongodb.Indexer] Cannot get object id. Skip the current item: [{ "ns" : "chaumeil.tmp.mr.DolibarrModules_3_inc" , "key" : { "0" : 1} , "name" : "0_1"}]
[2013-12-19 14:36:18,138][ERROR][org.elasticsearch.river.mongodb.Indexer] Unknown operation for id[] - entry [{ "create" : "tmp.mr.DolibarrModules_3" , "temp" : true}] - index[chaumeil] - type[null]
[2013-12-19 14:36:18,139][ERROR][org.elasticsearch.river.mongodb.Indexer] Unknown operation for id[] - entry [{ "create" : "tmp.mr.DolibarrModules_2" , "temp" : true}] - index[chaumeil] - type[null]
[2013-12-19 14:36:18,139][WARN ][org.elasticsearch.river.mongodb.Indexer] Cannot get object id. Skip the current item: [{ "ns" : "chaumeil.tmp.mr.DolibarrModules_3" , "v" : 1 , "key" : { "_id" : 1} , "name" : "id"}]
[2013-12-19 14:36:18,139][WARN ][org.elasticsearch.river.mongodb.Indexer] Cannot get object id. Skip the current item: [{ "ns" : "chaumeil.tmp.mr.DolibarrModules_2" , "v" : 1 , "key" : { "_id" : 1} , "name" : "id"}]
[2013-12-19 14:36:18,156][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] About to flush bulk request index[chaumeil] - type[tmp.mr.DolibarrModules_1_inc]
[2013-12-19 14:36:18,157][WARN ][index.mapper ] [American Ace] [chaumeil] Type [tmp.mr.DolibarrModules_3] contains a '.', it is recommended not to include it within a type name
[2013-12-19 14:36:18,185][INFO ][cluster.metadata ] [American Ace] [_river] update_mapping mongodb177
[2013-12-19 14:36:18,201][WARN ][index.mapper ] [American Ace] [chaumeil] Type [tmp.mr.DolibarrModules_3] contains a '.', it is recommended not to include it within a type name
[2013-12-19 14:36:18,206][INFO ][cluster.metadata ] [American Ace] [chaumeil] update_mapping tmp.mr.DolibarrModules_3
[2013-12-19 14:36:18,316][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] type[chaumeil] does not exist in index[view_listSubmenu]. No need to remove mapping.
[2013-12-19 14:36:18,374][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] type[chaumeil] does not exist in index[tmp.mr.DolibarrModules_1_inc]. No need to remove mapping.
[2013-12-19 14:36:18,880][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] About to start river: mongodb177
[2013-12-19 14:36:18,880][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb177
[2013-12-19 14:36:18,880][DEBUG][river.mongodb.util ] setRiverStatus called with mongodb177 - RUNNING
[2013-12-19 14:36:18,895][DEBUG][org.elasticsearch.river.mongodb.MongoDBRiver] Using mongodb server(s): host [localhost], port [27017]
[2013-12-19 14:36:18,896][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[1.7.3] - hash[4ca377f] - time[2013-12-05T14:34:13Z]
[2013-12-19 14:36:18,896][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [true], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [chaumeil], collection [mongodb177], script [null], indexing to [chaumeil]/[chaumeil]
[2013-12-19 14:36:18,898][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.4.8
[2013-12-19 14:36:18,903][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] serverStatus: { "serverUsed" : "localusr-LIFEBOOK-U772/127.0.1.1:27017" , "host" : "localusr-LIFEBOOK-U772" , "version" : "2.4.8" , "process" : "mongod" , "pid" : 11385 , "uptime" : 75.0 , "uptimeMillis" : 75763 , "uptimeEstimate" : 74.0 , "localTime" : { "$date" : "2013-12-19T13:36:18.897Z"} , "asserts" : { "regular" : 0 , "warning" : 0 , "msg" : 0 , "user" : 0 , "rollovers" : 0} , "backgroundFlushing" : { "flushes" : 1 , "total_ms" : 4 , "average_ms" : 4.0 , "last_ms" : 4 , "last_finished" : { "$date" : "2013-12-19T13:36:03.147Z"}} , "connections" : { "current" : 14 , "available" : 15986 , "totalCreated" : 31} , "cursors" : { "totalOpen" : 1 , "clientCursors_size" : 1 , "timedOut" : 0 , "totalNoTimeout" : 1} , "dur" : { "commits" : 30 , "journaledMB" : 0.0 , "writeToDataFilesMB" : 0.0 , "compression" : 0.0 , "commitsInWriteLock" : 0 , "earlyCommits" : 0 , "timeMs" : { "dt" : 3068 , "prepLogBuffer" : 0 , "writeToJournal" : 0 , "writeToDataFiles" : 0 , "remapPrivateView" : 0}} , "extra_info" : { "note" : "fields vary by platform" , "heap_usage_bytes" : 62650960 , "page_faults" : 176} , "globalLock" : { "totalTime" : 75763000 , "lockTime" : 19726 , "currentQueue" : { "total" : 0 , "readers" : 0 , "writers" : 0} , "activeClients" : { "total" : 0 , "readers" : 0 , "writers" : 0}} , "indexCounters" : { "accesses" : 149 , "hits" : 149 , "misses" : 0 , "resets" : 0 , "missRatio" : 0.0} , "locks" : { "." : { "timeLockedMicros" : { "R" : 2884 , "W" : 19726} , "timeAcquiringMicros" : { "R" : 2014 , "W" : 341}} , "admin" : { "timeLockedMicros" : { } , "timeAcquiringMicros" : { }} , "local" : { "timeLockedMicros" : { "r" : 37811 , "w" : 6} , "timeAcquiringMicros" : { "r" : 11084 , "w" : 32}} , "chaumeil" : { "timeLockedMicros" : { "r" : 5390 , "w" : 274} , "timeAcquiringMicros" : { "r" : 325 , "w" : 3}}} , "network" : { "bytesIn" : 21881 , "bytesOut" : 8538693 , "numRequests" : 303} , "opcounters" : { "insert" : 1 , "query" : 156 , "update" : 0 , "delete" : 0 , "getmore" : 7 , "command" : 144} , "opcountersRepl" : { "insert" : 0 , "query" : 0 , "update" : 0 , "delete" : 0 , "getmore" : 0 , "command" : 0} , "recordStats" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 1 , "chaumeil" : { "accessesNotInMemory" : 1 , "pageFaultExceptionsThrown" : 1} , "local" : { "accessesNotInMemory" : 0 , "pageFaultExceptionsThrown" : 0}} , "repl" : { "setName" : "symeos" , "ismaster" : true , "secondary" : false , "hosts" : [ "localusr-LIFEBOOK-U772:27017"] , "primary" : "localusr-LIFEBOOK-U772:27017" , "me" : "localusr-LIFEBOOK-U772:27017"} , "writeBacksQueued" : false , "mem" : { "bits" : 64 , "resident" : 51 , "virtual" : 9020 , "supported" : true , "mapped" : 4254 , "mappedWithJournal" : 8508} , "metrics" : { "document" : { "deleted" : 0 , "inserted" : 1 , "returned" : 10161 , "updated" : 0} , "getLastError" : { "wtime" : { "num" : 0 , "totalMillis" : 0} , "wtimeouts" : 0} , "operation" : { "fastmod" : 0 , "idhack" : 147 , "scanAndOrder" : 0} , "queryExecutor" : { "scanned" : 350} , "record" : { "moves" : 0} , "repl" : { "apply" : { "batches" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0} , "buffer" : { "count" : 0 , "maxSizeBytes" : 268435456 , "sizeBytes" : 0} , "network" : { "bytes" : 0 , "getmores" : { "num" : 0 , "totalMillis" : 0} , "ops" : 0 , "readersCreated" : 3} , "oplog" : { "insert" : { "num" : 0 , "totalMillis" : 0} , "insertBytes" : 0} , "preload" : { "docs" : { "num" : 0 , "totalMillis" : 0} , "indexes" : { "num" : 0 , "totalMillis" : 0}}} , "ttl" : { "deletedDocuments" : 0 , "passes" : 1}} , "ok" : 1.0}
[2013-12-19 14:36:18,904][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] process: mongod
[2013-12-19 14:36:18,924][WARN ][org.elasticsearch.river.mongodb.MongoDBRiver] Fail to start river mongodb177
java.lang.IllegalThreadStateException
at java.lang.Thread.start(Thread.java:684)
at org.elasticsearch.river.mongodb.MongoDBRiver.start(MongoDBRiver.java:220)
at org.elasticsearch.river.mongodb.StatusChecker.run(StatusChecker.java:26)
at java.lang.Thread.run(Thread.java:724)
[2013-12-19 14:36:18,935][DEBUG][river.mongodb.util ] setRiverStatus called with mongodb177 - START_FAILED

Answer 19 · 2013-12-19T15:25:55.000Z

@hepr
From ES log:

[2013-12-19 14:36:18,157][WARN ][index.mapper ] [American Ace] [chaumeil] Type [tmp.mr.DolibarrModules_3] contains a '.', it is recommended not to include it within a type name

This exception is a known issue (will be fixed in the next release):

[WARN ][org.elasticsearch.river.mongodb.MongoDBRiver] Fail to start river mongodb177
java.lang.IllegalThreadStateException
at java.lang.Thread.start(Thread.java:684)
at org.elasticsearch.river.mongodb.MongoDBRiver.start(MongoDBRiver.java:220)
at org.elasticsearch.river.mongodb.StatusChecker.run(StatusChecker.java:26)
at java.lang.Thread.run(Thread.java:724)

It looks like database or collection name contains . see naming limitation in MongoDB

[1] - http://docs.mongodb.org/manual/reference/limits/#naming-restrictions

Answer 20 · 2013-12-19T18:04:42.000Z

I search in all database and there is no "." in database name and no "." in collection name.
But on collection DolibarrModeles there is a MAP REDUCE function so it create tmp.mr.DolibarrModules in oplog :-(

Answer 21 · 2013-12-19T19:45:22.000Z

Ok could you please provide a simple example to reproduce the issue?

Sent via BlackBerry by AT&T

-----Original Message-----
From: Herve Prot notifications@github.com
Date: Thu, 19 Dec 2013 10:04:42
To: richardwilly98/elasticsearch-river-mongodbelasticsearch-river-mongodb@noreply.github.com
Reply-To: richardwilly98/elasticsearch-river-mongodb reply@reply.github.com
Cc: Richard Louaprerichard.louapre@gmail.com
Subject: Re: [elasticsearch-river-mongodb] Indexing the whole db (#177)

I search in all database and there is no "." in database name and no "." in collection name.
But on collection DolibarrModeles there is a MAP REDUCE function so it create tmp.mr.DolibarrModules in oplog :-(

Reply to this email directly or view it on GitHub:
#177 (comment)

Answer 22 · 2013-12-19T20:18:21.000Z

I'm using moogose

This is a simple example of map / reduce that will be stocked in a new collection (collection name view_listMenu)

My model :

var moduleSchema = new mongoose.Schema({
_id: {type: String},
numero: Number,
family: String,
name: String,
description: String,
version: String,
const_name: String,
picto: String,
dirs: [String],
boxes: [mongoose.Schema.Types.Mixed],
module_parts: [],
const: [],
tabs: [],
langfiles: [],
depends: [],
requiredby: [],
config_page_url: [],
rights_class: {type: String},
rights: [mongoose.Schema.Types.Mixed],
menus: [mongoose.Schema.Types.Mixed],
enabled: {type: Boolean},
import: [mongoose.Schema.Types.Mixed],
expport: [mongoose.Schema.Types.Mixed],
_createdAt: {type: Date, default: Date.now}
});

var ModuleModel = mongoose.model('module', moduleSchema, 'DolibarrModules');

Answer 23 · 2014-09-02T07:59:33.000Z

i want to index all collection in one db and I make river config like that
curl -XPUT 'http://192.168.94.50:9200/_river/mongodbsoilbiota/_meta' -d '
{
"type": "mongodb",
"mongodb": {
"servers":[{"host":"192.168.94.51","port":29025}],
"db": "soilbiota",
"options": {
"import_all_collections": true
}
},
"index": {
"name": "soilbiota"
}
}'

then I found that the log
[2014-09-02 15:47:58,097][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodbsoilbiota
[2014-09-02 15:47:58,116][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.1] - hash[445c35a] - time[2014-07-30T14:08:26Z]
[2014-09-02 15:47:58,117][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [false], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [soilbiota], collection [mongodbsoilbiota], script [null], indexing to [soilbiota]/[soilbiota]
[2014-09-02 15:47:58,793][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.6.3
[2014-09-02 15:47:58,879][INFO ][org.elasticsearch.river.mongodb.Slurper] MongoDBRiver is beginning initial import of soilbiota.species_baoyu
[2014-09-02 15:47:58,883][INFO ][org.elasticsearch.river.mongodb.Slurper] Collection species_baoyu - count: 239
[2014-09-02 15:47:58,965][INFO ][org.elasticsearch.river.mongodb.Slurper] Number documents indexed: 239
[2014-09-02 15:47:58,965][INFO ][org.elasticsearch.river.mongodb.Slurper] MongoDBRiver is beginning initial import of soilbiota.species_detail
[2014-09-02 15:47:58,967][INFO ][org.elasticsearch.river.mongodb.Slurper] Collection species_detail - count: 74
[2014-09-02 15:47:58,988][INFO ][org.elasticsearch.river.mongodb.Slurper] Number documents indexed: 74
[2014-09-02 15:47:58,988][INFO ][org.elasticsearch.river.mongodb.Slurper] MongoDBRiver is beginning initial import of soilbiota.species_images
[2014-09-02 15:47:58,989][INFO ][org.elasticsearch.river.mongodb.Slurper] Collection species_images - count: 34
[2014-09-02 15:47:58,994][INFO ][org.elasticsearch.river.mongodb.Slurper] Number documents indexed: 34
[2014-09-02 15:47:58,994][INFO ][org.elasticsearch.river.mongodb.Slurper] MongoDBRiver is beginning initial import of soilbiota.species_picture
[2014-09-02 15:47:58,995][INFO ][org.elasticsearch.river.mongodb.Slurper] Collection species_picture - count: 10
[2014-09-02 15:47:58,997][INFO ][org.elasticsearch.river.mongodb.Slurper] Number documents indexed: 10
[2014-09-02 15:47:58,997][INFO ][org.elasticsearch.river.mongodb.Slurper] MongoDBRiver is beginning initial import of soilbiota.species_sequence
[2014-09-02 15:47:58,998][INFO ][org.elasticsearch.river.mongodb.Slurper] Collection species_sequence - count: 308
[2014-09-02 15:47:59,031][INFO ][org.elasticsearch.river.mongodb.Slurper] Number documents indexed: 308

the results:
soilbiota
size: 615B (1.03ki)
docs: 0 (0)

but actually there is no documents index into es，could you please tell me that Is there something wrong ?

by the way mongo version: 2.6.3 elasticsearch version :1.2.2 river version: 2.0.1