Multi-thread fetch using transaction on separate database goes in LMDBException: invalid argument
Closed this issue · 7 comments
I have two databases attached to the same Env: relations and lookup, I need to first fetch the values from the lookup (key -> List of ids) and then use the ids to fetch from relations another list of IDs.
I wrote the transaction in the external try-with-resources because I want to keep the same consistency for the two get, however I'm fetching data from two different databases.
Here the piece of code:
@Override
public List<Identifier> fetchRelations(final String organisationOriginalID) {
System.out.println("fetch relations organisation...");
List<Identifier> relations = new ArrayList<>();
List<Identifier> uniqueIds = new ArrayList<>();
final Identifier identifier = new Identifier(organisationOriginalID, OrganisationAttributeNames.LABEL);
final byte[] serialisedIdentifier = bytes(identifier.toString());
try (Transaction tx = env.createReadTransaction()) {
try (Database lookupDB = env.openDatabase(LMDB_DATABASE_LOOKUPS)) {
byte[] value = lookupDB.get(tx, serialisedIdentifier);
if (value != null) {
List<Identifier> ids = deserialize(value);
uniqueIds.addAll(ids);
}
if (!CollectionUtils.isEmpty(uniqueIds)) {
Database relationDB = env.openDatabase(LMDB_DATABASE_RELATIONS);
uniqueIds.forEach(id -> {
byte[] value = relationDB.get(tx, bytes(id.toString()));
if (value != null) {
relations.addAll(deserialize(value));
}
});
}
}
return relations;
}
When I run it singe or multi-thread, the behaviour is a bit strange. The database is not written from other processes.
I tried several approaches, but I could not really understand whether I'm doing something wrong (not closing resources) or there is any other problems.
- when I use read Transaction, namely
byte[] value = lookupDB.get(tx, serialisedIdentifier);I get the following exception (pointing at the line where thegetis called):
Exception in thread "main" org.fusesource.lmdbjni.LMDBException: Invalid argument
at org.fusesource.lmdbjni.Util.checkErrorCode(Util.java:44)
at org.fusesource.lmdbjni.Database.get(Database.java:202)
at org.fusesource.lmdbjni.Database.get(Database.java:193)
at org.fusesource.lmdbjni.Database.get(Database.java:186)
at org.entitycooking.sample.hal.operations.OrganisationLMDBOperations.fetchRelations(OrganisationLMDBOperations.java:91)
at org.entitycooking.integration.mapping.EntityMapper.fetchGraph(EntityMapper.java:52)
at org.entitycooking.algorithm.training.EntityMatcherTrainer.extractFeaturesFromExample(EntityMatcherTrainer.java:165)
at org.entitycooking.algorithm.training.EntityMatcherTrainer.lambda$extractFeatures$4(EntityMatcherTrainer.java:153)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
- If I remove the transaction at all, then the application run for a bit longer time, until it crash badly:
java(35152,0x700005847000) malloc: *** error for object 0x7feb8bebdd40: double free
*** set a breakpoint in malloc_error_break to debug
Any idea, suggestion?
Thank you in advance
For the issue 1, I think the problem comes when I open the transaction before opening the database (in both single and multi-thread). Does this make sense?
Always open Env and Database as the application starts and keep them open. You maintain consistency by using the same transaction for both databases.
If you need multi-threading and sharing same transaction you need to set the NO_TLS option.
Never close Env or Database before an open transaction, or you will get a SIGSEGV.
Thanks!
The transaction is not shared among threads, it's created/destroyed within the method, so it's good like that.
Are this information somewhere in the documentation? If not I can help you updating it ;-)
The lmdb documentation is valid for lmdbjni aswell.
INFO: org.fusesource.lmdbjni.LMDBException: Invalid argument
Am getting this exception while setting maxreaders in LMDBJNI...
Env env = new Env();
env.open(LMDB_DIR);
env.setMaxReaders(150);
db = env.openDatabase("sub");
Env is created and initialized in the given path.. But I cannot set maxreaders in LMDBJNI as exception arises. If I skip setting maxreaders, then also am getting exception
org.fusesource.lmdbjni.LMDBException:MDB_READERS_FULL : Environment maxreaders limit
reached
at org.fusesource.lmdbjni.Util.checkErrorCode
at org.fusesource.lmdbjni.Env.createTransaction
at org.fusesource.lmdbjni.Env.createReadTransaction
at org.fusesource.lmdbjni.Database.get(Database.java:170)
Kindly resolve this issue as this is banging my head for more than a week... Since am new to java programming... Thanks in advance
@Johny-Kirth it's perhaps recommended to use https://github.com/lmdbjava/lmdbjava instead.
Which exception do you get when setting the max readers?
@Johny-Kirth it's perhaps recommended to use https://github.com/lmdbjava/lmdbjava instead.
Which exception do you get when setting the max readers?
INFO: org.fusesource.lmdbjni.LMDBException: Invalid argument
Sep 23, 2019 7:31:29 PM com.adventnet.wms.servercommon.logging.ConsoleLogger write
INFO: at org.fusesource.lmdbjni.Util.checkErrorCode(Util.java:44)
Sep 23, 2019 7:31:29 PM com.adventnet.wms.servercommon.logging.ConsoleLogger write
INFO: at org.fusesource.lmdbjni.Env.setMaxReaders(Env.java:347)