Query fails if one or more cluster nodes do not have records in the set
Closed this issue · 13 comments
query = CLIENT.query(NAMESPACE, SET);
stream = query.foreach();
/*
Get list of all avialable keys
*/
stream.on('error', (error) => {
throw error;
});
stream.on('data', (record) => {
console.info('data', record);
console.info('key', record.key.key);
});
stream.on('end', () => {
console.log('done!');
process.exit(0);
});
Receiving error - AerospikeError: Record does not exist in database. May be returned by read, or write with policy Aerospike.policy.exists.UPDATE.
error encountered in promise chain => { [AerospikeError: Record does not exist in database. May be returned by read, or write with policy Aerospike.policy.exists.UPDATE]
name: 'AerospikeError',
code: 2,
command:
QueryCommand {
client:
Client {
domain: null,
_events: {},
_eventsCount: 0,
_maxListeners: undefined,
config: [Object],
as_client: AerospikeClient {},
connected: true,
captureStackTraces: false },
args: [ 'sms_data', 'some_set', [Object], undefined ],
captureStackTraces: false,
key: undefined,
stream:
RecordStream {
aborted: false,
client: [Object],
_events: [Object],
_eventsCount: 3 } },
func: 'as_query_parse_records_async',
file: 'src/main/aerospike/aerospike_query.c',
line: 237,
inDoubt: false }
Although the data is present in the namespace and set.
query result :
aql> select * from sms_data.some_set;
+-----------------------------------------------------+----------+
| 0 | name |
+-----------------------------------------------------+----------+
| MAP('{"dummy":[{"x":"dgjasgdj"}], "name":"Vidur"}') | "Khanna" |
+-----------------------------------------------------+----------+
@khannavidur, as far as I know, the only condition under which Query#foreach results in an "record not found" error, is if the set that you passed in Client#query does not exist. I do see that you use the same namespace and set in the AQL query, though. So I'm not really sure what might be causing this, yet.
As you can see in the error itself, I am sending the same namespace and set as in aql.
error encountered in promise chain => { [AerospikeError: Record does not exist in database. May be returned by read, or write with policy Aerospike.policy.exists.UPDATE]
name: 'AerospikeError',
code: 2,
command:
QueryCommand {
client:
Client {
domain: null,
_events: {},
_eventsCount: 0,
_maxListeners: undefined,
config: [Object],
as_client: AerospikeClient {},
connected: true,
captureStackTraces: false },
args: [ 'sms_data', 'some_set', [Object], undefined ],
captureStackTraces: false,
key: undefined,
stream:
RecordStream {
aborted: false,
client: [Object],
_events: [Object],
_eventsCount: 3 } },
func: 'as_query_parse_records_async',
file: 'src/main/aerospike/aerospike_query.c',
line: 237,
inDoubt: false }
My code snippet is available at - https://gist.github.com/khannavidur/b1b81125bb2dd61d6012e6c5bc64f491
It works on my mac with Aerospike Query Client Version 3.15.1.2
but the foreach doesn't work on my linux box with Aerospike Query Client Version 3.15.0.4
Is it a version based issue? Because all the aerospike config file matches for both of them. The issue persists with both storage in memory and persistence mode.
Your code looks correct and works for me as expected.
The two version numbers you state are not client version numbers. They seem to be server version numbers. But actually, these two specific server versions do not exist. Did you mean 3.15.1.4 and 3.15.0.2 instead? In any case, I do not think that the server version or client OS makes any difference in this case.
At the moment, I really don't know, what could be the issue here. Are the client versions and Node.js version on your two Mac and Linux environments the same? And you are executing the exact same another_aerospike_example.js script on both machines?
Never mind about the two version numbers: I see now that you are referring to the AQL client version numbers. But what Aerospike Node.js Client version are you using?
Node JS Client version as per its package.json - "version": "3.2.0" on both the machines and yes I am using the same script to test on both the machines.
I was actually trying to test my local script on my AWS boxes. The only difference is on my local it is set up using vagrant and the cluster is just one node and on the cloud instance, it is running in a 3 node cluster. I am connecting to the DNS instead of a particular node' IP in general and when I try to debug using any one of the node's IP directly, I am still facing the same issue. Is there a configuration I am missing which happens by default in the vagrant installation and not in the linux one?
Also,
Aerospike Community Edition build 3.15.0.1 - on linux
Aerospike Community Edition build 3.15.1.4 - on mac
My bad - the Aerospike Query Client Version is 3.15.0.3 and not 3.15.0.4 on the linux box
Ah, I see what's happening now! You are running a 3 node cluster but you are only creating a single record in your test. Assuming that replication factor is set to 2, that means a copy of the record will be stored on 2 out of the 3 cluster nodes. The 3rd node will not have any records in this particular set.
I tried to replicate this scenario and when I run the query, I can see a log message like this on one of the three nodes:
Apr 04 2018 07:47:25 GMT: WARNING (scan): (scan.c:273) scan msg from 192.168.33.1:60049 has unrecognized set set3
("set3" is the set name I was using to test.)
The status message sent by the node on which the set name is not yet registered, seems to cause the client to return the "record not found" error, even though one or more of the remaining nodes may actually send back records.
As soon as you create additional records such that each cluster node as has least one record in this set (could be a replica), then the error goes away.
This seems to be an error in the Aerospike C client, which the Node.js client uses internally. I will follow up on this with the C client developer.
Thanks!
Few questions (ignore if stupid ignorance on my end as I am new to aerospike)
- The replication factor is set to 2, does that not mean besides the master copy of data on one node, the replica happen on 2 other nodes?
- since the connection is not getting closed by the client I assume, why does it search for .foreach operation on a separate node all together?
- The same issue is persisting if I am connecting to one particular node using its IP instead of the cluster' DNS, how so?
- .get, .exists works just fine shouldn't that be a problem too or is it an explicit bug for query.foreach ?
Also, thank you so much for the swift replies.
Quick update: I have submitted a fix for the C client. Once a new version of that client gets released, I can release a new Node.js client for this as well. Note, meanwhile, that using Client#scan instead of Client#query works as expected.
The replication factor is set to 2, does that not mean besides the master copy of data on one node, the replica happen on 2 other nodes?
Replication factor 2 means that a copy of the record gets stored on two cluster nodes, i.e. one master copy and one replica.
since the connection is not getting closed by the client I assume, why does it search for .foreach operation on a separate node all together?
For query operations, the client needs to send the request to every node in the cluster since it does not know which node might have records that match the query filter.
The same issue is persisting if I am connecting to one particular node using its IP instead of the cluster' DNS, how so?
Even if you are just connecting to a single node via it's IP, the client will find out about the rest of the cluster from this initial seed node and connect to all the cluster nodes.
.get, .exists works just fine shouldn't that be a problem too or is it an explicit bug fir query.foreach ?
For single-key operations, the client determines which node holds the master copy of that record based on the key digest, and then sends the request to only that particular node.
Thank you! Looking forward to the new release :)
This issue is fixed in aerospike@3.3.0.