Support for hmsetnx
SumanthNarendra opened this issue · 35 comments
Currently redis supports hmset. It would be really useful to have "nx" functionality built into hmsetnx.
Sumanth and I worked on a patch for this. Please be gentle and provide any feedback since this is the first time we've looked at the code base:
sammyyu@dc506bd
+1
+1
+1
Since this request seems to have exponentially increasing interest, I added HMSETNX as a loadable module at https://github.com/mattsta/krmt
Example usage:
127.0.0.1:6379> config set module-add /home/matt/repos/krmt/hmsetnx.so
OK
127.0.0.1:6379> hmsetnx bob field1 value1 field2 value2 field3 value3
(integer) 1
127.0.0.1:6379> hmsetnx bob field1 value1 field2 value2 field3 value3
(integer) 0@mattsta thanks for providing this as modules, however there is a specific reason why Redis has no a plugin-system by design, that is, we don't want to debug code which is non stable because of third party plugins, and, users will jump creating stuff which is often not needed and we end with an ecosystem of slightly different instances / APIs without a good reason.
So who is using plugins, is on his/her own, I'm not going to consider bug reports / crashes / issues if there is a plugin loaded. For the same reason, please could you make sure (if not already) that krmt reports the bug report with a very visible warning that this instance had krmt modules loaded? Thanks!
About the HMSETNX feature itself, given all the +1, it looks like this is a useful functionality. To see that the feature is popular is great, but I would love if some user that +1ed this could share some use case. Thank you.
I've got one right now. I'm using a hash instead of a set to turn this structure:
args:{TASK}= set({ARGS1}, {ARGS2}, ...)
eta:{TASK}:{ARGS1} = N
eta:{TASK}:{ARGS2} = N
...
into
eta:{TASK} = hash(
{ARGS1}: N,
{ARGS2}: N,
...
)
Having HMSETNX means I'd be able to in effect replicate what I like about SADD in the previous structure. I can add a bunch of argss with Ns to my task's hash without overwriting anything that's there, and then also know how many new args were written.
The current structure requires me to add the argss to the set, and then if there were any new additions, call MSETNX with the list of new additions (which I've got to get as well). HMSETNX would mean I can do the whole job in one command.
Apologies for the variable soup, hope that makes some amount of sense. It'd be really helpful to have this.
Since I still don't see this in the latest redis, here's my use case:
- I have multiple threads in a system, each working on parallel stream of information that needs to be initialized to some value, then incremented by multiple threads. Consider an email signature "X" with attributes:spam_score, source etc.
- Multiple threads need to do "HMSET X spam_score:50 source:"www.example.com", and then "HINCRBY X spam_score 10" or (-10) as the case may be.
- Multiple threads can get the same email (as is the case in spam), which will cause them to call HMSET, this will overwrite any HINCRBYs other threads might have issued.
The only alternative I have currently is to do multiple HSETNX for each attribute that has a parallel HINCRBY.
We've recently updated to 3.0 on our systems. I've updated from 3.0 upstream to include the updated hmsetnx patch along with some unit tests
sammyyu@02e58ce
Hope this helps
Running across this as well and HMSETNX would be very helpful. +1!
I needed this, so I worked out a way to do it with Lua. Of course native would be better in a future version if possible.
if redis.call('exists', KEYS[1]) == 0 then
redis.call('hmset', KEYS[1], unpack(ARGV))
return 1
end
return 0
👍
While this can be done with LUA it would be nice if this were done natively as serializing
args for Redis for HMSET is best handled by client-libraries.
but I would love if some user that +1ed this could share some use case. Thank you.
My personal use-case is a task which sets up stats about itself when queued and deletes them when it has finished running. In some-cases I would like to guarantee task uniqueness - hmsetnx would allow for the mutex + stats setup to happen in a single redis call, where as now it requires the LUA script.
+1
Adding to the use cases, I have the following use case:
Hashmap holds information that's discovered by a crawler that runs every so often. There are fields within the hash I store that will need updating, but a HMSET will clobber my first-seen field. HMSETNX would add only new information, but not mutate existing info.
I suppose there's a distinction between HMSETNX where the hashmap won't be changed at all if it exists, and another that would instead merge the new mapping with the old hash, keeping the old value where there's a conflict. I feel the second is possibly more useful?
+1
I think HMSETNX is more friendly to a redis cluster than "MSETNX".
I have the following use case:
I worked in an Electronic Commerce Company, the stock system is number sensitive. We use redis as distributed lock to make stock modify of one SKU(stock keeping unit) not be concurrent in a distribution enviroment.
We do the check like this:
for (Sku sku : skuList){
if (redis.setnx("lock_sku_" + sku.id, "") == 0) {
throw new SkuModifyException();
}
}Stock modify request comes from several ways, one is that user upload an excel like:
| skuid | stock |
|---|---|
| 111 | 123 |
| 112 | 2342 |
| ... |
One day, user upload an excel contains 100,000 records, the setnx is called by 100,000 times, it's inefficiency.
“MSETNX” may help?
No, we can't use redis deployed by ourselves but the one supplied by a team called jimdb. It's just a package of redis api, and an implementation of "REDIS-CLUSTER", redis data is sliced to multi redis instance by keys, it's difficult to support operation like "MSETNX".
HMSETNX use one key and multi fields, so I think it's friendly to "REDIS-CLUSTER".
I have an iteration which detect unique words and its ctypes and counts.
...
if( isset($wordHash[$word]) ){
$wordHash[$word]['count']++;
}
else{
isset($wordHash[$word]) ?: $wordHash[$word] = [
'id' => $word_id++,
'count' => 1,
'type' => $word_ctype
];
}
...
If i want to do this with redis;
if(!EXISTS tokens:$word)
HMSET tokens:$word id $word_id++ count 1 type $word_ctype
else
HINCRBY tokens:$word count 1
end
with HMSETNX command these operation could be much more clear.
HMSETNX tokens:$word id $word_id++ type $word_ctype
HINCRBY tokens:$word count 1
+1
+1
+1. It's been 6 years, how is this still not implemented? Seems like such a self-evident feature.
TBH I ended up just using the redis lua interface and it's a pleasure to use and perfect for this and many other situations like this one. I understand if this functionality has not been prioritized.
@johanbrandhorst I assume you end up sending 2 separate commands, so things are no longer atomic? I need (or at least, strongly desire) the atomicity that an hmsetnx command would offer.
Anything that happens inside a Lua script is done atomically, so you maintain atomicity.
@johanbrandhorst Game-changer alert! I wasn't aware scripts were atomic. Thanks for the info!
+1
Need this implemented!!! without HMSETNX, I have to query before HMSET, it's not efficient at all.
In my case, most of time the key should not exist, so the query is very likely a waste of effort.
+1
So it looks like there are three cases in this thread:
- Be able to replace an existing hash, IE DEL + HSET
- Update fields, only if those fields don't already exist
- Update fields, but only if they already exist (from a tweet)
The command could look like
HSETEX key [NX|XX] [DEL] FIELDS [field value ...]
XX: Only update elements that already exist. Never add elements.
NX: Don't update already existing elements. Always add new elements.
DEL: Delete the key if it already exists.
It would diverge from ZADD, which doesn't allow flags to be
This seems reasonably popular enough we could have someone implement it.
@redis/core-team Thoughts on implementing this?
I'm a bit uncomfortable with the DEL feature, what if the existing key is not a hash, do we want to return a WRONGTYPE error, or override it?
I suppose that in some way, it's similar to what SUNIONSTORE does (overrides the dest key regardless of it's type).
Given that it's a popular request (specifically the XX and NX features i guess), i think we should indeed implement it (i see Salvatore thought so too).
We need to carefully define the response type, maybe like ZADD, it also needs the CH flag?
if we consider that, maybe GT and LT too (hash does have HINCRBY feature)
what seems straight forward to me is just the XX, NX and CH flags.
Hi @oranagra @madolson I am not sure if XX, CH, or other flags are needed currently. Just for this issue. I think it is easy to solve if HSETNX command supports multi fields and it is compatible with old version, just like HSET command supports multi fields.
I submit a PR #8304, do i miss somethings?
@ShooterIT I agree it would resolve this issue, but I would still prefer to also release this with a more generic HMSET with flags. It's a pretty common pattern, to have a generic add with a specific implementation. Since it will likely involve refactoring into a generic function, I wouldn't decouple them.
I agree, I rather extend the capabilities of HSET than improve HSETNX (same as SETNX was abandoned in favor of SET NX).
But on the other hand the HSET command is taken, and impossible to extend, and improving HSETNX is so straight forward.
I'm also not sure I'm a fan of the FIELDS argument.
Would like to hear @itamarhaber
I tend to agree w/ @ShooterIT's reasoning, providing we don't need XX (haven't seen/don't remember that tweet) or CH (are there even any requests for that?), and that DEL is achievable with a MULTI/EXEC.
However, if we want to be ahead of the curve (give or take a decade), I can imagine people finding the use cases for the functionality. I would propose, however, that HSETEX should be more like BITFIELD's in the sense that it can accept multiple ops, so: HSETEX key [DEL] [CH] NX|XX|AX field value [...] (AX is always, i.e., XX || NX).
Furthermore, like Oran, the DEL smells iffy, especially because it isn't related to field-value tuples. I'd rather not have it.
@itamarhaber so you mean that DEL and CH can only be specified once (they are about the command, not the fields), and then we have a repetition of NX/XX/AX <field> <value>?
so we can distinguish the DEL and CH from a field / value since we know the fields are present only after the first NX/XX/AX modifier.
This gets more and more complicated, and compared to the really straight forward variadic HSETNX, i'm not sure what's the right move here.