feat: Optimize the logic of the HIncrby command with MergeOperator (Merge)
QlQlqiqi opened this issue · 16 comments
Which PikiwiDB functionalities are relevant/related to the feature request?
No response
Description
https://github.com/johnzeng/rocksdb-doc-cn/blob/master/doc/Merge-Operator.md
原来的 Hincrby 命令是先 get meta，然后 get data，然后 put data，现在是 get meta，然后 merge data。merge 还有一个好处就是不阻塞，肯定会比 put 快一点，当有 compact 或者 get 的时候，merge 才会被计算。但是在 HIncrby 命令中使用 Merge 有一个缺点：无法处理 overflow 的情况。因为原来会 get data，然后可以判断旧的数据和新的数据相加会不会溢出，但是现在将读旧数据的逻辑放 merge 中了，这块就不能处理溢出情况，因为无论如何，merge 不能处理失败必须返回 true。
这里的 Hincrby 命令指的是包括但不限于 Incrby 等 read-modify-write 命令。
Proposed solution
none
Alternatives considered
none
坏处：
- 处理不了以前可以判断的 overflow 的情况。
- HIncrby 命令无法立刻返回结果，所以需要修改相关测试。
Bot detected the issue body's language is not English, translate it automatically.
harm:
- Unable to handle overflow situations that could be judged before.
- The HIncrby command cannot return results immediately, so the relevant tests need to be modified.
wangshaoyi
Bot detected the issue body's language is not English, translate it automatically.
Wang Shaoyi
使用merge操作符，读性能应该也会有下降吧？
还有就是用了merge操作符，线上就不能做回滚了吧。
Bot detected the issue body's language is not English, translate it automatically.
When using the merge operator, the read performance should also be reduced, right?
Also, if the merge operator is used, rollback cannot be done online.
使用merge操作符，读性能应该也会有下降吧？ 还有就是用了merge操作符，线上就不能做回滚了吧。
- 因为 merge 可以让 incrby 这样的 read-modify-write 指令变快一点。我认为 merge 类似 put，就是记录一个 value + merge 标记，然后下次执行 Get 操作的时候，遇到有 merge 标记的 value，就会继续 next，然后直到遇到一个有确定值的 value（比如 put 操作），然后合并值，并设置值的 merge 标记。
- 我没明白你的意思，我感觉不影响回滚，因为不存在任何无法容忍的中间情况。
Bot detected the issue body's language is not English, translate it automatically.
When using the merge operator, the reading performance should also be reduced, right? Also, if the merge operator is used, rollback cannot be done online.
- Because merge can make read-modify-write instructions like incrby faster. I think merge is similar to put, that is, recording a value + merge mark, and then the next time you perform a Get operation, if you encounter a value with a merge mark, you will continue to next, and then until you encounter a value with a certain value (such as a put operation) ), then merge the values, and set the value's merge flag.
- I don’t understand what you mean. I don’t think it affects rollback because there are no intolerable intermediate situations.
使用merge操作符，读性能应该也会有下降吧？ 还有就是用了merge操作符，线上就不能做回滚了吧。
- 因为 merge 可以让 incrby 这样的 read-modify-write 指令变快一点。我认为 merge 类似 put，就是记录一个 value + merge 标记，然后下次执行 Get 操作的时候，遇到有 merge 标记的 value，就会继续 next，然后直到遇到一个有确定值的 value（比如 put 操作），然后合并值，并设置值的 merge 标记。
- 我没明白你的意思，我感觉不影响回滚，因为不存在任何无法容忍的中间情况。
- 对，是的。所以我理解get等单纯的查询操作就可能会读更多的sst文件。
2.merge操作符使用需要自己继承实现mergeoperator，定义如何做merge。如果我要回滚到之前的版本，之前的版本是没有实现mergeoperator的，那么读或者compaction时遇到merge类型如何处理？
Bot detected the issue body's language is not English, translate it automatically.
When using the merge operator, the reading performance should also be reduced, right? Also, if the merge operator is used, rollback cannot be done online.
- Because merge can make read-modify-write instructions like incrby faster. I think merge is similar to put, that is, recording a value + merge mark, and then the next time you perform a Get operation, if you encounter a value with a merge mark, you will continue to next, and then until you encounter a value with a certain value (such as a put operation) ), then merge the values, and set the value's merge flag.
- I don’t understand what you mean. I don’t think it affects rollback because there are no intolerable intermediate situations.
- Yes, yes. So I understand that simple query operations such as get may read more sst files.
- To use the merge operator, you need to inherit and implement the mergeoperator yourself and define how to do the merge. If I want to roll back to the previous version, the previous version did not implement mergeoperator, so how to deal with the merge type when reading or compacting?
使用merge操作符，读性能应该也会有下降吧？ 还有就是用了merge操作符，线上就不能做回滚了吧。
- 因为 merge 可以让 incrby 这样的 read-modify-write 指令变快一点。我认为 merge 类似 put，就是记录一个 value + merge 标记，然后下次执行 Get 操作的时候，遇到有 merge 标记的 value，就会继续 next，然后直到遇到一个有确定值的 value（比如 put 操作），然后合并值，并设置值的 merge 标记。
- 我没明白你的意思，我感觉不影响回滚，因为不存在任何无法容忍的中间情况。
- 对，是的。所以我理解get等单纯的查询操作就可能会读更多的sst文件。
2.merge操作符使用需要自己继承实现mergeoperator，定义如何做merge。如果我要回滚到之前的版本，之前的版本是没有实现mergeoperator的，那么读或者compaction时遇到merge类型如何处理？
- 是的，如果总是 merge 而不 get，那么确实会让下一次的 get 跨越多个 sst，但是 get 触发 1 次之后，下次的 get 就不会读取之前已经读取的 merge 了；如果长时间不读 merge，compact 的时候也会把这部分 merge 操作按照 get 的逻辑处理。也就是说，使用 merge 处的 read-modify-write 的一部分开销转嫁到了 get 操作处，但是因为 get 操作本身就在执行 get，merge+get 总体的开销我认为是小于 read-modity-write+get，而且如果连续执行多次 merge，那么这些 merge 大概率在一起，下次 get 的时候一起执行，总开销应该是比不用 merge 更低的。
- 版本回滚确实会有问题，可能导致异常情况。
Bot detected the issue body's language is not English, translate it automatically.
When using the merge operator, the reading performance should also be reduced, right? Also, if the merge operator is used, rollback cannot be done online.
- Because merge can make read-modify-write instructions like incrby faster. I think merge is similar to put, that is, recording a value + merge mark, and then the next time you perform a Get operation, if you encounter a value with a merge mark, you will continue to next, and then until you encounter a value with a certain value (such as a put operation) ), then merge the values, and set the value's merge flag.
- I don't understand what you mean. I don't think it affects rollback because there are no intolerable intermediate situations.
- Yes, yes. So I understand that simple query operations such as get may read more sst files.
- To use the merge operator, you need to inherit and implement the mergeoperator yourself and define how to do the merge. If I want to roll back to the previous version, the previous version did not implement mergeoperator, so how to deal with the merge type when reading or compacting?
- Yes, if you always merge without getting, then the next get will indeed span multiple SSTs, but after get is triggered once, the next get will not read the merge that has been read before; If the merge is not read for a long time, this part of the merge operation will be processed according to the logic of get during compaction. In other words, part of the overhead of using read-modify-write at merge is transferred to the get operation, but because the get operation itself is executing get, I think the overall overhead of merge+get is less than read-modity-write+get, and if merge is executed multiple times in a row, these merges are likely to be together and executed together the next time get is performed. The total overhead should be lower than without merge.
- Version rollback does have problems and may lead to abnormal situations.
使用merge操作符，读性能应该也会有下降吧？ 还有就是用了merge操作符，线上就不能做回滚了吧。
- 因为 merge 可以让 incrby 这样的 read-modify-write 指令变快一点。我认为 merge 类似 put，就是记录一个 value + merge 标记，然后下次执行 Get 操作的时候，遇到有 merge 标记的 value，就会继续 next，然后直到遇到一个有确定值的 value（比如 put 操作），然后合并值，并设置值的 merge 标记。
- 我没明白你的意思，我感觉不影响回滚，因为不存在任何无法容忍的中间情况。
- 对，是的。所以我理解get等单纯的查询操作就可能会读更多的sst文件。
2.merge操作符使用需要自己继承实现mergeoperator，定义如何做merge。如果我要回滚到之前的版本，之前的版本是没有实现mergeoperator的，那么读或者compaction时遇到merge类型如何处理？
- 是的，如果总是 merge 而不 get，那么确实会让下一次的 get 跨越多个 sst，但是 get 触发 1 次之后，下次的 get 就不会读取之前已经读取的 merge 了；如果长时间不读 merge，compact 的时候也会把这部分 merge 操作按照 get 的逻辑处理。也就是说，使用 merge 处的 read-modify-write 的一部分开销转嫁到了 get 操作处，但是因为 get 操作本身就在执行 get，merge+get 总体的开销我认为是小于 read-modity-write+get，而且如果连续执行多次 merge，那么这些 merge 大概率在一起，下次 get 的时候一起执行，总开销应该是比不用 merge 更低的。
- 版本回滚确实会有问题，可能导致异常情况。
get操作不会在rocksdb产生新key吧？所以除非你上层有cache，否则每次读都可能读多个文件。
Bot detected the issue body's language is not English, translate it automatically.
When using the merge operator, the reading performance should also be reduced, right? Also, if the merge operator is used, rollback cannot be done online.
- Because merge can make read-modify-write instructions like incrby faster. I think merge is similar to put, that is, recording a value + merge mark, and then the next time you perform a Get operation, if you encounter a value with a merge mark, you will continue to next, and then until you encounter a value with a certain value (such as a put operation) ), then merge the values, and set the value's merge flag.
- I don't understand what you mean. I don't think it affects rollback because there are no intolerable intermediate situations.
- Yes, yes. So I understand that simple query operations such as get may read more sst files.
- To use the merge operator, you need to inherit and implement the mergeoperator yourself and define how to do the merge. If I want to roll back to the previous version, the previous version did not implement mergeoperator, so how to deal with the merge type when reading or compacting?
- Yes, if you always merge without getting, then the next get will indeed span multiple SSTs, but after get is triggered once, the next get will not read the merge that has been read before. ; If the merge is not read for a long time, this part of the merge operation will be processed according to the logic of get during compaction. In other words, part of the overhead of using read-modify-write at merge is transferred to the get operation, but because the get operation itself is executing get, I think the overall overhead of merge+get is less than read-modity-write+get, and if merge is executed multiple times in a row, these merges are likely to be together and executed together the next time get is performed. The total overhead should be lower than without merge.
- Version rollback does have problems and may lead to abnormal situations.
The get operation will not generate a new key in rocksdb, right? So unless you have a cache in the upper layer, multiple files may be read each time.
使用merge操作符，读性能应该也会有下降吧？ 还有就是用了merge操作符，线上就不能做回滚了吧。
- 因为 merge 可以让 incrby 这样的 read-modify-write 指令变快一点。我认为 merge 类似 put，就是记录一个 value + merge 标记，然后下次执行 Get 操作的时候，遇到有 merge 标记的 value，就会继续 next，然后直到遇到一个有确定值的 value（比如 put 操作），然后合并值，并设置值的 merge 标记。
- 我没明白你的意思，我感觉不影响回滚，因为不存在任何无法容忍的中间情况。
- 对，是的。所以我理解get等单纯的查询操作就可能会读更多的sst文件。
2.merge操作符使用需要自己继承实现mergeoperator，定义如何做merge。如果我要回滚到之前的版本，之前的版本是没有实现mergeoperator的，那么读或者compaction时遇到merge类型如何处理？
- 是的，如果总是 merge 而不 get，那么确实会让下一次的 get 跨越多个 sst，但是 get 触发 1 次之后，下次的 get 就不会读取之前已经读取的 merge 了；如果长时间不读 merge，compact 的时候也会把这部分 merge 操作按照 get 的逻辑处理。也就是说，使用 merge 处的 read-modify-write 的一部分开销转嫁到了 get 操作处，但是因为 get 操作本身就在执行 get，merge+get 总体的开销我认为是小于 read-modity-write+get，而且如果连续执行多次 merge，那么这些 merge 大概率在一起，下次 get 的时候一起执行，总开销应该是比不用 merge 更低的。
- 版本回滚确实会有问题，可能导致异常情况。
get操作不会在rocksdb产生新key吧？所以除非你上层有cache，否则每次读都可能读多个文件。
是，你说的没错，我之前验证的时候忘记关闭 pika cache 了，我刚才验证了你说的是正确的，看来 merge 并不适用 hincrby 这块的场景。
Bot detected the issue body's language is not English, translate it automatically.
When using the merge operator, the reading performance should also be reduced, right? Also, if the merge operator is used, rollback cannot be done online.
- Because merge can make read-modify-write instructions like incrby faster. I think merge is similar to put, that is, recording a value + merge mark, and then the next time you perform a Get operation, if you encounter a value with a merge mark, you will continue to next, and then until you encounter a value with a certain value (such as a put operation) ), then merge the values, and set the value's merge flag.
- I don't understand what you mean. I don't think it affects rollback because there are no intolerable intermediate situations.
- Yes, yes. So I understand that simple query operations such as get may read more sst files.
- To use the merge operator, you need to inherit and implement the mergeoperator yourself and define how to do the merge. If I want to roll back to the previous version, the previous version did not implement mergeoperator, so how to deal with the merge type when reading or compacting?
- Yes, if you always merge without getting, then the next get will indeed span multiple SSTs, but after get is triggered once, the next get will not read the merge that has been read before. ; if the merge is not read for a long time, this part of the merge operation will be processed according to the logic of get during compaction. In other words, part of the overhead of using read-modify-write at merge is transferred to the get operation, but because the get operation itself is executing get, I think the overall overhead of merge+get is less than read-modity-write+get, and if merge is executed multiple times in a row, these merges are likely to be together and executed together the next time get is performed. The total overhead should be lower than without merge.
- Version rollback does have problems and may lead to abnormal situations.
The get operation will not generate a new key in rocksdb, right? So unless you have a cache in the upper layer, multiple files may be read each time.
Yes, you are right. I forgot to turn off pika cache when I verified it before. I just verified that what you said is correct. It seems that merge is not applicable to the hincrby scenario.