range delete
rvagg opened this issue · 8 comments
dupe of Level/levelup#23, needs to be done mostly here rather than there.
yeah, sure, but that's a very inefficient mechanism since for each key you're crossing the thread boundary twice via libuv (once for the iterator->JS 'data' and then once for the JS del()->leveldb) and you're also doing a memcpy() of the key both ways as well. The reason I think this should be in core is that it can be done on the same thread, directly with an iterator, no memcpy() and no JS involved, very efficient. This is one of those cases where it makes a lot of sense to do it in core because the efficiency gain is huge.
So, what % improvement will a benchmark show?
if this took iterator arguments, like limit: 1 you could use it to delete the last or first item in a range,
and use it to produce things like top 100 lists.
Could really benefit from this. Bumping into a case where I am managing capped timeseries and my deletes can't keep up with my writes.
Marking as "help wanted", a good project for someone wanting to get their hands dirty. I don't think this would ever be exposed at the LevelUP layer but features like this that are specific to back-ends should be exposed in the LevelDOWN layer so you can access them if you need them. level-range-delete could even detect the presence of a db.db.rangeDelete() function and use that if it exists.
Oh, while this has surfaced again,
I should mention something that I've realized recently. deletes are actually writes.
You have to write a record to the database that says "please forget X"
So if you are only deleting something to save space, maybe an idea would be to just not delete it? eventually the deletes should be compacted away, but short/mid term they are extra records.
Having a range-delete is really encouraging adding a lot of extra records to a database,
but telling the user they are being removed.