superfly/litefs

Improve deletion via LTX zero commit

Closed this issue · 6 comments

benbjohnson commented

Currently, LiteFS communicates database deletion to replicas by sending a "dropdb" frame in the stream. However, this doesn't work well for LiteFS Cloud as it acts as the data authority and doesn't receive a stream. LFSC also needs to retain old copies of the database in case of recovery in the future.

This issue is to track the integration of "zero commit" LTX files (superfly/ltx#46) into LiteFS & LFSC.

gc-victor commented

also needs to retain old copies of the database in case of recovery in the future

Sorry if this is too obvious, but I want to highlight this case.

In my use case, a new database can be recreated with the same name as a deleted database, but it has no relation to the old one except for using the same name.

Case:
DB X is created
DB X is deleted -> it is stored as a deleted DB
DB X is created (without any relation with the old DB other than the name)
DB X is deleted -> it is stored as a deleted DB

benbjohnson commented

The filename is the only identifier for the database in LiteFS so the two databases are related in that sense. On the LFSC side, you'd still see an entry for the database after deletion but it wouldn't show up on the LiteFS side if the database size was set to 0 in the most recent transaction.

gc-victor commented

If I delete and recreate a database with the same name, will it cause any problems? And if I delete it again, will it affect the previous deletion?

benbjohnson commented

No, it shouldn't cause any problems. With LiteFS, a transaction represents a specific set of bytes that make up the database at a given point in time. By making a zero-length database represent a deletion, it's just another transaction from LiteFS' perspective.

So internally, the TXID will keep incrementing even though you've created/deleted multiple times. e.g.

  • TXID 1: Create database.
  • TXID 2: Insert row.
  • TXID 3: Delete database. File no longer visible via LiteFS FUSE mount.
  • TXID 4: Create database again.
  • 👍2
benbjohnson commented

@gc-victor I implemented soft delete so you can actually delete databases while connected to LiteFS Cloud (#395). You can test it using the Docker image for the PR:

COPY --from=flyio/litefs:pr-395 /usr/local/bin/litefs /usr/local/bin/litefs
  • 🎉1
gc-victor commented

@gc-victor I implemented soft delete so you can actually delete databases while connected to LiteFS Cloud (#395). You can test it using the Docker image for the PR:

COPY --from=flyio/litefs:pr-395 /usr/local/bin/litefs /usr/local/bin/litefs

Great! I will test it next week.