Consistency / Transactions / Integrity

[KinanBab]

We currently have several sources of potential inconsistency:

1. No synchronization between tables and views ---> inconsistency when indices are updated during a concurrent table update (e.g. #118)
2. No synchronization between copies of the same record being moved/deleted/modified.
3. No synchronization between tables and Rocksdb Indices

Potential solution

1. Table-level locks in rocksdb connection or similar, which are acquired even when reading from views that are based on the tables. We use rocksdb transactions!

Transactions/atomicity:

1. Failure of sub-operations of a single logical operations (e.g. cascading variable ownership delete, updating mutliple copies of the same shared row) does not rollback.

Integrity:

1. Uniqueness of PK is not enforced and results in replace: consistency issue with views that do not see the replace (#134)
2. Integrity of FK is not enforced.

[KinanBab]

Cleanups for after the deadline:

Integrity (Consistency as in ACID):

• Enforce FK integrity
• NOT NULL, default, auto increment, unique

Transactions:

• Implement multi-statement transactions with REPEATED READS: Batch writes to dataflow across transaction. Read from batch and dataflow within transaction. Version numbers?

Consistency (as in linearizability vs eventual):

• Revisit deletes of dependent tables with changes to index.
• Optimization: improve locking duration of in memory index when used to handle insert.

Isolation:

• Change SK format: Sk -> PK (but not shard), then PK index gives us shard (allows us to lock PK index record for writes)
• Unlock unmatching records as soon as they are discarded during scans (optimization to match myrocks)
• Separate out reading with snapshot vs without snapshot into different functions in RocksdbTransaction, and RocksdbIndex.
• Read most recent version of a record after iterating (for both records and indices) (optimization to reduce write skew and match myrocks)

[KinanBab]

• Recover partitioned dataflow