WIP: propose the document level ownership

[imotai]

Motivations

1. solve the problem of data exchange between accounts

• Would you like to implement this feature? [y/n]

Solution

[jingchen2222]

Discussion Kick off

• goal: record level permission management @jingchen2222 @imotai
• architecture - data model @imotai @jingchen2222
• primary key generator (logic, size limit, order)
  • index key generator @jingchen2222
  • consider supporting aggregation by address @jingchen2222
  • additional schema: txid
  • implementation @jingchen2222
  • unified address design @imotai
• key optimization (future work)

[jingchen2222]

Design Doc: Document Database Storage

Background

Goal:

1. solve the problem of data exchange between accounts
2. record level permission management.
3. design primary key generator (logic, size limit, order)
4. design index key generator
5. support CreateDocument/UpdateDocument/DeleteDocument API based on KV operations
6. key optimization (future work)

Issue:

[propose the document level ownership #271](#271)

Structure

Database → Collection → Document

flowchart LR

A[Developers] --> B[database]
B --> C[colection 1]
C --> E[document 1]
C --> F[document 2]
C --> G[document 3]
B --> D[colection 2]
D --> H[document 4]
D --> I[document 5]
J[User] --> |"(Document, Signature, PublicKey)"|D

Loading

ID

ID Type	Size	Description
DbId	20	DbId := DB3Address
TxId	32	transaction id
AccountId	20	AccountId := DB3Address
BlockID	8	Current block height
MutationID	4	Index of mutation in the current block.
PutEntryIdx	4	Index of document in current database mutation
PutEntryID	16	PutEntryID := BlockID + MutationID + PutEntryIdx
CollectionID	16	CollectionID := PutEntryID
DocumentEntryID	16	DocumentEntryID := PutEntryID
DocumentID	33	DocumenID := DocumentEntryType+CollectionID + DocumentKeyID
IndexFiledID	4	Index of the index filed in the collection
IndexID	KeyBytes + 54	IndexId := IndexEntryType+CollectionID + IndexFiledID + KeyBytes + DocumentID

Document serialization

1. rust https://docs.rs/bson/latest/bson/
2. typescript https://github.com/mongodb/js-bson

Database & Collection Op

#297

Document OP

1. CreateDocument

sequenceDiagram
    Client->>+Node: 1.1 CreateDocument(document)
    Node->>+DocumentImpl:1.3 SubmitMutation(WRITE_DOCUMENT, document)
    Node-->>-Client: 1.4 Response(status, msg)
    DocumentImpl->>DocumentImpl: 2.1 documentId = GenerateDocumentId(blockId, autoId)
    DocumentImpl->>+KVStore: 2.2 InsertKV(documentId, txid, document)
    KVStore-->>-DocumentImpl: 2.3 Response(status, msg)
    DocumentImpl-->>DocumentImpl: 2.4 index_key_pairs = GetIndexKeyPairs(db, collection, document)
    loop Create Indexes: index_key_pairs.foreach
    DocumentImpl-->>DocumentImpl: 2.5.1 GenerateIndexId(db, collection, documentId, index, key)
    DocumentImpl->>+KVStore: 2.5.2 InsertKV(indexId, txid, documentId)
    KVStore-->>-DocumentImpl: 2.5.3 Response(status, msg)
    end
    DocumentImpl-->>-Node: 2.6 Response(status, msg, documentId)
    

Loading

2. UpdateDocument

sequenceDiagram
    Client->>+Node: 1.1 UpdateDocument(documentId, document)
    Node->>+DocumentImpl:1.2 SubmitMutation(UpdateDocument, documentId, document)
    Node-->>-Client: 1.3 Response(status, msg)
    DocumentImpl->>+KVStore: 2.1 GetKV(documentId)
    KVStore-->>-DocumentImpl: 2.2 Response(status, msg, address, old_document)
    DocumentImpl-->>DocumentImpl: 2.3 CheckOwnership(Signature, PublicKey, address)
    DocumentImpl-->>DocumentImpl: 2.4 modified_index_key_pairs = GetModifiedIndexKeyPairs(db, collection, old_document, document)
    DocumentImpl->>+KVStore: 2.5 Update documet: InsertKV(documentId, txid, document)
    KVStore-->>-DocumentImpl: 2.6 Response(status, msg)
    DocumentImpl->>DocumentImpl: 2.7 (indexes, keys) = CheckModifiedIndexs(document, index_keys)
    loop Update Indexes: modified_index_key_pairs.foreach(index, key)
    DocumentImpl-->>DocumentImpl: 2.8.1 indexId = GenerateIndexId(db, collection, index, key, documentId)
    DocumentImpl->>+KVStore: 2.8.2 InsertKV(indexId, txid, documentId)
    KVStore-->>-DocumentImpl: 2.8.3 Response(status, msg)
    end
    DocumentImpl-->>-Node: 2.9 Response(status, msg)
    
  
    

Loading

2. DeleteDocument

sequenceDiagram
    Client->>+Node: 1.1 DeleteDocument(documentId)
    Node->>+DocumentImpl:1.2 SubmitMutation(DeleteDocument, documentId)
    Node-->>-Client: 1.3 Response(status, msg)
    DocumentImpl->>+KVStore: 2.1 GetKV(documentId)
    KVStore-->>-DocumentImpl: 2.2 Response(status, msg, index_keys, txid)
    DocumentImpl-->>DocumentImpl: 2.3 CheckOwnership(Signature, PublicKey, txid)
    DocumentImpl->>+KVStore: 2.4 Delete documet: DeleteKV(documentId)
    KVStore-->>-DocumentImpl: 2.5 Response(status, msg)
    DocumentImpl-->>DocumentImpl: 2.6 index_key_pairs = GetIndexKeyPairs(db, collection, document)
    loop Delete Indexes:  index_key_pairs.foreach(index, key)
    DocumentImpl-->>DocumentImpl: 2.7.1 GenerateIndexId(db, collection, index, key, documentId)
    DocumentImpl->>+KVStore: 2.7.2 DeleteKV(indexId, txid)
    KVStore-->>-DocumentImpl: 2.7.3 Response(status, msg)
    end
    DocumentImpl-->>-Node: 2.8 Response(status, msg)
    
  
    

Loading

4. GetDocument

sequenceDiagram
    Client->>+Node: 1.1 GetDocumentById(documentId)
    Node->>+DocumentImpl:1.3 GetDocumentById(documentId)
    DocumentImpl->>+KVStore: 2.1 BatchGetKey(documentId)
    KVStore-->>-DocumentImpl: 2.2 Response(status, msg, document)
    DocumentImpl-->>-Node: 2.5 Response(status, msg)
    Node-->>-Client: 1.4 Response(status, msg)
    Client->>+Node: 2.1 GetDocuments(index, key)
    Node->>+DocumentImpl:2.2 GetDocuments(index, key)
    DocumentImpl->>DocumentImpl: 2.3 index_range = GenerateIndexRange(db, collection, index, key)
    DocumentImpl->>+KVStore: 2.3 GetRange(index_range)
    KVStore-->>-DocumentImpl: 2.4 Response(status, msg, documents)
    DocumentImpl-->>-Node: 2.5 Response(status, msg, documents)
    Node-->>-Client: 2.6 Response(status, msg, documents)
    

Loading

Implementation

[jingchen2222]

Discussion

@imotai

1. What is the relationship between collection and documents? 1:N or N:M?

Regarding to the rpc proto definition mentioned here, one document might be under multiple collections? Will we define and implement collection in the same way?

2. Will we support Project, Namespace, Collection Group, in db3.

I find those concepts are used as defined in the google firestore.

[imotai]

Discussion

@imotai

1. What is the relationship between collection and documents? 1:N or N:M?

Regarding to the rpc proto definition mentioned here, one document might be under multiple collections? Will we define and implement collection in the same way?

2. Will we support Project, Namespace, Collection Group, in db3.

I find those concepts are used as defined in the google firestore.

1. 1:n
2. db3 just support database , collection and document

[jingchen2222]

RPC Interface

DB3 will design and implement a set of APIs which are similar to google firestore so that the developers who are familiar to firestore can transfer to DB3 easily.

Reference:

• https://github.com/googleapis/googleapis/tree/master/google/firestore
• https://github.com/googleapis/googleapis/blob/master/google/firestore/admin/v1/firestore_admin.proto

DB3 Admin API

This API provides several administrative services for DB3 store.

P0

API	Description
CreateDatabase	Create database and linked with address
CreateCollection	Define indexes
ListIndexes	Lists composite indexes.
GetIndex	Gets a composite index.
GetDatabase	Gets information about a database.
GetField	Gets the metadata and configuration for a Field.
ListFields	Lists the field configuration and metadata for this database.

P1:

API	Description
UpdateDatabase	Updates a database.
UpdateField	Updates a field configuration. Currently, field updates apply only to single field index configuration.
ExportDocuments	Exports a copy of all or a subset of documents from DB3 store
ImportDocuments	Imports documents into DB3 store. Existing documents with the same name are overwritten.
ListDatabases	List all the databases in the project.
CreateIndex	Creates a composite index. This returns a db.longrunninbg.Operation (similar to google.longrunning.Operation) which may be used to track the status of the creation. The metadata for the operation will be the type IndexOperationMetadata
DeleteIndex	Deletes a composite index.

DB3 Document Op API

P0: API

API	Description
CreateDocument	Creates a new document.
GetDocument	Gets a single document.
ListDocuments	List documents
UpdateDocument	Updates or inserts a document.
DeleteDocument	Delete a document

P1: Document API

API	Description
BatchGetDocuments	Gets multiple documents. Documents returned by this method are not guaranteed to be returned in the same order that they were requested.
ListCollectionIds	List documents
RunQuery	Runs a query.
RunAggregationQuery	Return the number of documents in table given a filter. SELECT COUNT(*) FROM ( SELECT * FROM k where a = true );
PartitionQuery	Partitions a query by returning partition cursors that can be used to run the query in parallel.

DB3 Entity Proto

Entity	Description
Database	Gets multiple documents. Documents returned by this method are not guaranteed to be returned in the same order that they were requested.
Field	Represents a single field in the database. Fields are grouped by their "Collection Group", which represent all collections in the database with the same id.
Index	indexes enable simple and complex queries against documents in a database.
Document	A db3 document. Must not exceed 1 MiB - 4 bytes.

[jingchen2222]

Milestone and resource plan

Milestore	Task	Owner	Status
Design POC	Document Operation workflow design	@cj	Completed
Design POC	DB3 admin opertation workflow design	@wtz	Completed
Feature Implementation	Entity Proto implement P0	@wtz	In Progress
Feature Implementation	DB3 Admin P0 Op Implementation	@wtz	In Progress
Feature Implementation	DB document P0 Op implementation	@cj	Not started
QA	Unit test & integartion test	@cj	Not started
QA	Benchmark	@cj	Not started
JS SDK API	Implement JS SDK for db3 admin and document operation	@zhaojun	Not started

[jingchen2222]

Implementation

Basic operation

• cmd to create database and create/list collection, etc. #297
• cmd to create document with out index #302
• cmd to create document with index #302
• cmd to get a document #311
• cmd to simple equal query document with index #320
• support comparison bytes for index #316
• support >,<,>=,<= filter condition fro run query document with index #404
• cmd to list documents under a collection. #306
• delete a document #314
• update a document #322
• [ ]

Advance feature

• list document with page loading (page_token, page_size). Example
• list document with setting document mask
• using events to notify clients of mutation result

[jingchen2222]

Discussion

@imotai 🔔 👀 ♨️
I wanna introduce CollectionID (16 bytes) as a part of IndexID instead of using colecction_name varchar().

IndexId = |DbID(20)|CollectionId(16)|BlockID(16)|

[imotai]

Discussion

@imotai 🔔 👀 ♨️ I wanna introduce CollectionID (16 bytes) as a part of IndexID instead of using colecction_name varchar().

IndexId = |DbID(20)|CollectionId(16)|BlockID(16)|

agree

[jingchen2222]

Propose index encode solution

Goal

• index key encode into/decode from bytes
• index bytes can be sorted
• index bytes order should be consistent with the original value order
• support Null encode
• support Emptry String encode
• define the order between types. Reference: https://www.mongodb.com/docs/manual/reference/bson-type-comparison-order/

Reference

• https://www.mongodb.com/docs/manual/reference/bson-type-comparison-order/

Further work will be taken care by #321