🐉 Here be Dragons: this document is a work in progress. If you have any feedback feel free to open an issue or a pull request. 🐉

Sharing private data in a public blockchain

This paper specifies a standard to enable sharing of private information on a public blockchain.

Rationale

Blockchains are evolving from managing tokens of value to handling digital assets and securely executing business logic. BigchainDB is a prime example of this evolution in the area of digital assets.

Storing and sharing digital assets raises tough new privacy challenges, e.g. most of the people do not want to share their health records or their land property documents with the world, even if the law allowed this, and interesting regulatory challenges, e.g. how can regulators balance the necessary transparency to improve trust in the financial system while ensuring data privacy where it is warranted.

Privacy is a vast subject, and its interpretation depends heavily on the context. In some cases privacy means keeping an asset encrypted and readable only for designated users. Other times privacy means anonymity, for example when WikiLeaks publishes classified information keeping sources anonymous.

This paper focuses on exploring the questions of secure and private data sharing between one or more counterparties.

Disclaimer

This paper compares how different cryptocurrencies handle transactions confidentiality. The focus is on how information is stored in the transaction itself, not on mining, block ordering, block chaining, nor how a decentralized network is Byzantine fault tolerant.

This paper should not be relied on for legal advice on what data can be shared or how it can be shared. Different jurisdictions have different requirements for the collection, storage, processing, and sharing of personal data.

The cryptocurrencies featured in this paper are:

Bitcoin, for its wide adoption and historical importance.
Ethereum, for its innovative approach as a distributed computing platform.
Monero, for the focus on privacy and untraceability.
Zcash, for using a zero-knowledge proof crypto protocol.

Introduction

Confidentiality and privacy for blockchains is not a new topic. The extensive R3 report gives a great overview of the current status for privacy and confidentiality in the blockchain space. The following paragraph, taken from the introduction of the report, frames the intent of this document quite well:

In the early days of the Web, the Secure HyperText Transfer (HTTPS) and Secure Sockets Layer (SSL) protocols emerged as competing standards for adding privacy to the Web, with SSL eventually triumphing. We expect blockchain technology to follow a similar path, with multiple competing privacy standards emerging over the next few years.

Transparency and verifiability are key features of blockchains. These features are used by the blockchain consensus itself, to balance off accounts. To avoid double spends, a blockchain should be able to verify that an account has enough balance to satisfy the transaction to another account.

The logic behind a specific cryptocurrency must be able to extract some information from the transaction to decide if the transaction is valid or not. This brings us to an important conclusion:

An asset, in order to be a first-class citizen in a blockchain system, must allow computability.

If we want to automate and delegate the management of assets to a decentralized system, the system itself should be programmed in a way to understand those assets.

On asset types, verifiability, and confidentiality

This report uses the same definition of confidentiality used by R3:

[...] we use the term confidentiality in the context of protecting data (e.g. transaction details, price, asset types, account and wallet balances, the business logic of smart contracts) from unauthorised third parties.

We can identify three distinct properties associated with an asset:

Type: the nature of the information stored. It can be
- Scalar, a number or a string.
- Text, a paragraph of free text from a novel, or some source code. (Someone might argue this is a subset of scalar, but we mean something more like a text file). A human should usually be able to parse that.
- Structured, information organized in a specific manner, e.g. a JSON document.
- Binary, a blob of binary data, e.g. audio encoded in mp3 format, a compressed archive, or an image. A human likely will not be able to parse it.
Verifiability: can the network interpret the asset and give a response on asset validity?
Confidentiality and Verifiability: can be framed as the ability of an asset to still be verifiable while keeping confidentiality. In other words: if the asset is encrypted, can a program still verify it?

Starting from the asset type, let's see how different ledgers handle verifiability and confidentiality.

Scalar assets

Bitcoin, Ethereum, Monero, and Zcash natively support scalar quantities. This is a requirement for every cryptocurrency: without it, scarcity cannot be achieved, and scarcity is a fundamental property of everything having monetary value. BigchainDB, even if it's not a cryptocurrency, handles scalar quantities as well.

For the Bitcoin and Ethereum network, all the exchanges of value are publicly readable. As an example, anyone can easily check how many bitcoins were donated to XKCD. BigchainDB allows you to specify an amount on asset creation.

On the other side, cryptocurrencies like Zcash and Monero preserve transaction confidentiality, adding a new layer of privacy for the users. Zcash achieves this by using a zero-knowledge cryptography protocol called zkSNARK. Monero uses a combination of stealth addresses and ring signatures.

For every technology listed, scalar quantities are verifiable, but only Zcash and Monero maintain the confidentiality and verifiability of the transaction.

Text assets

Bitcoin does not natively support extra text data, but hacks are possible. Protocols have been built on top of the Bitcoin OP_RETURN. The SPOOL Protocol is a prime example of that.

Monero payment ID is an optional 32 bytes field that can be used to store additional information on a payment.

Zcash memo field allows you to attach an optional 512 bytes message to a transaction.

BigchainDB allows arbitrary JSON payloads, so strictly speaking text only assets are not allowed.

Those four technologies don't allow for verifiability for text assets. The network cannot approve or reject transactions based on their text content. Protocols built using text assets must rely on an overlay network to add a new layer of validation.

Ethereum allows users to attach a payload of arbitrary size to the transaction. The payload can be interpreted as a byte string. The bigger the transaction is, the more gas is needed for the network to process it. (The payload is also used to initialize smart contracts.)

Related to verifiability, a modified Ethereum ERC20 token may add a text field to the token itself. The field would then be processed by the smart contract and used to accept or reject transactions. This approach is similar to the overlay protocol explained before, with the benefit that the execution of the protocol would live in the network itself, hence being be decentralized.

Let's spend some more words on the concept of verifiability of a text asset. To discern valid transactions from invalid transactions, a program must be able to parse the text input. Even if an Ethereum smart contract can run and process arbitrary text, the text that has been tested for validity must be interpretable by the program itself.

In an Ethereum smart contract, confidentiality and verifiability might be allowed as well, depending on the nature of the text asset (can we run zero knowledge proof protocols on it?). Since smart contracts are Turing complete, the assumption is that a smart contract with enough gas can execute zero knowledge proof algorithms.

JSON (or structured) assets

Bitcoin has shown that arbitrary text can be stored. Since JSON is a subset of a generic text, everything we said in the previous section is still valid.

Considering the limited space in Monero payment ID, a JSON object is too large to be stored.

BigchainDB is able to store native JSON structures, and everything that has been said for Bitcoin in the previous section applies.

Ethereum can store arbitrary text, and everything that has been said in the previous section applies.

Binary assets

A binary blob can be encoded as text using different encodings. Everything said in the previous section applies.

Please keep in mind that storing binary data in a blockchain is inefficient and expensive.

(Fun fact: someone stored the iconic sentence I'm sorry Dave, I'm afraid I can't do that as an audio file in Ethereum.)

On verifiability and confidentiality

The previous section explained how verifiability and confidentiality can be achieved using scalar quantities. In case of different asset types, custom algorithms must be developed, but with a warning: adding verifiability to a confidential asset can be a hard problem to solve. Let's see some examples.

A blockchain for linked data

Let's say we want to store linked data in a blockchain, using the JSON-LD format. JSON-LD @context and @type are used to enforce the schema of the data. Given a property, its domain and range can be used to validate the semantics of the data itself.

Schema validation is a first step, we can climb up the semantic web ladder and add more and more validation rules.

The Semantic Web Stack

Let's stick with the simple case: schema validation. A transaction is considered valid only if it passes validation. In this case, verifiability is easy to implement, and the validation is generic enough to be used for any kind of JSON-LD document.

Combining confidentiality and verifiability would mean letting users exchange valid and encrypted transactions. This would require the creation of a new (eventually) non-interactive, zero-knowledge proof protocol that would allow validation nodes in the network to decide on the validity of the transaction.

A blockchain for cat images

Let's spice up our blockchain with some AI. The validation rule for this blockchain is quite simple: a transaction, to be considered valid, must contain a picture of one or more cats. As we already said, storing binary data in a transaction is not efficient, so for this use case a transaction will contain a pointer to a content-addressable picture (stored in IPFS, for example).

Given an image, a cat classifier returns the probability that the image contains a cat. Absolute certainty on the presence of a cat is not achievable, so let's assume that if the confidence is greater than 0.99 then the transaction is valid. This makes the transaction verifiable.

Combining confidentiality and verifiability would require running a classifier on encrypted data. Breakthrough encryption techniques might enable this, but it's still a field under active research and development.

Sharing encrypted assets

In this section we analyze how an exchange of encrypted asset is initiated and managed, and what its properties are.

Properties of the system

The system described in this paper wants to provide several guarantees for end users, specifically:

Proof of Sharing
Authentication
Data integrity
End to end encryption
Perfect forward secrecy

Proof of Sharing

BigchainDB is a general purpose blockchain database, and does not constrain users to any specific kind of (JSON) assets. Therefore, a generic approach is preferred. From our knowledge, there is no crypto system that can guarantee both verifiability and confidentiality of generic asset. Confidentiality can still be achieved with classic encryption protocols, but if we drop verifiability how can we make sure players in the system will cooperate?

Introducing Proof of Sharing. Proof of Sharing is an optimistic approach that allows cooperating actors to have frictionless and secure exchange of data. Let's say Alice wants to access some private data owned by Bob. The interaction between them is the following (technical details will be discussed throughout this paper):

Bob owns the asset X. He publishes some metadata and conditions on how to get X. A simple case would to set the price of the asset.
Alice wants to access and read X. She reads the metadata and conditions about X, and she makes a payment (or whatever is needed) using the required fiat or crypto currency.
Alice transfers her access token (in this case, a payment receipt) for X to Bob.
Bob verifies that the access token is valid, and starts a handshake to create a shared session key with Alice.
Alice participates in the handshake.
Alice and Bob now have the same session key.
Bob encrypts X using the previously calculated session key and shares it with Alice, either on- or off-chain.

If both Alice and Bob played by the rules, everyone is happy and no conflict resolution is needed. But what happens if one of them acts maliciously? Alice, once she obtained the private asset X, could say it was not the asset she paid for, and ask for a refund. On the other side, Bob could send Alice total garbage instead of X.

The important part is the handshake between Alice and Bob. Here the blockchain plays a crucial role. By storing all those steps as transactions in the system, Alice or Bob could expose malicious behaviour to a third party.

Alas, finding the culprit is not enough, especially when they are anonymous. To properly deter malicious actors, a stake is added to the handshake. The stake can be kept secret between parties, and revealed when a conflict occurs. By using crypto conditions combinatory logic, Alice and Bob can create a threshold condition n-of-m, where m defines the third parties involved that can eventually release the funds to the injured party.

Note: a stake is not a requirement, but it adds another level of protection to the transaction. If the identity of the parties is known, then the dispute can be resolved in other ways, such as the legal system. However, this may be less efficient than stake-based resolution.

Authentication

Authentication proves that a message came from a particular sender.

In case of symmetric encryption, this is usually achieved using a (Hashed) Message Authentication Code. For asymmetric encryption, a digital signature of the message is used.

Data Integrity

Data Integrity makes sure that tampered messages are detectable. TK: elaborate more

End-to-End encryption

TK: finish this section

Perfect Forward Secrecy

Perfect forward secrecy is a property of secure communication protocols in which compromise of long-term keys does not compromise past session keys. Forward secrecy protects past sessions against future compromises of secret keys or passwords. If forward secrecy is used, encrypted communications and sessions recorded in the past cannot be retrieved and decrypted should long-term secret keys or passwords be compromised in the future, even if the adversary actively interfered.

This property is crucial in a blockchain use case. Since all data is stored forever, if a key is leaked it can compromise a significant amount of assets. Also, proof of sharing might require that users reveal keys when dealing with a malicious actor.

The Transport Layer Security Protocol

Instead of creating a new protocol from scratch, we decided to stand on the shoulders of a widely used, reviewed, industry-ready standard: the Transport Layer Security 1.2 (TLS) protocol. (Version 1.3 is currently in development, we might upgrade this paper once it's ready.)

TLS is used to provide communications security over a computer network. For example, when a browser connects to a website using HTTPS, the session key used for the symmetric encryption is established using a combination of protocols defined by TLS. The process of establishing a secure connection is called handshake, and it's done in real time between the client and the server. Even if TLS is perceived as a "low latency" protocol (the handshake takes few hundreds of milliseconds), we argue that it can be used in a higher latency scenario such as a blockchain use case. This section explains which parts of TLS are suitable to help us fulfil the requirements described above.

TLS Handshake

TK: is it ok if I copy paste https://en.wikipedia.org/wiki/Transport\_Layer\_Security#TLS\_handshake ?

TLS Cipher Suites

TLS defines a large number of Cipher Suites. A cipher suite is a named combination of authentication, encryption, message authentication code, and key exchange algorithms used to negotiate the security settings.

If you have openssl installed in your machine, it's fairly easy to have a list of the cipher suites available:

$ openssl ciphers -v -tls1
DH-RSA-AES256-SHA256      TLSv1.2 Kx=DH/RSA Au=DH    Enc=AES(256)  Mac=SHA256
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH   Au=ECDSA Enc=AES(256)  Mac=SHA384
DH-RSA-AES256-SHA         SSLv3   Kx=DH/RSA Au=DH    Enc=AES(256)  Mac=SHA1
... 94 lines omitted ...

Each item on the list defines a combination of key exchange (Kx), authentication (Au), encryption (Enc), and message authentication (Mac).

Let's analyze the first line: DH-RSA-AES256-SHA256.

DH: key exchange will use Diffie-Hellman
RSA: authentication will use RSA for authentication
AES256: data will encrypted using Advanced Encryption Standard, with a key length of 256 bits.
SHA256: data integrity will be ensured by Secure Hash Algorithm 2, with a digest size of 256 bits.

As you can see, a cipher suite cover most of the properties we described before, specifically:

Authentication
End-to-end encryption
Data integrity

What is still out of the equation is perfect forward secrecy, but no worries: TLS provides a collection of ephemeral key exchange algorithms that fulfill this requirement. We can easily query for them:

$ openssl ciphers -v -tls1 | grep DHE
ECDHE-RSA-AES256-GCM-SHA384   TLSv1.2 Kx=ECDH Au=RSA   Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-SHA384       TLSv1.2 Kx=ECDH Au=RSA   Enc=AES(256)    Mac=SHA384
ECDHE-ECDSA-AES256-SHA384     TLSv1.2 Kx=ECDH Au=ECDSA Enc=AES(256)    Mac=SHA384
ECDHE-RSA-AES256-SHA          SSLv3 Kx=ECDH   Au=RSA   Enc=AES(256)    Mac=SHA1
ECDHE-ECDSA-AES256-SHA        SSLv3 Kx=ECDH   Au=ECDSA Enc=AES(256)    Mac=SHA1
... 28 lines omitted ...

A key exchanged using an ephemeral protocol must not be stored for longer that the session itself.

Now that we have an overview on how TLS and cipher suite work, we can dig into details and describe how we can adapt TLS to suite our needs.

On authentication

All transactions on a blockchain network are digitally signed with the signing key(s) of the issuer(s), so each message is authenticated per se. TLS has a set of cipher suites that allow us to disable authentication. In this case, if a signed message satisfies the parties involved, they can ignore extra certificates and move on.

Extending support to X.509 certificates

It might be useful for some use cases to enable X.509 certificates. Most of the old web still relies on them, so why not use them?

On symmetric encryption

Since 2008, Intel and AMD CPUs provide an extension to the x86 instruction set architecture called Advanced Encryption Standard Instruction Set. For Intel, this feature has been added to commodity CPUs like i5 and i7. Even if AES is the standard symmetric encryption algorithm for many use cases, it's worth noting that it performs extremely well on those specific CPUs. The openssl command can run a speed test on specific encryption algorithms. To taste the speed-up on your machine, you can run openssl speed.

On a Intel Core i5-2520M CPU, 2.50GHz, this is the result of the benchmark without and with the instruction set enabled.

$ openssl speed -elapsed aes-128-cbc
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128 cbc      95411.96k   102425.79k   104104.19k   105198.25k   105502.04k


$ openssl speed -elapsed -evp aes-128-cbc
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc     559001.23k   603807.10k   615512.66k   619206.31k   617106.09k

A normal CPU can provide a 6x increment in performance just by setting a flag. Nice!

Protocol Definition

Time to talk about how this works with a blockchain. The scenario is the one described before: Alice just spent some bitcoin to buy a digital asset X from Bob.

Ideally, Alice and Bob will pass the same transaction around, creating an audit trail of their handshake.

Step 0: Bob publishes metadata about his payload

Bob has few private assets he is willing to share under certain conditions. For each asset, he creates a transaction and pushes it to the network. Those transactions contain metadata about the assets. We haven't defined a schema for metadata yet, but the schema will be expressive enough to cover these use cases and more:

sell an asset for a fixed price
give read permission to an asset for a time interval
etc.

If needed, each payload offer can have an extra field with the stake that Bob puts to vouch the offer. This stake must be read as "if I'm not delivering what I promise, the stake is yours".

A reputation system could be used, both as a complementary guarantee, or as an alternative to stake.

Step 1: Alice starts a negotiation with Bob.

Alice starts the handshake by transferring an asset to Bob. Bob has to reply to know what Alice wants and do actual work. This is of course susceptible to spam, so Alice puts stake to prove she is not spamming Bob. If there is a trust relationship between Alice and Bob, she might omit the stake.

The transaction contains a payload structured like:

{
  "stake_id": "<reference to a no-spam stake she puts on the table>",
  "nonce": "<a sequence of 32 random bytes>",
  "ciphersuites": [
    "ECDHE-NULL-AES256-GCM-SHA384",
    "DHE-NULL-AES256-GCM-SHA384",
    "<and more ciphersuites that Alice can manage"
  ]
}

Step 2: Bob starts the pre-master secret exchange

Bob, after verifying the validity of the transaction, and assuming that he finds a cipher suite that he can handle, proceeds with the key exchange re-transferring the asset to Alice. There is no need to specify any session id, or add extra information to keep track of this key exchange: the trail is kept in the transaction history.

{
  "nonce": "<a sequence of 32 random bytes>",
  "stake_id": "<reference to a stake he puts on the table>",
  "ciphersuite": "ECDHE-NULL-AES256-GCM-SHA384",
  "params": "<Elliptic-curve Diffie-Hellman parameters>"
}

Step 3: Alice sends back her parameters and puts her (optional) stake on the table

Alice chooses her Diffie-Hellman parameters, calculates the pre-master secret, derives the session key, and sends the transaction back to Bob. She can now exchange encrypted payloads with Bob!

Note: from now on, cipher contains the encrypted payload, encoded in Base64. For the sake of clarity, the examples contain the plaintext.

If the asset is "pay and forget", Alice puts her stake on the table, and she specifies which asset she needs.

Or she can start a secure, confidential negotiation between Bob and her, sharing his terms with him. If Bob find those terms acceptable, he will share the actual payload with Alice, otherwise he won't, and the negotiation will continue until both parties are satisfied.

{
  "params": "<Elliptic-curve Diffie-Hellman parameters>",
  "cipher": {
    "asset_id": "<the ID of the asset she wants to obtain>",
    "stake_id": "<the actual stake she will lose if she reveals the payload>"
  }
}

Step 4: Bob put his (optional) stake on the table

This step can be skipped if stake is not involved.

{
  "cipher": {
    "stake_id": "<the actual stake he will lose if he is sharing garbage>"
  }
}

Step 5: Alice shares her receipt id with Bob

Alice can now reveal her access token. If Bob doesn't fulfill her request, she can reveal the pre-master key and get Bob's stake.

{
  "cipher": {
    "access_token": "<reference to the access token (can be a bitcoin tx)>",
  }
}

Step 6: Bob shares his asset

Bob can finally verify the access token and share his secret with Alice. Alice has one week to verify the validity of the asset. In this week, if Bob uploaded the asset off-chain, the asset must be retrievable for one week. This will allow audits from third parties in case of disputes.

Channels

Alice and Bob have created an encrypted, secure channel on the blockchain, that they can reuse in the future. They might need to refresh keys once in a while, but this topic is out of the scope of the current document.

Conclusion

Ensuring privacy in a public blockchain is not a trivial task. Leveraging well tested technologies like TLS can help us reaching the level of confidentiality we need in order to unlock new use cases for blockchain technology.

Other resources:

Some other interesting links on the topic:

Contributors

Contributors to this document, in alphabetical order:

Alberto Granzotto alberto@bigchaindb.com
Greg McMullen greg@ipdb.foundation

RANDOM NOTES, IGNORE PLS

Solicited (pull) vs unsolicited (push) sharing.

Protocol

Alice --C{[proof], AlicePk, protocol=ECDHv1}--> Bob
Bob --T{BobPk}--> Alice
Alice --T enc{assetId, timeInterval}--> Bob
Bob --T enc{uri, hash}--> Alice

Bob can cheat and:

ignore Alice's request: Alice can show the proof details to a third party
not upload the file: Alice can ask a third party to check the URI
upload another file: the hash won't match the file
upload a file matching the hash but with other content: Alice will reveal the session key to a third party that will verify the content Alice cannot cheat: Bob can reveal the session key to a third party and prove he was not cheating.

Alice can ask Bob for other assets by doing:

Alice --T enc{assetId, timeInterval, proof}--> Bob where the new parameter proof is the "receipt" that Alice "paid" for assetId.

bigchaindb/privacy-protocols