Object Store

Status

A object storage system with a gRPC interface and strong encryption properties. Currently, only the Provenance DIME format is accepted. Supporting material can be found here.

This service can be used directly, but the most common case is to use this alongside the P8e Execution Environment in order to process Provenance scopes and memorialize them on chain.

Features

Strong end-to-end encryption.
Peer-to-peer replication to parties (third party object-stores) you want to share data with.
Capable of publishing traces to Datadog.

Authentication

gRPC metadata based authentication is provided on a per key basis. When adding a public key to the database, an auth_type and auth_data can be provided. These can either be leveraged directly as an api key or indirectly be combining it with a proxy capable of authentication and header forwarding. Setting both of these fields to null and a service level config property of USER_AUTH_ENABLED=false disables all authentication - this can be used if the object store is meant for internal use and not exposed publicly.

Example authentication configuration

NOTE: Requires settings the service level configuration to USER_AUTH_ENABLED=true.

public_key=BH6YrLjN+I7JzjGCgrIWbfXicg4C4nZaMPwzmTB2Yef/aqxiJmPmpBi1JAonlTzA6c1zU/WX4RKWzAkQBd7lWbU=
public_key_type=secp256k1
auth_type=header
auth_data=x-custom-header:6eace982-f682-4b1d-9f8e-82ed9ab15813

With such a configuration all requests for this public key will have to contain this metadata.

Backends

This service was designed to support many underlying storage backends. The currently supported backends are postgres, google cloud storage, and the local file system. In practice, a sizeable number of objects this system stores are very small. For this reason the postgres backend, along with a byte threshold, is provided so that items smaller than the threshold specified can have thier bytes stored directly in the database.

Local Development

In order to run locally a postgres connection and data directory is required. A base set of environment variables can be edited and sourced from ./bin/env.

Running Locally

The most common use case for object-store is to run it alongside p8e in order to write to the Provenance Blockchain. The simplest way to get this up and running is here.