data-api: A Python repository from unytics

Serve Data with Low-Latency

1. Features 🎍

data-api discovers your datastore data & exposes them as a REST read-only API.

💡 By datastore we mean firestore in datastore mode.

Features include:

generate open-api spec definition
generate Swagger-UI documentation page
expose datastore data via key-value or property-filtering.
cache responses with configurable duration
manage permissions with api-keys or openID tokens.

💡 data-api is great to create a low-latency API for your BigQuery data (that you can export to datastore with one sql query. More below).

2. Deploy 🚀

Run

gcloud run deploy data-api \
  --source . \
  --set-env-vars "PROJECT=$PROJECT,DATABASE=$DATABASE"

with:

$PROJECT the Google Cloud Project where you datastore database resides
$DATABASE the name of your datastore database (if not given, it will use the default database).

You can then get the url of your data-api Cloud Run Service by running:

gcloud run services describe --format "value(status.url)"

3. Exposed Routes 🚚

💡 Datastore Reminder:

Data in datastore is organized in a hierarchy: database/namespace/kind (like the database/schema/table hierarchy in relational databases).

Inside a kind (think table), the entities (think rows) have a key (a string or integer) and a value (a dict).

data-api Cloud Run service:

exposes the data of the database defined as environment variable at deploy time (see above).
considers each namespace as a different api (which has its own open-api spec definition and Swagger UI).
exposes the following routes (with GET method):

url	Description
`/`	Redirects to `/api/`
`/api/`	Returns the list of namespaces in `database`
`/api/<namespace>/`	Returns details on `namespace` api including its `kinds` and urls
`/api/<namespace>/openapi.json`	Returns the openapi spec definition of the `namespace` api
`/api/<namespace>/swagger-ui.html`	Returns the Swagger UI (documentation portal) of the `namespace` api
`/api/<namespace>/<kind>/`	Returns a list of entity values of `kind`
`/api/<namespace>/<kind>/<key>`	Returns the entity value of `key`
`/api/<namespace>/<kind>/?foo=bar`	Returns a list of entity values of `kind` for which `foo` property is equal to `bar`

4. [BONUS] Export BigQuery table to datastore ✍️

data-api is great to create a low-latency API for your BigQuery data.

You can export a BigQuery table into datastore with one sql command to make it available to data-api:

call bigfunctions.eu.export_table_to_datastore(
  'your-project.dataset.table',
  'user_id',
  'your-project/your-database/default/users'
);

💡 If you don't want to use the public bigfunction, you can deploy the function in your own BigQuery project. Check the function documentation.

5. Contribute 👋

Any contribution is more than welcome 🤗!

Add a ⭐ on the repo to show your support
Join our Slack and talk with us
Raise an issue to raise a bug or suggest improvements
Open a PR!

6. TODO 📝