Add PostgreSQL as a backend
Closed this issue · 1 comments
pudo commented
Let's have a PostgreSQL backend implemented via SQLAlchemy or raw psycopg2, ideally with the following columns:
id
,fragment
(unique together)dataset
properties
schema
timestamp
We should make sure to use bulk loading when possible. It might be worth to look into using ON CONFLICT ... UPDATE with postgres to do database-side upserts:
https://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#insert-on-conflict-upsert
The table should be generated automatically when a user attempts to insert records for the first time.
Both the table name and the database connection string should be read from the envrionment. Or should we bake the dataset ID into the table name somehow?