alephdata/followthemoney-store

Add PostgreSQL as a backend

Closed this issue · 1 comments

pudo commented

Let's have a PostgreSQL backend implemented via SQLAlchemy or raw psycopg2, ideally with the following columns:

  • id, fragment (unique together)
  • dataset
  • properties
  • schema
  • timestamp

We should make sure to use bulk loading when possible. It might be worth to look into using ON CONFLICT ... UPDATE with postgres to do database-side upserts:

https://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#insert-on-conflict-upsert

The table should be generated automatically when a user attempts to insert records for the first time.

Both the table name and the database connection string should be read from the envrionment. Or should we bake the dataset ID into the table name somehow?

pudo commented

Thanks @sunu