innoq/spacy

Have a real database in the background

Closed this issue · 2 comments

joyheron commented

It would be nice if we could run spacy using a real database in the background instead of the in memory database that we are currently using.

jstepien commented

When starting the Crux component:

(let [opts {:crux.jdbc/connection-pool {:dialect {:crux/module 'crux.jdbc.sqlite/->dialect}
                                        :pool-opts {}
                                        :db-spec {:dbtype "sqlite"
                                                  :dbname "sample.db"}}
            :crux/tx-log {:crux/module 'crux.jdbc/->tx-log
                          :connection-pool :crux.jdbc/connection-pool}
            :crux/document-store {:crux/module 'crux.jdbc/->document-store
                                  :connection-pool :crux.jdbc/connection-pool}}
      node (crux/start-node opts)]
  ,,,
  (assoc component :node node))

Seems to work locally. We still need to:

  • decide how we handle configuration, such as :db-spec values,
  • pass that :db-spec into the Crux component, and
  • add a corresponding JDBC driver in project.clj.
joyheron commented

I'm open to ideas about how to best handle configuration. I think we should retrieve the things which are really secret from environment variables (like the DB password), but I'm not sure how to handle the rest of the config.

I see here that Crux seems to support mysql out-of-the-box: https://github.com/juxt/crux/blob/master/crux-jdbc/src/crux/jdbc.clj#L56

One idea I have is that we could modify our Crux component to always use crux.jdbc, but pass in the db-spec to the componentn on creation. Then we could create two system maps in spacy/system.clj, a dev-system and a prod-system. In the dev-system we could pass in a sqllite config, and for prod-system we could retrieve the db-spec config from an environment variable.

  • 👍1