/phoenix

A plugin for configuring, co-ordinating and reloading Components

Primary LanguageClojure

Phoenix

https://badges.gitter.im/Join%20Chat.svg

Phoenix has not been maintained for years, and no longer represents my opinion on how best to wire up Clojure applications. We live and learn!

I would recommend investigating Nomad and Bounce

Phoenix is a library that helps with simple configuration and credential management for Component-based systems, based on Nomad and Frodo.

If you’ve written a traditional Component-based system, you’ve probably experienced having to create a plethora of ‘system.clj’, ‘dev.clj’, ‘user.clj’ etc type namespaces in order to wire-up the system, set up configuration-based switches, and duplicate the code to start/stop/reload the system.

Phoenix removes the need for all of this, replacing it with one (or more, if you choose) EDN declaration of how your system should be wired up.

It also (optionally) includes a number of common Components to get you started - a ClojureScript compiler component, as well as components for http-kit and Aleph, amongst others. After that, it composes well with any ‘Lifecycle’ components that you (or anyone else) may have already written.

Contents

Getting started:

Phoenix is architected in a ’batteries included, but removable’ style (credit to Docker, I think, for originally coining that phrase).

As a result, it has two APIs:

  • a ’batteries included’ API, in the phoenix namespace, suitable for most use-cases - it’s what I use the vast majority of the time. It does rely on a little bit of global state but, for that cost, you get much simpler/quicker development setup and turn-around times.
  • a pure, ’batteries removable’ API, in the phoenix.core namespace, for when you need more composability/flexibility (e.g. test harnesses). Indeed, the ‘batteries included’ API is built on top of this core.

The rest of this section will get you started with the ‘batteries included’ API; further details on the ‘batteries removable’ API are below.

‘Just get me up and running already!’

For an ‘all-batteries-included’ project, try either of the ‘phoenix-api’ or the ‘phoenix-webapp’ Leiningen templates:

lein new phoenix-webapp <your-project>
cd <your-project>
lein dev

This will start an nREPL server on port 7888, start a web server on port 3000, and automatically re-compile your ClojureScript files if they change.

When it starts, head to http://localhost:3000 for further instructions.

‘But I already have a project…’

No worries, I understand :)

First, include Phoenix as a plugin in your ‘project.clj’:

:plugins [[jarohen/phoenix "0.1.3"]
           ...]

Next, make a small config file on your classpath - let’s say, in the ‘resources’ directory, at ‘config/myapp-config.edn’:

{:phoenix/nrepl-port 7888}

Then, tell Phoenix where to find your main config file by putting a :phoenix/config key in your ‘project.clj’:

{:phoenix/config "config/myapp-config.edn"}

Now, if you run lein phoenix, you should see the nREPL server open. Normally, at this point, Phoenix would also start your system, as and when you declare one in your config file.

Using Phoenix without Leiningen

You’ll need to add the Phoenix runtime to your dependencies:

[jarohen/phoenix.runtime "0.1.3"]

Then, from a REPL, run

(require 'phoenix
         '[clojure.java.io :as io])

(phoenix/init-phoenix! (io/resource "config-file.edn"))

After that, Phoenix should behave as documented below.

Reloading the system

Just as real-life Phoenixes die and are re-born from the ashes, so too does this Phoenix. At your REPL, you have a choice of three Phoenix commands, with fairly self-explanatory behaviour:

(phoenix/reload!)

;; or, explicitly:

(phoenix/stop!)
(phoenix/start!)

When you reload a Phoenix system (or explicitly stop and start it, in fact), the following steps happen:

  • The currently running system is gracefully stopped (if it’s running)
  • Any namespaces that have changed on disk are reloaded
  • The configuration file is re-read
  • The system is re-started

For more details about how the system is reloaded, and how to architect your application accordingly, I’d recommend reading Stuart Sierra’s excellent ‘My Clojure Workflow, Reloaded’ blog.

For Emacs users:

I use (phoenix/reload!) so much that I bind it to a single keypress, as follows:

(defun phoenix-reload ()
  (interactive)
  (save-some-buffers)
  (with-current-buffer (cider-current-repl-buffer)
    (cider-interactive-eval
     "(phoenix/reload!)")))

(define-key cider-mode-map (kbd "C-`") 'phoenix-reload)
(define-key clojure-mode-map (kbd "C-`") 'phoenix-reload)

Configuring components:

The system we’ve just created doesn’t have any components yet (unless you used the template, of course!) - let’s add some:

Adding components:

Let’s say we’ve written a component that makes a database connection pool:

(There is, in fact, a JDBC Pool Component already written for this!)

(ns myapp.database
  (:require [com.stuartsierra.component :as c]))

(defprotocol DatabasePool
  (db-conn [_]
    "Returns a JDBC connection, suitable for passing to
    clojure.java.jdbc/query et al"))

;; make-pool! and stop-pool! left as exercises to the reader

(defn make-pool! [opts]
  {:a-dummy :pool})

(defn stop-pool! [pool]
  (println "Stopping pool!"))

(defrecord PoolComponent []
  c/Lifecycle
  (start [{:keys [host user pass port database]}]
    (println "Starting DB pool...")
    (assoc this
      ::pool (make-pool! {...})))

  (stop [{:keys [::pool] :as this}]
    (println "Stopping DB pool...")
    (dissoc this ::pool))

  DatabasePool
  (db-conn [{:keys [::pool] :as this}]
    pool))

(defn make-database-pool [{:keys [host user pass port database] :as opts}]
  (map->PoolComponent opts))

(I’ll come back to why we’ve created a DatabasePool protocol later, when we come to use it)

We can add this as a component of our Phoenix system by creating an entry in the config map:

{:phoenix/nrepl-port 7888

 :database {:phoenix/component myapp.database/make-database-pool
            :host "db-host"
            :port 5432
            ...}}

The :phoenix/component entry in the :database map lets Phoenix know that this is a Component that needs to be started, by calling the provided function. Phoenix passes the remainder of the :database map to that function, so any configuration that the component needs can be stored here.

Let’s reload the system, and see the component started!

(phoenix/reload!)

The currently running Phoenix system is always available at phoenix/system, so you can use this to see what’s been created:

(:database @phoenix/system)

N.B phoenix/system is intended for debugging/REPL use only - fundamentally, it’s a global variable, so it’s best not to rely on it in live code! Phoenix has other, more composable ways of linking Components.

Adding dependencies between Components:

Having created our database pool, we’d now like to use it in the rest of our application.

We do this by registering a :phoenix/dep in the configuration map:

{:phoenix/nrepl-port 7888

 :database {:phoenix/component myapp.database/make-database-pool
            :host "db-host"
            :port 5432
            ...}

 :my-foo {:phoenix/component myapp.foo/map->FooComponent
          :database :phoenix/dep
          ...}}

The database will then be provided to the Foo component in the Component’s start function:

(ns myapp.foo
  (:require [myapp.database :as db]
            [clojure.java.jdbc :as jdbc]
            [com.stuartsierra.component :as c]))

(defrecord FooComponent []
  c/Lifecycle
  (start [{:keys [database]}]
    (prn "Here's all our users:"
         (jdbc/query (db/db-conn database)
                     ["SELECT * FROM users"])))

  (stop [this]
    ;; ...
    ))

Here, we’re using the db-conn protocol function to get access to the database connection - while we could access it directly within the record, it’s probably better to have a layer of indirection between them. This way, you can test the FooComponent in isolation by passing it a mocked out instance of DatabasePool.

I haven’t bothered creating a make-foo-component in this case - Clojure automatically creates a map->RecordName function for all records, which happens to have the same signature. In fact, if you don’t have to process the config map before passing it to the Component, I’d recommend you do the same!

Dependency aliases:

We don’t necessarily need to have the same name for the dependent key and the dependency - if we chose instead to call the database component ‘:postgres’, for example, we could alias it in ‘:my-foo’ as follows:

{:phoenix/nrepl-port 7888

 :postgres {:phoenix/component myapp.database/make-database-pool
            :host "db-host"
            :port 5432
            ...}

 :my-foo {:phoenix/component myapp.foo/map->FooComponent
          :database [:phoenix/dep :postgres]
          ...}}

As far as the Foo component is concerned, it can still refer to it’s database dependency under the ‘:database’ key.

Location-aware configuration:

Phoenix (like it’s predecessor, Nomad) allows you to specify different configuration, depending on where the system is running. You can switch on:

  • Hostname
  • Hostname/User
  • ‘Environment’ - start Phoenix with either: a ‘PHOENIX_ENV=…’ environment variable, or a ‘-Dphoenix.env=…’ Java system property

Location-specific should be included in the config under various ‘:phoenix/…’ keys, as follows:

{:database {:host "dev-db.mycompany.com"
            :port 5432
            :user "devapp"
            :pass "..."}

 :phoenix/hosts {"daves-laptop"
                 {:database {:host "localhost"
                             :port 13152
                             :user "dave"
                             :pass "..."}}

                 "test-box.mycompany.com"
                 {:database {...}

                  :phoenix/users {"user-a" {:database {...}}
                                  "user-b" {...}}}}

 :phoenix/environments {"stg"
                        {:database {:host "stg-db.mycompany.com"
                                    ...}}

                        "prod"
                        {:database {:host "prod-db.mycompany.com"
                                    ...}}}}

Configuration from the various locations is deep-merged - i.e. if you only specify the database username/password in a particular environment, then the username/password will be overridden in that environment, but the host will fall back to the main declaration.

The order of preference (in decreasing order) is: environment, host+user, host, general.

You can also override the ‘current location’ - e.g. to test the configuration values of other environments. When the system’s stopped:

(phoenix/stop!)

(phoenix/set-location! {:environment "stg"
                        :hostname "dev-machine"
                        :user "james"})

(phoenix/start!)

You can include/exclude entries from that location map, as required.

You can also pass the location map as an argument to ‘reload’:

(phoenix/reload! {:environment "stg"})

Referencing other config files:

You might have some configuration values that you don’t want to check into version control - passwords, or API keys, for example.

You can add a :phoenix/includes key into your configuration, which is expected to be a vector of external files. Phoenix provides two reader macros for this: #phoenix/file and #phoenix/resource, which can be used as follows:

;; myapp-config.edn

{:phoenix/includes [#phoenix/file "~/.myapp/passwords.edn"]

 :database {:host "..."
            :user "..."
            ...}}

;; ~/.myapp/passwords.edn

{:database {:pass "..."}
 ...}

The configuration in included files is deep-merged into the main map, with the included value taking preference if both specify the same key.

Includes can also be specified in the environment, host or user maps - for files that should only be included in a given location.

(You can use these reader macros throughout the rest of your config as well!)

Config in environment variables:

Configuration keys can also reference environment variables, using either [:phoenix/env-var :env-var-name] or [:phoenix/edn-env-var :env-var-name]. Environment variable names are automatically converted to ‘UPPER_SNAKE_CASE’. The difference between :phoenix/env-var and :phoenix/env-edn is that environment variables referenced with :phoenix/edn-env-var are parsed as EDN before being passed to your application.

To provide a default, in case the environment variable isn’t specified, include it with the vector: [:phoenix/env-var :my-env-var "default"]

{:my-component {:port 3000
                :username [:phoenix/env-var :myapp-user "admin"]
                :password [:phoenix/env-var :myapp-password "password-123"]}}
MYAPP_USER=another-user MYAPP_PASSWORD=pr0dp455w0rd lein phoenix

Config in JVM properties

Configuration can refer to JVM properties in the same way as environment variables, using either [:phoenix/jvm-prop :property.name] or [:phoenix/edn-jvm-prop :property.name], both of which take defaults as an optional third element in the vector.

You can then either supply the JVM properties in your Lein configuration, under the :jvm-opts key (which can itself be within a Lein profile), or by supplying it as an option to java, e.g.:

java -Dproperty.name=my-value -jar thingy.jar command-line-args...

Building Phoenix-based projects

You can build Phoenix-based projects by running:

lein phoenix uberjar

This creates an executable JAR file, which can then be run with:

# Replace this with the actual path to the uberjar
java -jar target/myapp-standalone.jar

Managing your passwords/credentials

Phoenix can manage your passwords/credentials in the same source repository as the rest of your configuration, but without checking plain-text credentials into version control.

It does this through encrypting the credentials using 256-bit AES, with the keys stored in a separate configuration file.

Setting up:

  1. Generate your first key: 
    (phoenix.secret/generate-key)

;; for example:
;; => "b14127be18a2408ed7037c98e7a3a6783651881539d1b8df4ebbc27ab335caf2"
  2. Create a keys file outside of version control (either outside the VCS root, or ‘ignored’ by your VCS), under the :phoenix/secret-keys key, as follows: 
    ;; ~/.my-phoenix-keys.edn
{:phoenix/secret-keys {:my-first-key "b14127be18a2408ed7037c98e7a3a6783651881539d1b8df4ebbc27ab335caf2"}}

    Here, :my-first-key is our Key ID. Share this with other developers, and place it on production machines, as necessary. You can also encrypt production credentials with a different key, if need be.

  3. Include that file in our checked-in configuration: 
    {:phoenix/includes [#phoenix/file "~/.my-phoenix-keys.edn", ...]

 ...}
  4. Encrypt your first password: 
    (phoenix.secret/encrypt "password-123" ; plain-text
                        "b14127be18a2408ed7037c98e7a3a6783651881539d1b8df4ebbc27ab335caf2") ; key

;; => "6a1623eeda59772a6e948b2b7e17fdcf28cec8398243a2307b781819fb360bd1"
;; although will be different when you run it, even if you run this example

    Optionally, you can decrypt it again with:

    (phoenix.secret/decrypt "6a1623eeda59772a6e948b2b7e17fdcf28cec8398243a2307b781819fb360bd1" ; cypher-text
                        "b14127be18a2408ed7037c98e7a3a6783651881539d1b8df4ebbc27ab335caf2") ; key

;; => "password-123"

    You can encrypt any EDN data structure using (phoenix.secret/encrypt ...), not just strings:

    (let [sample-key (phoenix.secret/generate-key)]
  (-> {:a 1, :b 2}
      (phoenix.secret/encrypt sample-key)
      (phoenix.secret/decrypt sample-key)))

;; => {:a 1, :b 2}
  5. Include that in your main configuration file

    You’ll need to let Phoenix know: a) that it’s encrypted; and b) what key it was encrypted with, which you can do as follows:

    {:db {:phoenix/component ...
      :user "my-user"
      :password [:phoenix/secret :my-first-key "6a1623eeda59772a6e948b2b7e17fdcf28cec8398243a2307b781819fb360bd1"]}}
  6. Retrieve the credential as you would any other Phoenix configuration value - it’s decrypted automatically: 
    (defrecord DBComponent []
  c/Lifecycle
  (start [{:keys [user password]}]
    ;; Would advise against _actually_ doing this, of course...
    ;; => "My database password is: password-123"

    (println "My database password is:" password))


  (stop [_]
    ...))


(get-in @phoenix/system [:db :password])
;; => "password-123"

Security Auditing

This part of the codebase has not been security audited as yet (as far as I know!), and so, as such, I’d advise against its use in critical systems. If you can help by casting more pairs of eyes over this (it’s only about 70LoC, based atop Buddy), I’d be very grateful!

Removing the batteries

The ‘batteries included’ Phoenix API simply calls through to the ‘batteries removed’ API in order to start a system. It does this in 5 stages:

  1. Load config files + handle :phoenix/includes: 
    (phoenix.core/load-config {:config-source (io/resource "...") ; or (io/file "...")
                           :location {:environment "live"}})

    Using :location (optional key), you can load the configuration for a different location (i.e. changing the :environment, :host or :user).

  2. The result is analyzed to determine the component dependency order, and aliases: (phoenix.core/analyze-config loaded-config)
  3. The analyzed config is turned into a com.stuartsierra.component/SystemMap: (phoenix.core/make-system analyzed-config {:targets targets})

    If you do not want the whole system started (e.g. for testing a sub-system), specify the component keys that you do want started as targets, otherwise, feel free to omit the second parameter entirely.

  4. The system is started: (com.stuartsierra.component/start-system system)
  5. Later, the system is stopped with (com.stuartsierra.component/stop-system started-system)

There’s nothing to stop you doing this, as well!

If you need the flexibility/composability, you can adapt any one of these steps to suit your needs. e.g.:

  • Replace the load-config step to pass a config map directly (without reading it from a file)
  • Pass a different location to step 1, to see what configuration would be present under a different environment
  • Just run step 1 to see what configuration values Phoenix is using (e.g. to test out the location switching)
  • Update the configuration map between steps 1 & 2, or 2 & 3, in order to temporarily override a configuration value
  • And many more…

You can also use the phoenix.core/with-running-system macro to set up and tear down a system, for testing purposes:

(require '[phoenix.core :as pc])

(pc/with-running-system [{:keys [component-under-test]} (-> (pc/load-config {:config-source (io/resource "app-config.edn")})
                                                            pc/analyze-config
                                                            (pc/make-system {:targets [:component-under-test]}))]

  ;; test 'component-under-test' - it (and all of its dependencies) will be started before, and stopped after, this block

  (is (= ...)))

‘Built-in’ components

Phoenix has a number of optional ‘built-in’ components, each with their own documentation:

Questions/Suggestions/Bugs/Features/PRs?

Yes please! Feel free to get in touch, either through GitHub, Twitter (@jarohen) or e-mail (on my profile).

Cheers!

Licence

Copyright © 2015 James Henderson

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.