“Programming should be interactive, as beautiful as possible, modular, and it should generate assets that are easy to use and learn.” – Michael O. Church
Jig is an application harness providing a beautifully interactive development experience for Clojure projects.
Feedback is at the heart of all agile software development processes. If you can reduce the time between making a change and seeing the result, you can proceed faster, with more confidence and accuracy.
For the majority of developers, the impact of changing code cannot be determined until the entire system has been re-built and re-tested. While there are many advantages to frequent and thorough testing, for developers the wait is long enough for us to lose our concentration and focus.
Unless you have experienced a development environment that offers instant feedback upon change it is difficult to describe the impact it can have on your ability to solve difficult problems, produce reliable code quickly and make programming more enjoyable.
Clojure, in the tradition of dynamic languages, comes very close to providing this kind of experience right out-of-the-box. In LISP, functions are bound to vars, which can be rebound, and when functions are applied, the latest binding of the function is used.
In practice, however, there are some minor quirks that impair the
dependability of the code reloading : adding a new library dependency,
redeclaring a defmulti or Clojure protocol, stale state referenced
in defs and defonces are some examples. One of Jig's aims is
to handle these incidentals for you, letting you concentrate more fully
on your programming.
Jig builds upon Stuart Sierra's excellent reloaded workflow pattern. Therefore it's important that you're familiar with the general idea of this pattern: the developer invokes a 'reset' function at various intervals which causes a minimalist reloading of just the code that has changed since the last reset, and anything else that needs reloading as a result (Due to the intricacies of Clojure and the JVM, this is a non-trivial problem that Stuart has solved). Typically, the reset function is bound to a hotkey, Emacs keybinding (I use "C-c r") or something more exotic. The price of entry is that a developer has to ensure all application state is held in a single map (or record) called the system. Otherwise the pattern doesn't work but in practice this is a good architectural policy to establish regardless.
Jig extends Stuart's work by providing some optional extra features that can help in the development of large Clojure systems:
- modularity through componentisation
- configuration
- support for multiple projects
- a growing set of common re-usable infrastructure components.
Stuart describes the System Constructor which creates the initial
state of a system. Jig provides an implementation of the System
Constructor that delegates the job of creating the system to components,
each component having the same lifecycle interface as Stuart describes:
init, start and stop. The System is created by
threading it through all the components.
A reset stops and restarts all components. Components are initialized and started (in dependency order) and stopped (in reverse dependency order) allowing for a clean shutdown of resources.
There are many good reasons for dividing your System into separate components. One is that it gives you looser coupling between your components, which can make it easier to re-use them (among other benefits). Another is that it allows you flexibility and architectural options at deploy time.
For example, you can deploy all your components in a single JVM for a test environment, while in production you could distribute components across multiple JVMs, for scaleability. If you are forced to deploy lots of small JVMs, in all environments, this can be an inefficient use of precious memory. I prefer to run a smaller number of JVMs, each with more memory. Developing Clojure applications as monolithic systems works well to begin with but can reduce flexibility down the road.
I want the option of deploying early versions of new projects quickly, without the hassle of setting up a new server or incurring the cost of a dedicated JVM (200Mb is not an insignificant overhead when you have dozens of Clojure-based web applications). Jig lets me quickly hook up new web applications and services onto an existing Clojure deployment.
Here's an example of one of my component graphs.
Rather than using a lein template to generate the project and the corresponding dev System, Jig separates these concerns. You can use Jig's workflow to develop on existing projects that don't have a built-in dev workflow.
You can also fork and improve Jig to your own requirements. The aim is to retain internal re-prgrammability to your development tooling.
One Jig project can be used against many different projects, even simultaneously (see Components)
I wrote Jig because I wanted to create new Clojure projects quickly without having to maintain the development harness for each one. As I make a small improvement to one development harness, I don't want the hassle of going through all my other projects to update them in the same way, but neither do I want dozens of development harnesses that differ from each other. I want one development harness, re-usable 'jig' that I can use against multiple Leiningen-based projects.
Jig lets you specify configuration for your components in a single configuration file. However, components can source their own configuration if desired.
Jig does not have opinions as to how you should build your applications. However, if does provide good support for writing Ring and Pedestal services should you wish to keep using Jig in your deployment. More details can be found below.
Pedestal boasts bidirectional routes, so that URIs can be generated from
route definitions rather than determined some other way. Jig provides a
url-for function in the Pedestal context, and defaults the
app-name and request to make it easy to generate paths that
make sense in the context of the page on which the link is placed.
It can be cost-effective for multiple web applications to share the same
JVM. Jig allows you to host web applications under contextual URI
prefixes. This is a feature made possible by the provision of the
url-for function, since 'portable' web applications can use this
to generate URIs for web links in their content, without resorting to
hard-coding URI paths.
Errors thrown from components that fail during initialization or start-up do not cause the entire system to fail. This reduces the number of times that you have to reboot the JVM. Only components that are successfully initialized are started, and only those that are successfully started are stopped. Any failures are indicated in the REPL, with full details and stack traces written to the log file..
Normally you'll be used to creating with a project with lein new,
cd'ing to the newly created directory, perhaps adding some plugins to
its project.clj file, and running lein. Working with Jig is
different, in that you run lein from the Jig project directory and
'point' the configuration at one (or more) of your existing Leiningen
projects.
Clone the Jig repository as you would any other Clojure project.
$ git clone https://github.com/juxt/jig
Configure Jig by copying in a config file into the
config/config.edn. You can skip this step if you want to see Jig
running in its default configuration which includes examples.
If you're using Emacs, load up Jig's project.clj and
M-x nrepl-jack-in
(that's usually bound to 'Control-c, Meta-j')
In the *nrepl* buffer that (eventually) appears, type
user> (go)
Alternatively, on the command line, type
lein repl
and then
user> (go)
Sync the application by calling reset
user> (reset)
Resetting the application will cause namespaces to reload (thanks to
Stuart's work in org.clojure/tools.namespace) such that the
application will be back in sync with the code-base. This is the feature
that makes development fast and interactive, and it's all thanks to
Stuart's hard work.
You should find yourself typing (reset) rather a lot, and soon
even that becomes burdensome. Here's some Emacs code you can paste into
your $HOME/.emacs.d/init.el to provide a shortcut.
(defun nrepl-reset ()
(interactive)
(save-some-buffers)
(set-buffer "*nrepl*")
(goto-char (point-max))
(insert "(user/reset)")
(nrepl-return))
(global-set-key (kbd "C-c r") 'nrepl-reset)or, if you are using cider:
(defun cider-repl-reset ()
(interactive)
(save-some-buffers)
(with-current-buffer (cider-current-repl-buffer)
(goto-char (point-max))
(insert "(user/reset)")
(cider-repl-return)))
(global-set-key (kbd "C-c r") 'cider-repl-reset)After re-evaluating (or restarting Emacs) you'll be able to reset the application using 'Control-c r'.
A configuration specifies :
- which components you want to include in your system
- where to find these components (the Leiningen project) and
- the configuration settings they will get
Usually you've have one configuration file per project per environment (e.g. dev, uat, prod).
By default, Jig looks for a config/config.edn file, but you can override this by placing a config file in $HOME/.jig/config.edn (where $HOME is your home directory). If a configuration file can't be found, a default will be used.
A configuration is a map which usually contains a :jig/components key listing the
components in a map (each key in the map is the component's label). You can get a good idea of the format by looking at the config/default.edn file.
It's also possible to link to other configuration files, which are merged into a single config :
{
:jig/include ["/home/malcolm/src/octopus/config.clj" "/home/malcolm/src/juxtweb/config.clj"]
}While .edn files are preferred, you can use .clj if you want
to evaluate Clojure expressions, such as calculations.
Important: To avoid injection attacks, never use the .clj suffix
for configuration you don't entirely control, including user submitted
configuration.
A very useful trick is to use Clojure's #= reader macro in the
$HOME/.jig/config.clj (you can't use an .edn file for this
trick). A useful pattern is to load in the default configuration and
make tweaks to it for the environment you're running Jig in. This helps
to prevent a proliferation of diveraging configuration files, which is a
common problem with multiple environments in any configuration
system. Each configuration can inherit from the defaults and use a
Clojure program to derive a modified version. What better language than
Clojure to tweak a map?
Here's an example.
#=(eval
(->
;; Start with the original configuration, stored in the git repo
(clojure.core/read-string (slurp (str (System/getProperty "user.home") "/src/my-proj/config.clj")))
;; Add a 'private' component
(assoc-in [:jig/components :firefox-reloader]
{:jig/component 'jig.web.firefox-reload/Component
:jig/dependencies [:service :cljs-server]
:jig.web.firefox-reload/host "localhost"
:jig.web.firefox-reload/port 32000
})
;; Oh dear, some tricky path munging. We have to change lots of paths!
;; Let's walk the tree.
;; Due to symlinks not working well on Dropbox mounts,
;; replace all instances of relative paths with absolute ones
((partial clojure.walk/postwalk
(fn [x]
(if-let
[path (and (string? x)
(second (re-matches #"\.\./my-proj(.*)" x)))]
(str "/home/malcolm/another-path/src/my-proj" path)
x))))))You can write your own components by defining a type or record. At the
very least it needs to implement the jig.Lifecycle protocol.
(:ns org.example.core
(:import (jig Lifecycle)))
(deftype Component [config]
Lifecycle
(init [_ system] system)
(start [_ system] system)
(stop [_ system] system))In Stuart's reloaded workflow, the init function is responsible
for creating the System. In Jig's component model, the system map is
threaded through each component's init function, giving it a
chance to add stuff. Likewise for the start and stop
functions. The minimum you need to do is return the original system, for
the next component in the chain.
Once you have declared your component, you need to reference it in the
config file config/config.edn. Jig needs to know which components
you want activated. If this file doesn't already exist, copy over the
contents from config/sample.config.edn. If you need to evaluate
Clojure expressions in your config, use a .clj suffix,
e.g. config/config.clj
For example...
{:jig/components
{:hello-app {:jig/component org.example.core/Component}}}Components will be instantiated with a single argument: the component's
configuration value as specified in the config file. So if you want to
pass configuration to your component, an easy option is to add it to the
component's entry in the config/config.edn file.
If you want to see the state of the system at any time, it's available from the REPL
user> system
It's nicer if you have pretty printing enabled at the REPL.
M-x cust-var<RET>nrepl-use-pretty-printing
Alternatively, you can be explicit.
user> (pprint system)
Sometimes a component will rely on the existence of
another. Dependencies can be specified in the component configuration
under the :jig/dependencies key, where the value is a vector of
keys to other components in the configuration.
For example, let's suppose component Y is dependent on component X.
{:jig/components
{"X" {:jig/component org.example.core/X}
"Y" {:jig/component org.example.core/Y
:jig/dependencies ["X"]}}}You can also view the component dependency graph from the REPL :-
user> (graph)
There are some situations where you want state to survive across
resets. Values that are stored in the system anywhere under the
:jig/safe branch will not be purged. In the vast majority of cases
you should prefer state to be reinitialised on a reset.
Jig comes with its own components, providing useful functionality and demonstrate how components are written. Each component is configurable. If you need a component which isn't in this list, I am happy to provide it.
Provides a Jetty service that can be used by Ring applications.
Amalgamates Compojure routes contributed by other components into a single handler.
Pulls the latest code from a remote git repository. This can be useful as part of an automatic continuous delivery mechanism.
Provides a reload capability, invokable as a JMX operation.
Clears out the stencil cache to ensure stale Mustache templates do not survive a system reset.
Purges an nginx reverse proxy to ensure cached pages do not survive a system reset.
Provides an nREPL server, useful if the system isn't started with
lein repl but an nREPL service is still desired.
A trigger to get Firefox to reload the current page upon every rest. Requires the Remote Control addon)is installed and enabled (the icon should be green).
Provides a Pedestal service on a Tomcat or Jetty listener, and ensures that the System is made available to each Pedestal handler. This unifies Stuart's approach with the Pedestal framework, allowing you to enjoy Stuart's rapid development workflow while writing Pedestal services.
One of the major benefits of Pedestal over Ring is the support for bi-directionality between routes and handlers. For me, this is a stand-out feature because of the importance of hyperlinks, both in web pages and RESTful web applications.
Asking the library to generate URLs for you, rather than hard-coding
them in your application, reduces the risk of broken links and
maintenance cost. Jig injects a useful function into the Pedestal
context, under the :url-for key, that lets you generate URLs
relative to request you are processing. By default, URLs are generated
to handler within the same application, but you can specify
:app-name to target other applications hosted in the same JVM.
Look at the use of url-for in the example below. See how easy it
is to generate URLs to target other Pedestal handlers. Nice.
(defhandler my-index-page [request]
{:status 200
:headers {"Content-Type" "text/plain"}
:body "Hello World!})
(defbefore my-root-page [{:keys [url-for] :as context}]
(assoc context :response
(ring.util.response/redirect
;; Look, no nasty hard-coded URLs!
(url-for ::my-index-page))))Provides a web application abstraction on top of
jig.web.server/Component. Other components can add Pedestal routes
to a web application component. These are then combined to form a
complete Pedestal route table.
Applications can share the same jig.web.server/Component, allowing
for 'virtual hosts'. Applications can specify a different host, scheme
or be rooted at a sub-context. This are really useful for hosting
different Clojure-powered websites on the same JVM.
These examples assume a little knowledge about Pedestal, see http://pedestal.io/documentation/service-routing/ for more details.
Adding a step to compile ClojureScript on every reset can be accomplished with this component.
:cljs-builder
{:jig/component jig.cljs/Builder
:jig/project "../myproj/project.clj"
:output-dir "../myproj/target/js"
:output-to "../myproj/target/js/main.js"
:source-map "../myproj/target/js/main.js.map"
:optimizations :none
:clean-build false
}Setting :clean-build to true will cause the output directory to be
deleted on every reset, otherwise an incremental build will be
performed. The compiler environment is stored under the :jig/safe
branch to make incremental builds possible. Incremental builds are a lot
faster and are therefore enabled by default.
You may want to configure different builder components for different pages, or for different optimizations. Multiple builds can be achieved by adding multiple instances of this component to your configuration.
Once the JavaScript and source-map files have been generated, it is
often useful to be able to serve it from a web server. A minimum of two
dependencies must be specified, one should be a jig.cljs/Builder
component instance, the other should be the a
jig.web.app/Component component instance. A web context should be
specified, under which resources will be available.
:cljs-server
{:jig/component jig.cljs/FileServer
:jig/dependencies [:cljs-builder :web]
:jig.web/context "/js"
}You point Jig at your own projects by specifying a :jig/project
configuration entry which specifies the project containing the component you wish to include.
:juxtweb/service {:jig/component pro.juxt.website.core/Component
:jig/dependencies [:juxtweb/web]
:jig.web/app-name :juxtweb/web
:jig/project "../juxtweb/project.clj"}Leiningen dependencies that are added to a project during development are automatically can be added to the classpath, so you don't have to restart the JVM if you are simply adding a dependency to a project.
You can run Jig in production the same way as you run in development. If you are using a single project which depends on Jig (or at least one Jig extension) you can uberjar your project.
Create a main.clj or other similar namespace with the following content.
(ns main (:gen-class))
(defn -main [& args] (user/go))In your project.clj, add the following
:main main :aot [main]Or see https://github.com/mastodonc/kixi.hecuba for an example of a project which deploys this way.
I don't see any reason why others couldn't create their own 'jigs' (a jig is a separate Leiningen project after all). Although this project is named eponymously, there can be others customized for specialist contexts (either new projects, or forks of this one).
If that happens and there's a need to share components between jigs (for
reasons of component portability) then it will make sense to promote the
Lifecycle protocol (and maybe others) to a common library that
different jigs can use.
If an external project depends on a third-party jar which isn't on Jig's
own classpath, then it will not have been visible to Clojure when
clojure.core is first loaded. This means that any data readers declared
in /data_readers.clj will not work. If you use the
edn/read-string in these projects you cannot assume the
*data-readers* dynamic var will contain the data readers declared
in the third-party jar. The workaround is to use the 2-arg form of
edn/read edn/read-string and supply the readers explicitly in the option
map.
For example, when loading Datomic data from a file, you would use the following form :
(edn/read-string
{:readers {'db/id datomic.db/id-literal
'db/fn datomic.function/construct
'base64 datomic.codec/base-64-literal}}
"my-data.edn")If a project has a /user.clj file in one of its source
directories, then it will be loaded on a reset and override Jig's user
namespace. Jig will then stop working. The current workaround is to
ensure such a file isn't visible to Jig, either by renaming it or by
removing the source directory containing it in the project's
project.clj file. It is hoped that future versions of Jig will
avoid this issue by disabling the loading of such files by tools.namespace.
Clojure already does code reloading! Why do I need all this stuff?
Clojure, being a LISP, allows you to reload functions at will. But Stuart's pattern (which Jig builds on) extends this to state, compile-time macros, protocols/types/records and multi-methods. Experienced Clojure developers know when to reload some namespaces and when to restart the JVM. It seems better to raise the abstraction so you don't have to think about these technicalities all the time, just 'reset' each time you want consistency while you develop.
See this comment for a better explanation: https://news.ycombinator.com/item?id=5819912
Jig is trying to provide you with a better development experience, while nudging you towards a modular architecture that will help you when your system grows to a certain size.
Where can I find examples of projects using Jig?
Mastodon C's Hecuba project uses Jig for development, deploying using lein uberjar.
JUXT Accounting is a full application that is developed with Jig 1.0.x.
JUXT's website also uses Jig 1.0.x, for both development and deployment.
What's the relationship between Jig and Up?
Up is deprecated, Jig replaces it (and has stolen all the worthy bits).
The main difference between Up and Jig is that Up added config to the Leiningen project and the dependency relation between projects and infrastructure has been reversed. In due course Jig will add a protocol that components can extend to communicate with other components over a System bus (core.async channel), and will then provide a superset of Up's functionality. Jig is better.
Don't forget the system is available in the REPL, as user/system. That's useful for testing what functions are returning.
You can check what's in the system by outputting it to the
console. (keys user/system) is often useful if the system gets
very large and you need to navigate it.
By default, Jig configures logback and logs are written to
$HOME/.jig/logs. Events such as component init, startup and
shutdown, and exceptions that occur during these phases, are written to
here. When the REPL tells you to check the log files, this is where you
should look.
You should return the system value, modified if necessary, from each component function. If you don't return any value, Jig will detect this and throw an error.
Make sure that you do modify the system, but then return using the symbol representing the unmodified system. That's a common mistake. Jig won't detect that you have returned the same system it gave you, because that's normal behaviour.
java.lang.IllegalArgumentException: No implementation of method: :init of protocol: #'jig/Lifecycle found for class
Ensure you require jig before importing the jig/Lifecycle protocol.
By default, a console is provided at http://localhost:8001.
Currently this supports the following (limited) functionality:
- This README
- A system map browser
- Dynamically served Codox API documentation
- A list of TODOs
- Triggering a reload
Security advisory: Since this console allows remote reloads, ensure that you disable this console if you are intending to run production applications on Jig, or secure access to whatever port you run it on.
Many people have asked for a demonstration of how to configure a simple
Ring application. This is provided as a Sudoku example in the
examples/ directory. It is incorporated in the default
configuration and the website is accessible at http://localhost:8091/sudoku.html
This is a network service built on Netty. It shows how easy it is to use Java service libraries directly using only Clojure and Jig. Please refer to the README.md in examples/mqtt-broker for more details.
Copyright © 2013 JUXT. All rights reserved.
The use and distribution terms for this software are covered by the Eclipse Public License 1.0 which can be found in the file epl-v10.html at the root of this distribution. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software.