Small Clojure Interpreter
I want a limited dialect of Clojure for a single-purpose, scripted application. Sci will fit nicely.
— @tiagoluchini
Quickstart
Use from Clojure(Script)
(require '[sci.core :as sci])
(sci/eval-string "(inc 1)") => ;; 2
(sci/eval-string "(inc x)" {:bindings {'x 2}}) ;;=> 3
Why
You want to evaluate code from user input, or use Clojure for a DSL inside your
project, but eval
isn't safe or simply doesn't work.
This library works with:
- Clojure on the JVM
- Clojure compiled with GraalVM native
- ClojureScript, even when compiled with
:advanced
, and (as a consequence) JavaScript
Projects using sci
Sci is used in:
- Babashka. A Clojure scripting tool that plays well with Bash.
- Bootleg. An HTML templating CLI.
- Bytefield-svg. NodeJS library to generate byte field diagrams.
- Chlorine. Socket-REPL and nREPL package for Atom editor.
- Clj-kondo. A Clojure linter that sparks joy.
- Closh. Bash-like shell based on Clojure. GraalVM port is work in progress.
- Dad. A configuration management tool.
- Firn. Org-mode static site generator.
- Jet. CLI to convert between JSON, EDN and Transit.
- Logseq. A local-only outliner notebook which supports both Markdown and Org mode.
- Malli. Plain data Schemas for Clojure/Script.
- PCP. Clojure Processor (PHP replacement).
- PGMig. Fast Standalone PostgreSQL Migration Runner.
- Prose. Alternate syntax for Clojure, similar to what Pollen brings to Racket.
- SICMUtils. Computer Algebra System in Clojure, tailored for math and physics investigations.
- Spire. Pragmatic provisioning using Clojure.
Status
Experimental. Breaking changes are expected to happen at this phase.
Installation
Use as a dependency:
API docs
For Clojure, see the generated codox documentation.
Usage
The main API function is sci.core/eval-string
which takes a string to evaluate
and an optional options map.
In sci
, defn
does not mutate the outside world, only the evaluation
context inside a call to sci/eval-string
.
By default sci
only enables access to most of the Clojure core functions.
More functions can be enabled, at your own risk, by using :bindings
. Normally
you would use sci's version of println
but here, for the purposes of
demonstration, we use use Clojure's version of println
instead:
user=> (require '[sci.core :as sci])
user=> (sci/eval-string "(println \"hello\")" {:bindings {'println println}})
hello
nil
It is also possible to provide namespaces which can be required:
user=> (def opts {:namespaces {'foo.bar {'println println}}})
user=> (sci/eval-string "(require '[foo.bar :as lib]) (lib/println \"hello\")" opts)
hello
nil
In fact {:bindings ...}
is just shorthand for {:namespaces {'user ...}}
.
You can provide a list of allowed symbols. Using other symbols causes an exception:
user=> (sci/eval-string "(inc 1)" {:allow '[inc]})
2
user=> (sci/eval-string "(dec 1)" {:allow '[inc]})
ExceptionInfo dec is not allowed! [at line 1, column 2] clojure.core/ex-info (core.clj:4739)
Providing a list of disallowed symbols has the opposite effect:
user=> (sci/eval-string "(inc 1)" {:deny '[inc]})
ExceptionInfo inc is not allowed! [at line 1, column 2] clojure.core/ex-info (core.clj:4739)
Providing a macro as a binding can be done by providing a normal function that:
- has
:sci/macro
on the metadata set totrue
- has two extra arguments at the start for
&form
and&env
:
user=> (def do-twice ^:sci/macro (fn [_&form _&env x] (list 'do x x)))
user=> (sci/eval-string "(do-twice (f))" {:bindings {'do-twice do-twice 'f #(println "hello")}})
hello
hello
nil
Vars
To remain safe and sandboxed, sci evaluated Clojure does not have access to Clojure runtime vars. Sci has its own var type, distinguished from Clojure vars.
In a sci program these vars are created with def
and defn
just like in normal Clojure:
(def x 1)
(defn foo [] x)
(foo) ;;=> 1
(def x 2)
(foo) ;;=> 2
Dynamic vars with thread-local bindings are also supported:
(def ^:dynamic *x* 1)
(binding [*x* 10] x) ;;=> 10
(binding [*x* 10] (set! x 12) x) ;;=> 12
x ;;=> 1
Pre-creating vars that can be used in a sci program can be done using
sci/new-var
:
(def x (sci/new-var 'x 10))
(sci/eval-string "(inc x)" {:bindings {'x x}}) ;;=> 11
To create a dynamic sci var you can set metadata or use sci/new-dynamic-var
:
(require '[sci.core] :as sci)
(def x1 (sci/new-var 'x 10 {:dynamic true}))
(sci/eval-string "(binding [*x* 12] (inc *x*))" {:bindings {'*x* x1}}) ;;=> 13
(def x2 (sci/new-dynamic-var 'x 10))
(sci/eval-string "(binding [*x* 12] (inc *x*))" {:bindings {'*x* x2}}) ;;=> 13
Pre-created sci vars can also be externally rebound:
(def x (sci/new-dynamic-var 'x 10))
(sci/binding [x 11] (sci/eval-string "(inc *x*)" {:bindings {'*x* x2}})) ;;=> 11
The dynamic vars *in*
, *out*
, *err*
in a sci program correspond to the
dynamic sci vars sci.core/in
, sci.core/out
and sci.core/err
in the API. These
vars can be rebound as well:
(def sw (java.io.StringWriter.))
(sci/binding [sci/out sw] (sci/eval-string "(println \"hello\")")) ;;=> nil
(str sw) ;;=> "hello\n"
A shorthand for rebinding sci/out
is sci/with-out-str
:
(sci/with-out-str (sci/eval-string "(println \"hello\")")) ;;=> "hello\n"
Stdout and stdin
To enable printing to stdout
and reading from stdin
you can sci bind
sci.core/out
and sci.core/in
to *out*
and *in*
respectively:
(sci/binding [sci/out *out*
sci/in *in*]
(sci/eval-string "(print \"Type your name!\n> \")")
(sci/eval-string "(flush)")
(let [name (sci/eval-string "(read-line)")]
(sci/eval-string "(printf \"Hello %s!\" name)
(flush)"
{:bindings {'name name}})))
Type your name!
> Michiel
Hello Michiel!
When adding a Clojure function to sci that interacts with *out*
(or *in*
or *err*
), you
can hook it up to sci's world. For example, a Clojure function that writes to *out*
can be Clojure bound to sci's out
:
user=> (defn foo [] (println "yello!"))
#'user/foo
user=> ;; without binding *out* to sci's out, the Clojure function will use its default *out*:
user=> (sci/eval-string "(with-out-str (foo))" {:bindings {'foo foo}})
yello!
""
;; let's hook foo up to sci's world:
user=> (defn wrapped-foo [] (binding [*out* @sci/out] (foo)))
#'user/wrapped-foo
user=> (sci/eval-string "(with-out-str (foo))" {:bindings {'foo wrapped-foo}})
"yello!\n"
Futures
Creating threads with future
and pmap
is disabled by default, but can be
enabled by requiring sci.addons.future
and applying the sci.addons.future/install
function
to the sci options:
(ns my.sci.app
(:require
[sci.core :as sci]
[sci.addons.future :as future]))
(sci/eval-string "@(future (inc x))"
(-> {:bindings {'x 1}}
(future/install)))
;;=> 2
For conveying thread-local sci bindings to an external future
use
sci.core/future
:
(ns my.sci.app
(:require
[sci.core :as sci]
[sci.addons.future :as future]))
(def x (sci/new-dynamic-var 'x 10))
@(sci/binding [x 11]
(sci/future
(sci/eval-string "@(future (inc x))"
(-> {:bindings {'x @x}}
(future/install)))))
;;=> 12
Classes
Adding support for classes is done via the :classes
option:
(sci/eval-string "(java.util.UUID/randomUUID)"
{:classes {'java.util.UUID java.util.UUID}})
;;=> #uuid "312ba519-37e2-4109-b164-97fb140b57b0"
To make this work with GraalVM
you will also need to add an entry to your
reflection
config
for this class. Also see reflection.json
.
State
Sci uses an atom to keep track of state changes like newly defined namespaces
and vars. You can carry this state over from one call to another by providing
the atom yourself as the value for the :env
key:
(def env (atom {})
(sci/eval-string "(defn foo [] :foo)" {:env env})
(sci/eval-string "(foo)" {:env env}) ;;=> :foo
The contents of the the :env
atom should be considered implementation detail.
Using an :env
atom you are allowed to change options at each invocation of
eval-string
. If your use case doesn't require this, the recommendation is to
use a sci context instead.
A sci context is derived once from options as documented in
sci.core/eval-string
and contains the runtime state of a sci session.
(def opts {:namespaces {'foo.bar {'x 1}}})
(def sci-ctx (sci/init opts))
Once created, a sci context should be considered final and should not be mutated by the user. The contents of the sci context should be considered implementation detail.
The sci context can be re-used over successive invocations of
sci.core/eval-string*
.
The major difference between eval-string
and eval-string*
is that
eval-string
will call init
on the passed options and will pass that through
to eval-string*
. When you create a sci context yourself, you can skip the
extra work that eval-string
does and work directly with eval-string*
.
(sci/eval-string* sci-ctx "foo.bar/x") ;;=> 1
(sci/eval-string* sci-ctx "(ns foo.bar) (def x 2) x") ;;=> 2
(sci/eval-string* sci-ctx "foo.bar/x") ;;=> 2
In a multi-user environment it can be useful to give each user their own
context. This can already be achieved with eval-string
, but for performance
reasons it may be desirable to initialize a shared context. This shared context
can then be forked for each user so that changes in one user's context
aren't visible to other users:
(def forked (sci/fork sci-ctx))
(sci/eval-string* forked "(def forked 1)")
(sci/eval-string* forked "forked") ;;=> 1
(sci/eval-string* sci-ctx "forked") ;;=> Could not resolved symbol: forked
Implementing require and load-file
Sci supports implementation of code loading via a function hook that is invoked
by sci's internal implementation of require
. The job of this function is to
find and return the source code for the requested namespace. This passed-in
function will be called with a single argument that is a hashmap with a key
:namespace
. The value for this key will be the symbol of the requested
namespace.
This function can return a hashmap with the keys :file
(containing the
filename to be used in error messages) and :source
(containing the source code
text) and sci will evaluate that source code to satisfy the
require. Alternatively the function can return nil
which will result in sci
throwing an exception that the namespace can not be found.
This custom function is passed into the sci context under the :load-fn
key as
shown below.
(defn load-fn [{:keys [namespace]}]
(when (= namespace 'foo)
{:file "foo.clj"
:source "(ns foo) (def val :foo)"}))
(sci/eval-string "(require '[foo :as fu]) fu/val" {:load-fn load-fn})
;;=> :foo
Note that internally specified namespaces (either those within sci itself or
those mounted under the :namespaces
context setting) will be utilised first
and load-fn will not be called in those cases, unless :reload
or :reload-all
are used:
(sci/eval-string
"(require '[foo :as fu])
fu/val"
{:load-fn load-fn
:namespaces {'foo {'val (sci/new-var 'val :internal)}}})
;;=> :internal
(sci/eval-string
"(require '[foo :as fu] :reload)
fu/val"
{:load-fn load-fn
:namespaces {'foo {'val (sci/new-var 'val :internal)}}})
;;=> :foo
Another option for loading code is to provide an implementation of
clojure.core/load-file
. An example is presented here.
(ns my.sci.app
(:require [sci.core :as sci]
[clojure.java.io :as io]))
(spit "example1.clj" "(defn foo [] :foo)")
(spit "example2.clj" "(load-file \"example1.clj\")")
(let [env (atom {})
opts {:env env}
load-file (fn [file]
(let [file (io/file file)
source (slurp file)]
(sci/with-bindings
{sci/ns @sci/ns
sci/file (.getAbsolutePath file)}
(sci/eval-string source opts))))
opts (assoc-in opts [:namespaces 'clojure.core 'load-file] load-file)]
(sci/eval-string "(load-file \"example2.clj\") (foo)" opts))
;;=> :foo
REPL
Implementing a REPL can be done using the following functions:
sci/reader
: returns reader for parsing source code, either from a string orio/reader
sci/parse-next
: returns next form from readersci/eval-form
: evaluates form returned byparse-next
.
See examples for examples for both Clojure and ClojureScript. Run instructions are included at the bottom of each example.
To include an nREPL server in your sci-based project, you can use babashka.nrepl.
Location metadata
Sci includes location metadata on forms that can carry it:
(sci/eval-string "(meta [1 2 3])")
;;=> {:line 1, :column 7, :end-line 1, :end-column 14}
This metadata is used by sci for error reporting.
GraalVM
Random numbers
To make the rand-*
functions behave well when compiling to a GraalVM native binary, use this setting:
--initialize-at-run-time=java.lang.Math\$RandomNumberGeneratorHolder
Java 11
To use sci with GraalVM java11 override the dependency
[borkdude/sci.impl.reflector "0.0.1"]
to [borkdude/sci.impl.reflector "0.0.1-java11]
in your project.clj
or deps.edn
.
Also you'll likely need a fix for clojure.lang.Reflector
:
See clj-graal-docs and clj-reflector-graal-java11-fix.
Use as native shared library
To use sci as a native shared library from e.g. C, C++, Rust, read this tutorial.
Limitations
Currently sci doesn't support deftype
and definterface
.
Test
Required: lein
, the clojure
CLI and GraalVM.
To succesfully run the GraalVM tests, you will have to compile the binary first
with script/compile
.
To run all tests:
script/test/all
For running individual tests, see the scripts in script/test
.
Thanks
- adgoji for financial support.
- Clojurists Together for financial support.
- Lee Read for the logo.
- contributors and other users posting issues with bug reports and ideas
License
Copyright © 2019-2020 Michiel Borkent
Distributed under the Eclipse Public License 1.0. This project contains code from Clojure and ClojureScript which are also licensed under the EPL 1.0. See LICENSE.