ironhide

Ironhide, the data transformer.

Idea

Create a runtime agnostic bidirectional data-driven transformation domain-specific language for fun and profit.

Problem

There are a lot of data, which has to be represented in different shapes. For this reason created a lot of query/transformation languages such as XSLT, AWK, etc, but most of them have a significant disadvantage: they work only in one direction and you can't get original data from the result of transformation.

It worth noting that there are other languages like boomerang, which doesn't have this significant (in some cases) weakness, but have others : )

Simplified real life example of representation person name in different systems:

"form": {
  "name": "Firstname Lastname"
}

"fhir": {
  "name": {
    "given": [
      "Firstname"
    ],
    "family": "Lastname"
  }
}

By different reasons both respresentations should be availiable and moreover syncronized. Syncronization can be done by implementing and applying when needed two function f and f ^-1 for each field or subset of fields, but you already probably know how hard to maintain such code?)

It is hard to implement such functions for big nested tree data structures and much harder to keep f ^-1 in sync with f.

Solution

ironhide is an attempt to create a bidirectional data transformation language described by a data structure stored in EDN (you can think about edn like a better JSON). ironhide still in early stage of development, but already covers some practical usecases. It's also declarative, bidirectional, data-driven and simple.

Following code in ironhide solves example above:

#:ih{:direction [:fhir :form]
     :rules     [{:form [:name :ihs/str<->vector [0]]
                  :fhir [:name [0] :given [0]]}

                 {:form [:name :ihs/str<->vector [1]]
                  :fhir [:name [0] :family]}]}

The direction of transformation controlled by :ih/direction key and this simple snippet allows to transform data in both ways out of the box.

Full grammar defined using clojure.spec in core namespace.

Usage

This section contains examples of clojure ironhide interpreter usage with a little explanation, to get the taste of dsl capabilities. More detailed info provided in Description section.

Add to :deps in deps.edn:

healthsamurai/ironhide {:mvn/version "RELEASE"}

hello_world.clj:

(ns hello-world.core
  (:require [ironhide.core :as ih]))

;; (ih/execute shell)
;; (ih/get-data shell)

Field to field mapping

(def update-name-shell
  #:ih{:direction [:form :form-2]

       :rules [{:form   [:name]
                :form-2 [:fullname]}]
       :data  {:form   {:name "Full Name"}
               :form-2 {:fullname "Old Name"}}})

(get-data update-name-shell)
;; => {:form {:name "Full Name"}, :form-2 {:fullname "Full Name"}}

Create missing fields

(def create-name-shell
  #:ih{:direction [:form :form-2]

       :rules [{:form   [:name]
                :form-2 [:fullname]}]
       :data  {:form   {:name "Full Name"}}})

(get-data create-name-shell)
;; => {:form {:name "Full Name"}, :form-2 {:fullname "Full Name"}}

Default values

(def default-name-shell
  #:ih{:direction [:form :form-2]

       :values {:person/name "Name not provided by the form"}
       :rules  [{:form     [:name]
                 :form-2   [:fullname]
                 :ih/value {:form-2 [:ih/values :person/name]}}]
       :data   {:form   {}
                :form-2 {:fullname "Old Name"}}})

(get-data default-name-shell)
;; => {:form {}, :form-2 {:fullname "Name not provided by the form"}}

Sight for string

(def sight-name-shell
  #:ih{:direction [:form :fhir]

       :rules [{:form   [:name :ihs/str<->vector [0]]
                :fhir [:name [0] :given [0]]}]
       :data  {:form   {:name "Full Name"}}})

(get-data sight-name-shell)
;; => {:form {:name "Full Name"}, :fhir {:name [{:given ["Full"]}]}}

See Sight section for the detailed explanation.

Update and create if not-exists

(def create-and-update-phone-shell
  #:ih{:direction [:form :fhir]

       :rules [{:form [:phones [:*]]
                :fhir [:telecom [:* {:system "phone"}] :value]}]
       :data  {:form {:phones ["+1 111" "+2 222"]}
               :fhir {:telecom [{:system "phone"
                                 :use    "home"
                                 :value  "+3 333"}
                                {:system "email"
                                 :value  "test@example.com"}]}}})

(get-data create-and-update-phone-shell)
;; =>
;; {:form {:phones ["+1 111" "+2 222"]},
;;  :fhir {:telecom [{:system "phone", :use "home", :value "+1 111"}
;;                   {:system "email", :value "test@example.com"}
;;                   {:system "phone", :value "+2 222"}]}}

Micro

(def micro-name-shell
  #:ih{:direction [:fhir :form] ;; !!!

       :micros #:ihm {:name<->vector [:name {:ih/sight  :ihs/str<->vector
                                             :separator ", "}]}

       :rules [{:form [:ihm/name<->vector [0]]
                :fhir [:name [0] :given [0]]}
               {:form [:ihm/name<->vector [1]]
                :fhir [:name [0] :family]}]
       :data  {:form {:name "Full, Name"}
               :fhir {:name [{:given ["First"] :family "Family"}]}}})

(get-data micro-name-shell :form)
;; => {:name "First, Family"}

Micros can be parametrized in the same way as sights.

Description

shell is a tree datastructure, which contains declaration of transformation rules + data itself. ironhide interpreter can execute shell's.

It consists of few main parts:

:ih/data a data for transformation
:ih/values similar to previous one, but used mostly for default values
:ih/micros shortcuts for long repetitive pathes in rules
:ih/direction default transformation direction (rule can define its own)
:ih/rules vector of transformation rules

Simple shell executed with get-data:

(get-data
 #:ih{:direction [:form :fhir]
      :data      {:form {:first-name "Firstname"}
                  :fhir {}}
      :rules     [{:form [:first-name]
                   :fhir [:name [0] :given [0]]}]})
;; => {:form {:first-name "Firstname"}, :fhir {:name [{:given ["Firstname"]}]}}

Path and pelem

Path is a vector consist of pelems (path elements), which describes how to get to some node in data-sources. Something similar to XPath, JsonPath, but not exactly.

There are few types of pelems:

mkey
vnav
sight
micro

term	definition
`mkey`	a simple edn `:keyword`, which tells `shell` executor to navigate to specific key in the map.
`vnav`	a vector, which consists of `vkey` and optional `vfilter`.
`vkey`	an `index` (some non-negative integer) or `wildcard` `:*` (keyword).
`vfilter`	a map used for pattern matching and templating
`sight`	`:ihs/` namespaced keyword or `{:ih/sight :ihs/sight-name :arg1 :value1}`
`micro`	`:ihm/` namespaced keyword or `{:ih/micro :ihm/micro-name :arg1 :value1}`

Full grammar defined in core namespace.

Example of paths and get-values results:

;; {:k1 {:k2 :v3}}
[:k1] ;; => [[{:k2 :v3}]]
[:k1 :k2] ;; => [[:v3]]

;; [{:a :b} {:k :v :k1 :v2}]
[[0]] ;; => [[{:a :b}]]
[[:*]] ;; => [[0 {:a :b}] [1 {:k :v :k1 :v1}]]
[[1 {:a :b}]] ;; => [[nil]]
[[:* {:k :v}]] ;; => [[0 {:k :v :k1 :v1}]]

;; {:name "Firstname, Secondname"}
[:name :ihs/str<->vector [0]] ;; => [["Firstname,"]]
;; [:name {:ih/sight :ihs/str<->vector :separator ", "} [0]]
[:ihm/first-name] ;; => [["Firstname"]]

;; [[1 2] [3 4 5]]

get-values always returns a vector of indexed values. Each wildcard inside the path creates one dimension of index. Source and sink path should have same number of wildcards to make transformation possible. ironhide interpreter will align the shape automatically (without deleting existing data).

(get-values
 [[:v1 :v2] [:v3 :v4 :v5]]
 [[:*] [:*]])
;; => [[0 0 :v1] [0 1 :v2] [1 0 :v3] [1 1 :v4] [1 2 :v5]]
;; the result of get-values is a vector of indexed values
;; 1 2 - is a multi-dimensional index

Sight

sight is a special type of pelem, which allows to percieve current node of data-source differently. It's useful when you want to treat a string as a vector of words for example:

;; {:name "Firstname Secondname"}
[:name :ihs/str<->vector [0]] ;; => [["Firstname"]]

It allows to navigate inside node of data-source differently and more preciesly, but don't change original structure of it.

It's possible to extend sights by defining ironhide.core/get-global-sight method or using nested sights.

Micro

micro is a parametrized shortcut for part of the path.

(microexpand-path
 #:ih{:micros #:ihm {:name [:name [:index] :given [0]]}}
 [:ihm/name])
;; => [:name [:index] :given [0]]

(microexpand-path
 #:ih{:micros #:ihm {:name [:name [:index] :given [0]]}}
 [{:ih/micro :ihm/name :index 10}])
;; => [:name [10] :given [0]]

Default values for micros not supported yet.

Rule

Rule specifies relation between parts of data-sources. It is a map, which can contain few different key types:

data-source name, which associated with path to exact part of data-source
:ih/direction, which associated with a pair of source and sink data-sources
:ih/defaults, which associated with map of data-source name keys and path-to-default-value values

{:form [:firstname]
 :fhir [:name [0] :given [0]]

 :ih/defaults  {:fhir [:ih/values :firstname]}
 :ih/direction [:form :fhir]}

Thanks

Special thanks to:

Nathan Marz for specter
Nikolai Ryzhikov for matcho and 2way

Contribution

PRs are welcome, but merging not guaranteed. Create issue or contact abcdw if you need or want.

License

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.

abcdw/ironhide

ironhide

Idea

Problem

Solution

Usage

Field to field mapping

Create missing fields

Default values

Sight for string

Update and create if not-exists

Micro

Description

Path and pelem

Sight

Micro

Rule

Thanks

Contribution

License