Clara rules throws a Null Pointer Exception even though there is no Null values
Opened this issue · 8 comments
Description
I am building a ETL Software and I use clara rules to clean data by applying certain constraints (rules) to shape the data as I want. Rules always work well, until I make my ETL Software a multi-threading parallel synchronous processor. After this, I started running stress testing on the software and I was able to caught an exception that is weird because Clara is screaming at me for trying to divide a null value.
Steps to reproduce
- Protocol Event
(defrecord Event [_id carrier comment tank-category rate reference uom volume ])
- Rule
(defrule convert-to-barrels
"If the unit of measure is gallons convert it to barrels."
[?event <- Event (= uom :ga) (some? volume)]
=>
(as-> (get @evnts (:_id ?event)) _
(assoc _ :uom :barrels)
(update _ :volume #(/ % 42))
(update _ :rate #(/ % 42))
(swap! evnts #(assoc % (:_id ?event) _)))
- Fire rules
(defn apply-rules
"Apply the rules in the namespace to the given events."
[events]
(reset! evnts (seq/seq-map->map events :_id))
(-> events
(mk-session 'etl.software.flowers.rules)
(insert-all events)
(fire-rules))
(vals @evnts))
Actual Behavior
clojure.lang.ExceptionInfo: Exception in etl.software.flowers.rules/convert-to-barrels with bindings
{:?event #etl.software.flowers.core.Event{:_id "733B772B94101049D1C8520FF8A9FB109CBC66BDB6A577E8A0E6311698837E12",
:created "2019-10-02T14:36:50Z",
:deleted false,
:rate 48763.0,
:reference "DFP0248029",
:start #object[java.time.ZonedDateTime 0x2a488968 "2019-10-04T01:03Z[UTC]"],
:stop #object[java.time.ZonedDateTime 0x6de70bdd "2019-10-04T23:00:49Z[UTC]"],
:uom :ga,
:volume 1071000}}
at clara.rules.engine$fire_rules_STAR_$fn__10629$fn__10630.invoke(engine.cljc:1797)
at clara.rules.engine$fire_rules_STAR_$fn__10629.invoke(engine.cljc:1763)
at clara.rules.engine$fire_rules_STAR_.invokeStatic(engine.cljc:1756)
at clara.rules.engine$fire_rules_STAR_.invoke(engine.cljc:1720)
at clara.rules.engine.LocalSession.fire_rules(engine.cljc:1909)
at clara.rules$fire_rules.invokeStatic(rules.cljc:44)
at clara.rules$fire_rules.invoke(rules.cljc:29)
at etl.software.flowers.rules$apply_rules.invokeStatic(rules.clj:159)
at etl.software.flowers.rules$apply_rules.invoke(rules.clj:152)
at etl.software.flowers.core$process_schedules.invokeStatic(core.clj:123)
at etl.software.flowers.core$process_schedules.invoke(core.clj:117)
at etl.software.ingest.io$run.invokeStatic(io.clj:79)
at etl.software.ingest.io$run.invoke(io.clj:73)
at etl.software.ingest.io$router.invokeStatic(io.clj:93)
at etl.software.ingest.io$router.invoke(io.clj:84)
at etl.software.ingest.io$lightweight_process$fn__19235$state_machine__6366__auto____19238$fn__19240.invoke(io.clj:158)
at etl.software.ingest.io$lightweight_process$fn__19235$state_machine__6366__auto____19238.invoke(io.clj:158)
at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
at etl.software.ingest.io$lightweight_process$fn__19235.invoke(io.clj:158)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
at clojure.lang.Numbers.ops(Numbers.java:1068)
at clojure.lang.Numbers.divide(Numbers.java:189)
at clojure.lang.Numbers.divide(Numbers.java:3881)
at etl.software.flowers.rules$eval53013$fn__53022$fn__53023.invoke(NO_SOURCE_FILE:55)
at clojure.core$update.invokeStatic(core.clj:6196)
at clojure.core$update.invoke(core.clj:6188)
at etl.software.flowers.rules$eval53013$fn__53022.invoke(NO_SOURCE_FILE:55)
at clara.rules.engine$fire_rules_STAR_$fn__10629$fn__10630.invoke(engine.cljc:1764)
... 25 common frames omitted
19-10-04 15:46:29 Tardis ERROR [etl.software.ingest.io:82] -
java.lang.Thread.run Thread.java: 748
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 624
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1149
...
etl.software.ingest.io/lightweight-process/fn io.clj: 158
clojure.core.async.impl.ioc-macros/run-state-machine-wrapped ioc_macros.clj: 977
clojure.core.async.impl.ioc-macros/run-state-machine ioc_macros.clj: 973
etl.software.ingest.io/lightweight-process/fn/state-machine--auto-- io.clj: 158
etl.software.ingest.io/lightweight-process/fn/state-machine--auto--/fn io.clj: 158
etl.software.ingest.io/router io.clj: 93
etl.software.ingest.io/run io.clj: 79
etl.software.flowers.core/process-schedules core.clj: 123
etl.software.flowers.rules/apply-rules rules.clj: 159
clara.rules/fire-rules rules.cljc: 44
clara.rules.engine.LocalSession/fire-rules engine.cljc: 1909
clara.rules.engine/fire-rules* engine.cljc: 1756
clara.rules.engine/fire-rules*/fn engine.cljc: 1763
clara.rules.engine/fire-rules*/fn/fn engine.cljc: 1764
etl.software.flowers.rules$eval53013$fn__53022.invoke NO_SOURCE_FILE: 55
clojure.core/update core.clj: 6196
etl.software.flowers.rules$eval53013$fn__53022$fn__53023.invoke NO_SOURCE_FILE: 55
...
java.lang.NullPointerException:
clojure.lang.ExceptionInfo: Exception in etl.software.flowers.rules/convert-to-barrels with bindings {:?event #etl.software.flowers.core.Event{:_id "733B772B94101049D1C8520FF8A9FB109CBC66BDB6A577E8A0E6311698837E12",
:created "2019-10-02T14:36:50Z",
:deleted false,
:rate 48763.0,
:reference "DFP0248029",
:start #object[java.time.ZonedDateTime 0x2a488968 "2019-10-04T01:03Z[UTC]"],
:stop #object[java.time.ZonedDateTime 0x6de70bdd "2019-10-04T23:00:49Z[UTC]"],
:uom :ga,
:volume 1071000}}
batched-logical-insertions: []
batched-rhs-retractions: []
batched-unconditional-insertions: []
bindings: {:?event
{:_id
"733B772B94101049D1C8520FF8A9FB109CBC66BDB6A577E8A0E6311698837E12",
:company-id
"B18D748B4B7A2CE3FD5C18543FA5DE639F5119D5480F0B9A06D8614B7985CBFF",
...}}
listeners: []
name: "etl.software.flowers.rules/convert-to-barrels"
rhs: (do
(as->
(get @evnts (:_id ?event))
_
(assoc _ :uom :barrels)
(update _ :volume (fn [v] (/ v 42)))
(update _ :rate (fn* [p1__18408#] (/ p1__18408# 42)))
(swap! evnts (fn* [p1__18409#] (assoc p1__18409# (:_id ?event) _)))))
@stLalo,
Its not recommended to do state-based manipulation in the LHS or RHS of rules.
Clara makes no guarantees on the number of times the LHS/RHS is executed, only that the steady state is factually correct, meaning that due to truth maintenance the RHS might execute many times. Additionally, the way that clara manages logical retractions would likely lead to less than ideal scenarios when using stateful actions in the RHS.
As to why the null pointer is thrown, do you have the state of the atom when the RHS is fired?
From the error it looks like that either:
evntsdoesn't contain the event id in question- the event within
evntsdoesn't contain a: volumeor:ratevalue.
@EthanEChristian
During the weekend, I explore more my situation and I figured out the problem. You said,
- evnts doesn't contain the event id in question
And this is correct.
As the ETL Software is a Parallel Synchronous multi-thread processor, the state was lost somewhere in between threads. I think my proper solution could be the following steps:
- For every rule, insert the Event into Clara rules in-memory database
- Query for the events that fitted the Clara rules
- Apply corresponding changes for each Clara rule type
- Repeat step 1 to 4 until there is no more Records to apply the rules too.
Please bare with me as I am refactoring legacy code and it is the first time encountering this problem.
@stLalo,
I agree with:
- For every rule, insert the Event into Clara rules in-memory database
- Query for the events that fitted the Clara rules
Without knowing the constraints of your codebase I can't say for sure, but I feel that all events could be processed in the same session for:
- Apply corresponding changes for each Clara rule type
- Repeat step 1 to 4 until there is no more Records to apply the rules too.
But regardless of 3&4, I think that the Insert & Query pattern should achieve what you need.
So what I have in mind would be something like
- Insert all my events in one session
(-> events
(mk-session 'some.namespace.rules)
(insert-all events)
(fire-rules))
- Matching rules will be inserted such as this
(defrule do-something-coll
[?event <- Event (= type :tulip)]
=>
(insert! (->Event ?event))
(defquery get-tulips
[]
(Event (= :tulip (:type ?event))))
- Apply changes to type = tulip
(defn thingamagiq
[Event]
(assoc Event :howdy "howdy"))
Repeat the same steps with other rules and queries.
I based my approach from this example
(defrule do-something-coll
[?event <- Event (= type :tulip)]
=>
(insert! (->Event ?event))
Rules that bind and insert the same fact type can be dangerous as they are prone to infinite looping.
For queries, you could use parameters to be more flexible in the event that you need other event types:
(defquery get-events-by-type
[:?type]
[?event <- Event (= ?type type)])
So to continue with this thread, I went back and did the following to my rules.
- Create a new Record with similar attributes from the main Event Record.
-
(mk-session 'some.namespace.rules) (insert-all events) (fire-rules)) -
(defrule do-something-coll [?event <- Event (= type :tulip)] => (insert! (map->NewEvent (assoc ?event :color "blue")) -
(defquery get-events [] [?newevents <- NewEvent])
This works wonderful for only one rule. However, I have cases where Event Records will be true for several Rules. Inserting! the true fact into NewEvent, will cause the creation of duplicates of the same events but with different values here and there.
I was thinking about using Retract! on the rules after the first one, but it seems not to return the previous NewEvent
@stLalo,
I think by modeling this slightly differently you could get away from the need to have explicit retractions. Again retractions can lead to unintended looping behavior.
From some of the things that you posted so far, it looks like you are trying to apply a series of changes to the original model. The way I was thinking would be something like this:
Facts:
(defrecord Event [id type color stem-length])
(defrecord FinalEvent [id type color stem-length])
(defrecord Correction [id field val])
Correction Rules:
(defrule correction-one
[?event <- Event (= type :tulip) (= ?id id)]
=>
(insert! (->Correction ?id :color "blue")))
(defrule correction-two
[?event <- Event (= type :tulip)
(= ?id id)
(> stem-length 15)]
=>
(insert! (->Correction ?id :stem-length 15)))
Consolidation:
(defrule final-event
[?event <- Event (= ?id id)]
[?corrections <- (acc/all) :from [Correction (= id ?id)]]
=>
(let [final-event (reduce (fn [final correction]
(assoc final (:field correction) (:val correction)))
?event
?corrections)]
(insert! (map->FinalEvent final-event))))
Query:
(defquery get-events
[]
[?final-event <- FinalEvent])
This side steps the duplicate event behavior by adding an accumulator and consolidation rule.
I suppose if the Session had enough scope it might include rules that wanted to correct the same field. That might require a hierarchy of what corrections need to be applied in what order.
The approach above could also be done where the correction fact would contain a fn instead of key/value pairs.