NPE with nested sequences

Question

NPE with nested sequences

jcf opened this issue 10 years ago · 13 comments

Hi Steve,

I think Herbert could come in really handy for validating some big Datomic transaction maps in a couple of my tests, and have setup a test namespace that uses schemas to make it easy to test types rather than values where I don't have deterministic output.

I have encountered an NPE from miner.herbert.private/mkprb where pred, which is nil, is called.

I can reproduce the NPE with the following schema and data. I have noticed the data doesn't match (order's wrong) but don't imaging that's the issue. Perhaps I'm using pred incorrectly?

(def uuid? (partial instance? java.util.UUID))

(def schema
  [[:create-resource
    [:resource/id '(pred uuid?)]
    {:occurrence/place 281474976711832,
     :occurrence/start-time #inst "1970-01-01T00:00:00.001-00:00"}]
   [:create-resource
    [:resource/id '(pred uuid?)]
    {:occurrence/place 281474976711832,
     :occurrence/start-time #inst "1970-01-01T00:00:00.002-00:00"}]
   [:create-resource
    [:resource/id '(pred uuid?)]
    {:occurrence/place 281474976711832,
     :occurrence/start-time #inst "1970-01-01T00:00:00.003-00:00"}]])

(def data
  [[:create-resource
    [:resource/id #uuid "552fc7b3-8905-46fd-b50b-dd613c940504"]
    {:occurrence/place 281474976711832,
     :occurrence/start-time #inst "1970-01-01T00:00:00.003-00:00"}]
   [:create-resource
    [:resource/id #uuid "552fc7b3-5ffa-4eb8-b3ac-8e3b15bcead0"]
    {:occurrence/place 281474976711832,
     :occurrence/start-time #inst "1970-01-01T00:00:00.002-00:00"}]
   [:create-resource
    [:resource/id #uuid "552fc7b3-af79-489c-8eaf-09e3845106d0"]
    {:occurrence/place 281474976711832,
     :occurrence/start-time #inst "1970-01-01T00:00:00.001-00:00"}]])

(miner.herbert/conforms? schema data)
;; => NPE

(pst)
NullPointerException 
    clojure.core/apply (core.clj:626)
    clojure.core/partial/fn--4228 (core.clj:2468)
    miner.herbert.private/mkprb/fn--70585 (private.clj:93)
    miner.herbert.private/mk-key/fn--70664 (private.clj:357)
    miner.herbert.private/mkmap/fn--70659/fn--70660 (private.clj:333)
    clojure.core.protocols/fn--6086 (protocols.clj:143)
    clojure.core.protocols/fn--6057/G--6052--6066 (protocols.clj:19)
    clojure.core.protocols/seq-reduce (protocols.clj:31)
    clojure.core.protocols/fn--6078 (protocols.clj:54)
    clojure.core.protocols/fn--6031/G--6026--6044 (protocols.clj:13)
    clojure.core/reduce (core.clj:6289)
    miner.herbert.private/mkmap/fn--70659 (private.clj:330)

Full trace here: https://gist.github.com/jcf/2c4daefc55b45418b915

Any help would be much appreciated. I'll keep digging into the problem in case PEBCAK.

Kind regards,

James

Answer 1 · 2015-04-16T14:47:36.000Z

I've tried replacing (pred uuid?) with any, and I've also fixed the order so my data theoretically matches my schema. I'm still getting an NPE.

Answer 2 · 2015-04-16T14:58:29.000Z

This is the simplest case I've found:

(let [d (java.util.Date.)] (miner.herbert/conforms? {:d d} {:d d}))

I'd expect this to work, because I might want to check for both the type and value of an instant in a blob of EDN, but maybe I'm missing something?

Answer 3 · 2015-04-16T15:16:35.000Z

Yep, definitely was just me being silly. I ended up solving the problem through use of tag like so:

(import '[java.text SimpleDateFormat])
(import '[java.util TimeZone])

(defn t [n]
  (java.util.Date. n))

(defn ts [t]
  (.format
   (doto (SimpleDateFormat. "yyyy-MM-dd'T'HH:mm:ss.SSS-00:00")
     (.setTimeZone (TimeZone/getTimeZone "GMT")))
   t))

(let [d (java.util.Date.)]
  (miner.herbert/conforms? {:d '(tag inst (-> 3 t ts))} {:d d}))

Hopefully this will help someone else in the not too distant future. 😄

Sorry for the noise, and thanks for Herbert!

Answer 4 · 2015-04-16T15:19:18.000Z

I'm glad you found a work-around, but I think the original problem is a bug in Herbert. Dates and other non-Clojure values are not being treated as "literals" (really I should call them constants). I'll have to fix that.

Answer 5 · 2015-04-16T15:20:42.000Z

Hi Steve,

Thanks for getting back to me. Unfortunately, my workaround doesn't work now that I've restarted my REPL and pasted the above snippet.

Do you have any suggestions on how I might be able to pass a constant into a schema and get the desired behaviour of testing for equality?

Answer 6 · 2015-04-16T15:21:46.000Z

I will investigate and see if I can come up with a quick fix.

Answer 7 · 2015-04-16T16:15:33.000Z

I believe the root of the problem is that a Java Date is not an EDN constant. I think I can fix that but I need to do a bit of work. The issue for the Herbert project is that Herbert schemas are supposed to be EDN expressions and I'm not sure I want to allow non-EDN values in the schemas. I'll have to think about that. Certainly, I should do better with detecting and reporting the error.

If your dates don't have to be exact, you could use an expression like (tag inst) to match any date. However, I assume you want exact matches of specific dates. In that case, I suggest you define constants and predicates for those dates.

(def d1 #inst "1970-01-01T00:00:00.001-00:00")
(def d1? (partial = d1))

Then you can use Herbert expressions like '(pred d1?) in your schema. Perhaps = is too exact but that's a separate issue.

By the way, I noticed that your original schema and data had the date values in a different order so I had to rearrange things slightly to get a successful match.

Answer 8 · 2015-04-16T16:20:30.000Z

I should clarify that the #inst "1970" expression get translated into a Java date by the Clojure reader so Herbert only sees the date value, not the original #inst expression. I'm thinking I should extend the tag notation to handle this case. You should be able to use a Herbert term like (tag inst "1970") and have it match an exact date.

Answer 9 · 2015-04-16T16:30:57.000Z

If your dates don't have to be exact, you could use an expression like (tag inst) to match any date.

Unfortunately, I want to test the value as well as type to ensure I'm getting the right data in my transactions.

You should be able to use a Herbert term like (tag inst "1970") and have it match an exact date.

This is exactly what I've been trying to do, but with no luck. I'd expect the following to work, but I don't get a match.

(java.util.Date. 0)
;; => #inst "1970-01-01T00:00:00.000-00:00"

(let [d (java.util.Date. 0)]
  (miner.herbert/conforms? '(tag inst "1970-01-01T00:00:00.000-00:00") d))

Thanks for your help, Steve. I really appreciate it.

Edit Oh, wait. Does this mean it's not possible to do what I'm attempting to?

You should be able to use a Herbert term like (tag inst "1970") and have it match an exact date.

As in "you should be able to" but can't as is.

Answer 10 · 2015-04-16T19:55:45.000Z

Added support for constant literals so '(tag inst "1970") will work as expected. The new release is version 0.6.9. Used a slightly modified version of the original bug report as a new test. Let me if the new version works for you.

Answer 11 · 2015-04-16T21:07:05.000Z

Decided to change it a bit in release 0.6.10.

Answer 12 · 2015-04-17T09:46:59.000Z

I'm using 0.6.10 now, and can confirm literals work as expected with tag. It'd be fantastic if I could use a function or macro to build the pattern match/literal used inside tag in order to do something like this:

(import '[java.text SimpleDateFormat])
(import '[java.util TimeZone])

(defn t [n]
  (java.util.Date. n))

(defn t->inst-str [t]
  (.format
   (doto (SimpleDateFormat. "yyyy-MM-dd'T'HH:mm:ss.SSS-00:00")
     (.setTimeZone (TimeZone/getTimeZone "GMT")))
   t))

;; This won't currently work. You'll get an NPE.
(miner.herbert/conforms? '(tag inst (-> 1 t t->inst-str)) (java.util.Date. 1))
;; => NPE

Once again, thank you for helping me resolve this issue. I hope you have a great weekend, Steve!

Answer 13 · 2015-04-17T14:23:12.000Z

Glad to hear it works for you. I hesitate to allow code evaluation within the Herbert patterns. I've done a bit of that, and it leads to complications, especially with free references (or bindings within the pattern). I think it's better to use some sort of template code external to Herbert to create the constant pattern that you want.

It's a little bit ugly to use Clojure backquote in your example because you need to preserve the un-namespaced symbols that Herbert currently requires. I'll have to think about that. Anyway, this should work:

(miner.herbert/conforms? `(~'tag ~'inst ~(-> 1 t t->inst-str)) (java.util.Date. 1))

And you could probably write a macro that would make approach simpler to use. Take a look at
https://github.com/brandonbloom/backtick . I haven't tried it, but I think his template-fn might work to avoid unwanted auto-namespacing.