save! and as-text weirdess

[thorwil]

I already though I would have solved my issues with saving an entity with a Text property, but suddenly I couldn't reproduce my success, as if there are Gremlins in the system.

I started to simplify and experiment:

(ds/defentity Eek [body])

(defn save-article []
  (ds/save! (Eek. (ds/as-text "first")))
  (ds/save! (Eek. "second")))

Results in 2 entities with bodies: <Text: first>, second

  (ds/save! (Eek. (ds/as-text "first")))
  (ds/save! (Eek. (ds/as-text "second"))))

After clearing the datastore, results in no bodies

So what is it that I fail to understand, or what's going wrong?

[gcv]

Here's a trivial core.clj file where your example works as I expect:

(ns testme.core
  (:require [appengine-magic.core :as ae]
            [appengine-magic.services.datastore :as ds]))

(ds/defentity Eek [body])

(defn save-article-1 []
  (ds/save! (Eek. (ds/as-text "first")))
  (ds/save! (Eek. "second")))

(defn save-article-2 []
  (ds/save! (Eek. (ds/as-text "first")))
  (ds/save! (Eek. (ds/as-text "second"))))

(defn testme-app-handler [request]
  {:status 200
   :headers {"Content-Type" "text/plain"}
   :body "Hello, world!"})

(ae/def-appengine-app testme-app #'testme-app-handler)

I tested it using (ds/query :kind Eek) and it seems to return the right values. The /_ah/admin datastore console looks good, too.

Could you try again with a fresh project?

[thorwil]

With a fresh project and that code, (save-article-1) on the repl fails with:

ClassNotFoundException appengine_magic.services.datastore.EntityProtocol  java.net.URLClassLoader$1.run (URLClassLoader.java:217)

Pasting (ds/defentity Eek [body]) on the REPL doesn't help.

If I copy the defentity line to within the functions, I get the strange results as described before.

I updated Leiningen to 1.5 and tried both Clojure 1.2 and 1.3-alpha6, no luck.

Amalloy suggested that I replace the defentity with a defrecord to see if it's a classpath issue, but translating from your macro to a straight defrecord is beyond my current level of understanding.

[gcv]

Might be a Clojure version problem. appengine-magic 0.4.0 requires Clojure 1.2.1 (I'd expect 1.3-alpha6 to also work, but I haven't tested it). 1.2.1 is not completely compatible with 1.2.0 because of the changed handling of underscores and hyphens in namespace names. Could you paste in the exact project.clj from the fresh project with which you have the problem?

[thorwil]

Ah, I didn't pay enough attention to the versions and the pre-1.5 Leiningen I started out with put in clojure 1.2. Sorry about that.

So I turned

(defproject texst "1.0.0-SNAPSHOT"
  :description "FIXME: write"
  :dependencies [[org.clojure/clojure "1.2.0"]
                           [org.clojure/clojure-contrib "1.2.0"]]
  :dev-dependencies [[appengine-magic "0.4.1"]
                                  [swank-clojure "1.2.1"]])

into

(defproject texst "1.0.0-SNAPSHOT"
  :description "FIXME: write"
  :dependencies [[org.clojure/clojure "1.2.1"]]
  :dev-dependencies [[appengine-magic "0.4.1"]
                                  [swank-clojure "1.3.0"]])

The ClassNotFoundException is gone, but there's still a problem at least with the datastore viewer.

texst.core=> (save-article-1)
#:texst.core.Eek{:body "second"}
texst.core=> (save-article-2)
#:texst.core.Eek{:body #<Text <Text: second>>}
texst.core=> (ds/query :kind Eek)
(#:texst.core.Eek{:body #<Text <Text: first>>} #:texst.core.Eek{:body "second"} #:texst.core.Eek{:body #<Text <Text: first>>} #:texst.core.Eek{:body #<Text <Text: second>>})

After clearing the datastore:

(save-article-1)
texst.core=> (ds/query :kind Eek)
(#:texst.core.Eek{:body #<Text <Text: first>>} #:texst.core.Eek{:body "second"})

Up to here everything seems alright and the datastore viewer is of the same opinion.

After clearing again:

(save-article-2)
texst.core=> (ds/query :kind Eek)
(#:texst.core.Eek{:body #<Text <Text: first>>} #:texst.core.Eek{:body #<Text <Text: second>>})

Now the datastore viewer lists the entities without body, just Key and ID/Name. Until calling (save-article-1) again, then all 4 entries will be listed correctly.

Removing war/WEB-INF/appengine-generated/local_db.bin before a run doesn't help.

[gcv]

How do you clear the datastore? Just removing local_db.bin probably isn't enough. The App Engine environment keeps the development datastore in-memory, and just flushes it to disk once in a while. You have to stop the server (using ae/stop), then delete local_db.bin, then restart.

[thorwil]

Where I say clear, I just deleted all entries via the datastore viewer.

I removed local_db.bin with no server, not even a REPL running. Again, (save-article-2) does what it should according to ds/query, but the viewer claims there are no bodies. Apparently until one entry with a plain string body is created.

If I now get the same behavior with my real project, I can live with it, but it's sure very confusing.

[gcv]

I think I understand now, but just to be clear: does ds/query return the expected results under all circumstances?

[thorwil]

AFAICS, ds/query does return the expected result, always.

[gcv]

The development mode's datastore viewer leaves much to be desired. In any case, appengine-magic just makes it available from the SDK, so I don't control it. Since you found a reproducible bug, you might want to open a ticket with Google, although I suspect you'll have to provide a Java test case.

Try deploying your test project into the production App Engine and see if what happens there. The production datastore viewer works reasonably well, and I have not seen consistency problems with it.

[thorwil]

I've lost too much time with this already, so i will not write my first ever Java to perhaps create a bug report that won't be thrown out ;)

Please consider adding a warning about this to your Readme.

Thanks for your work and patience!