gcv/appengine-magic

repl mystery

mobileink opened this issue · 4 comments

mobileink commented

Here's something puzzling about running an ae-magic app in the repl. ae/def-appengine-app creates a map with keys :handler and :war-root. But if you precompile, the map seems to go away, leaving only the handler, which chokes ae/start.

$ lein clean
$ rm -rf war/WEB-INF/classes/ ;; start from scratch
$ lein repl
Compiling...
... ;; at this point everything is precompiled, no?
user=> (require '[appengine-magic.core :as ae])
nil
user=> sibtest.user/sibtest-user ;; my servlet; was compiled by lein repl
prepping sibtest-user servlet ;; this is from a println in servlet code - must be compiling?
CompilerException java.lang.RuntimeException: Unable to find static field: sibtest-user in class sibtest.user, compiling:(NO_SOURCE_PATH:1:606) ;; why not?
user=> (compile 'sibtest.user) ;; try to force it
sibtest.user ;; no recompile - lein compile knows src hasn't changed?
user=> sibtest.user/sibtest-user

'sibtest.user/user-handler ;;; BOINK! should be a map with :handler and :war-root keys

;; note that explicit compile made sibtest-user field available,
;; but what happened to the map created by ae/def-appengine-app?
user=> (ae/start sibtest.user/sibtest-user)
NullPointerException java.io.File. (File.java:389) ;; because no :war-root key

user=> (compile 'sibtest.user) ;; try again
sibtest.user ;; no "prepping" msg - lein compile knows src hasn't changed
user=> sibtest.user/sibtest-user

'sibtest.user/user-handler ;; no effect

;; touch or edit source
user=> (compile 'sibtest.user)
prepping sibtest-user servlet ;; this time it compiles
sibtest.user
user=> sibtest.user/sibtest-user
{:handler #<core$wrap_war_static$fn__1810 appengine_magic.core$wrap_war_static$fn__1810@8b7458e>, :war-root "/Users/gar/lein/sibtest/test/../war"} ;; bingo. this is what ae/start expects
user=> (ae/start sibtest.user/sibtest-user)
...
2013-03-29 06:02:23.609:INFO::Started SocketConnector@0.0.0.0:8080

<Server Server@31689ae1> ;; that's better

What baffles me is what happens to the map that ae/def-appengine-app? It appears that the only way to get it right is to force a recompile while in the repl. But my expectation is that if I precompile everything it should just work. Why does the runtime find only the handler of the app and not the map?

Thanks,

Gregg Reynolds

gcv commented

I'd need to stare at this problem more closely to fully understand it,
but in general, you should not manually call the compile form on
anything you use in interactive mode. .class files on the classpath
will cause some trouble for interactive mode (though as you
discovered, they can be overcome). I'm not sure if your example uses
your plugin, or what your plugin automatically compiles for you, or if
lein2 compiles things automatically. lein1 did not compile anything
without an explicit :aot directive in project.clj, and I recommend
avoiding it for interactive development. appengine-magic goes to great
lengths to avoid writing .class files until the very last moment of
packaging an app for deployment, and then it tries to clean them up
immediately after.

On Fri, Mar 29, 2013 at 4:15 AM, Gregg Reynolds
notifications@github.com wrote:

Here's something puzzling about running an ae-magic app in the repl. ae/def-appengine-app creates a map with keys :handler and :war-root. But if you precompile, the map seems to go away, leaving only the handler, which chokes ae/start.

$ lein clean
$ rm -rf war/WEB-INF/classes/ ;; start from scratch
$ lein repl
Compiling...
... ;; at this point everything is precompiled, no?
user=> (require '[appengine-magic.core :as ae])
nil
user=> sibtest.user/sibtest-user ;; my servlet; was compiled by lein repl
prepping sibtest-user servlet ;; this is from a println in servlet code - must be compiling?
CompilerException java.lang.RuntimeException: Unable to find static field: sibtest-user in class sibtest.user, compiling:(NO_SOURCE_PATH:1:606) ;; why not?
user=> (compile 'sibtest.user) ;; try to force it
sibtest.user ;; no recompile - lein compile knows src hasn't changed?
user=> sibtest.user/sibtest-user
#'sibtest.user/user-handler ;;; BOINK! should be a map with :handler and :war-root keys
;; note that explicit compile made sibtest-user field available,
;; but what happened to the map created by ae/def-appengine-app?
user=> (ae/start sibtest.user/sibtest-user)
NullPointerException java.io.File. (File.java:389) ;; because no :war-root key

user=> (compile 'sibtest.user) ;; try again
sibtest.user ;; no "prepping" msg - lein compile knows src hasn't changed
user=> sibtest.user/sibtest-user
#'sibtest.user/user-handler ;; no effect
;; touch or edit source
user=> (compile 'sibtest.user)
prepping sibtest-user servlet ;; this time it compiles
sibtest.user
user=> sibtest.user/sibtest-user
{:handler #, :war-root "/Users/gar/lein/sibtest/test/../war"} ;; bingo. this is what ae/start expects
user=> (ae/start sibtest.user/sibtest-user)
...
2013-03-29 06:02:23.609:INFO::Started SocketConnector@0.0.0.0:8080

;; that's better

What baffles me is what happens to the map that ae/def-appengine-app? It appears that the only way to get it right is to force a recompile while in the repl. But my expectation is that if I precompile everything it should just work. Why does the runtime find only the handler of the app and not the map?

Thanks,

Gregg Reynolds

—
Reply to this email directly or view it on GitHub.

mobileink commented

On Mar 29, 2013, at 10:12 AM, Constantine Vetoshev notifications@github.com wrote:

I'd need to stare at this problem more closely to fully understand it,
but in general, you should not manually call the compile form on
anything you use in interactive mode. .class files on the classpath
will cause some trouble for interactive mode (though as you
discovered, they can be overcome). I'm not sure if your example uses
your plugin, or what your plugin automatically compiles for you, or if
lein2 compiles things automatically. lein1 did not compile anything
without an explicit :aot directive in project.clj, and I recommend
avoiding it for interactive development. appengine-magic goes to great
lengths to avoid writing .class files until the very last moment of
packaging an app for deployment, and then it tries to clean them up
immediately after.

Hi Constantine,

Thanks for the quick response. I didn't realize that about compiling (still learning clojure), so I switched to load-file, and it works like a charm. I spent some time trying to get DevAppServerMain to run on the repl by using a shell script to source the lein script, set jvm params etc. (mimicking Kickstart) but I can't get past a security "access denied" problem. However, it turns out to be easy to test multiple servlets using the repl - just write a little function to reload code and do ae/serve. See the :repo-options key in the project.clj you get from running lein new gaem (https://github.com/greynolds/gaem).

Eventually I'd like to get DevAppServerMain running in a repo, but doing rapid development with the current repl followed by system testing with dev_appserver is just about as good, so I think I'll get back to my application.

Thanks!

-Gregg

mobileink commented

Ok, I see what's happening. core.clj checks to see what the execution env is, then does:

(if (= :interactive (appengine-environment-type))
    (load "core_local")
    (load "core_google"))

each of which has its own idea of what def-appengine-app means. Since one of the env types is :compiling, it looks like compiling at the repl cli causes ae to load core_google instead of core_local. The former makes def-appengine-app just return the handler instead of a map.

Loading different code based on exec env seems dubious but I can see it was needed to make the :interactive env work. One way to get rid of this might be to just move the war-root lookup code from core_local.clj to the server startup routine. There doesn't seem to be any functional reason why the server shouldn't be responsible for finding the war root.

-Gregg

gcv commented

The production App Engine on Google's servers runs in a sandboxed environment. This sandbox has heavy-handed classloader restrictions. These restrictions make filesystem access and introspection impossible. In most cases, just loading up a class which references a restricted class makes the application crash. The appengine-magic interactive environment does a number of things which violate production App Engine restrictions; this includes the war-root lookup code. It is therefore required that everything which enables interactive mode is never compiled into the deployed application. Hence the loading of different code based on execution environment.