nathanmarz/cascalog

IllegalArgumentException No implementation of method: :to-sink of protocol: #'cascalog.cascading.types/ISink found for class: cascalog.cascading.tap.CascalogTap

sorenmacbeth opened this issue · 19 comments

sorenmacbeth commented

I get the following exception trying to use a (:trap) with a cascalog-tap on a cluster

IllegalArgumentException No implementation of method: :to-sink of protocol: #'cascalog.cascading.types/ISink found for class: cascalog.cascading.tap.CascalogTap
clojure.core/-cache-protocol-fn (core_deftype.clj:541)
cascalog.cascading.types/eval9074/fn--9075/G--9065--9080 (types.clj:84)
cascalog.logic.platform/init-trap-map (platform.clj:20)
cascalog.logic.platform.CascadingPlatform/fn--9574 (platform.clj:30)
clojure.core/apply (core.clj:619)
clojure.core/update-in (core.clj:5587)
cascalog.logic.platform.CascadingPlatform (platform.clj:29)
cascalog.logic.predicate/generator-node (predicate.clj:150)
cascalog.logic.predicate/build-predicate (predicate.clj:224)
clojure.core/apply (core.clj:619)
clojure.core/partial/fn--445 (core.clj:2396)
clojure.core/map/fn--466 (core.clj:2485)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:60)
clojure.lang.RT.seq (RT.java:484)
clojure.core/seq (core.clj:133)
clojure.core.protocols/seq-reduce (protocols.clj:30)
clojure.core.protocols/eval2802/fn--2803 (protocols.clj:54)
clojure.core.protocols/eval2735/fn--2736/G--2726--2749 (protocols.clj:13)
clojure.core/reduce (core.clj:6177)
clojure.core/group-by (core.clj:6492)
cascalog.logic.parse/build-rule (parse.clj:637)
cascalog.logic.parse/parse-subquery (parse.clj:687)
ybot.analytics.donbot.jobs.language-model/eval13864 (NO_SOURCE_FILE:1)
clojure.lang.Compiler.eval (Compiler.java:6619)
clojure.lang.Compiler.eval (Compiler.java:6582)
clojure.core/eval (core.clj:2852)
clojure.main/repl/read-eval-print--6252/fn--6255 (main.clj:259)
clojure.main/repl/read-eval-print--6252 (main.clj:259)
clojure.main/repl/fn--6261 (main.clj:277)
clojure.main/repl (main.clj:277)
clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn--6321 (interruptible_eval.clj:56)
clojure.core/apply (core.clj:617)
clojure.core/with-bindings* (core.clj:1788)
clojure.tools.nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:41)
clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn--6362/fn--6365 (interruptible_eval.clj:171)
clojure.core/comp/fn--409 (core.clj:2330)
clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn--6355 (interruptible_eval.clj:138)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
java.lang.Thread.run (Thread.java:724)
nil

sorenmacbeth commented

@sritchie It seems that this is being treated as a class and not an IRecord?

sritchie commented

Oh, I think I know what's happening. Hmm. When Cascalog reboots a function on the remote system, it loads up a fresh JVM and then requires the namespace of the function's symbol; so, we'd get this error if the submitting process hadn't required all of the right files, and bootstrapped the protocol properly.

The defrecord actually does define a new class, just one that extends IRecord (and a bunch of other java interfaces).

sorenmacbeth commented

Ah, yeah. That makes sense.

sorenmacbeth commented

@sritchie Do you have ideas or pointer of where I should start looking? Happy to do a PR if you can help point me in the right direction.

sritchie commented

yeah, we basically need to see if the extend-protocol lines are getting called in those files; one way would be to throw an exception above the definition; hacky, but that would work.

This one's really strange (since everything is working locally!)

sorenmacbeth commented

hmm, that doesn't seem to work, I can't get it to build since it throw the exception right away

sritchie commented

Okay, I can give this a hack potentially next week... lots of travel
coming up :-/

I think the next debug step is to open a repl on the jobtracker and see
if you can get this thing to work at ALL on the jobtracker. If it does,
then it's something about the way the main method is getting instantiated.

What if you explicitly stick a (use 'cascalog.api) in before whatever's
calling this line?

ybot.analytics.donbot.jobs.language-model/eval13864 (NO_SOURCE_FILE:1)

Soren Macbeth mailto:notifications@github.com
September 26, 2013 7:15 PM

hmm, that doesn't seem to work, I can't get it to build since it throw
the exception right away

—
Reply to this email directly or view it on GitHub
#190 (comment).

Soren Macbeth mailto:notifications@github.com
September 23, 2013 2:18 PM

I get the following exception trying to use a (:trap) with a
cascalog-tap on a cluster

IllegalArgumentException No implementation of method: :to-sink of
protocol: #'cascalog.cascading.types/ISink found for class:
cascalog.cascading.tap.CascalogTap
clojure.core/-cache-protocol-fn (core_deftype.clj:541)
cascalog.cascading.types/eval9074/fn--9075/G--9065--9080 (types.clj:84)
cascalog.logic.platform/init-trap-map (platform.clj:20)
cascalog.logic.platform.CascadingPlatform/fn--9574 (platform.clj:30)
clojure.core/apply (core.clj:619)
clojure.core/update-in (core.clj:5587)
cascalog.logic.platform.CascadingPlatform (platform.clj:29)
cascalog.logic.predicate/generator-node (predicate.clj:150)
cascalog.logic.predicate/build-predicate (predicate.clj:224)
clojure.core/apply (core.clj:619)
clojure.core/partial/fn--445 (core.clj:2396)
clojure.core/map/fn--466 (core.clj:2485)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:60)
clojure.lang.RT.seq (RT.java:484)
clojure.core/seq (core.clj:133)
clojure.core.protocols/seq-reduce (protocols.clj:30)
clojure.core.protocols/eval2802/fn--2803 (protocols.clj:54)
clojure.core.protocols/eval2735/fn--2736/G--2726--2749 (protocols.clj:13)
clojure.core/reduce (core.clj:6177)
clojure.core/group-by (core.clj:6492)
cascalog.logic.parse/build-rule (parse.clj:637)
cascalog.logic.parse/parse-subquery (parse.clj:687)
ybot.analytics.donbot.jobs.language-model/eval13864 (NO_SOURCE_FILE:1)
clojure.lang.Compiler.eval (Compiler.java:6619)
clojure.lang.Compiler.eval (Compiler.java:6582)
clojure.core/eval (core.clj:2852)
clojure.main/repl/read-eval-print--6252/fn--6255 (main.clj:259)
clojure.main/repl/read-eval-print--6252 (main.clj:259)
clojure.main/repl/fn--6261 (main.clj:277)
clojure.main/repl (main.clj:277)
clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn--6321
(interruptible_eval.clj:56)
clojure.core/apply (core.clj:617)
clojure.core/with-bindings* (core.clj:1788)
clojure.tools.nrepl.middleware.interruptible-eval/evaluate
(interruptible_eval.clj:41)
clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn--6362/fn--6365
(interruptible_eval.clj:171)
clojure.core/comp/fn--409 (core.clj:2330)
clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn--6355
(interruptible_eval.clj:138)
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:615)
java.lang.Thread.run (Thread.java:724)
nil

—
Reply to this email directly or view it on GitHub
#190.

Sam Ritchie, Twitter Inc
703.662.1337
@sritchie

sorenmacbeth commented

I already did that. Nothing at all works, not even basic playground queries work in distributed hadoop mode.

sritchie commented

Okay, let's try to get on IRC tomorrow. Failing when you run "hadoop jar ... your-namespace", or when you open a repl at the jobtracker? Let's check it out tomorrow.

Quantisan commented

Looks like this issue is related to #193. This post on Clojure mailing list 6 months ago describes the problem. I tried :aot but didn't work. Refactoring defrecord constructors seem promising in my trials so far.

sritchie commented

I wonder if we could move the protocols into their own file and AOT
compile only that file? I think that might get us past this.

Paul Lam mailto:notifications@github.com
October 9, 2013 12:16 PM

Looks like this issue is related to #193
#193. This post on
Clojure mailing list
https://groups.google.com/d/msg/clojure/ZeENLkHAQTU/KDdDAmEjBqoJ 6
months ago describes the problem. I tried :aot but didn't work.
Refactoring defrecord constructors seem promising in my trials so far.

—
Reply to this email directly or view it on GitHub
#190 (comment).

Soren Macbeth mailto:notifications@github.com
September 23, 2013 11:18 AM

I get the following exception trying to use a (:trap) with a
cascalog-tap on a cluster

IllegalArgumentException No implementation of method: :to-sink of
protocol: #'cascalog.cascading.types/ISink found for class:
cascalog.cascading.tap.CascalogTap
clojure.core/-cache-protocol-fn (core_deftype.clj:541)
cascalog.cascading.types/eval9074/fn--9075/G--9065--9080 (types.clj:84)
cascalog.logic.platform/init-trap-map (platform.clj:20)
cascalog.logic.platform.CascadingPlatform/fn--9574 (platform.clj:30)
clojure.core/apply (core.clj:619)
clojure.core/update-in (core.clj:5587)
cascalog.logic.platform.CascadingPlatform (platform.clj:29)
cascalog.logic.predicate/generator-node (predicate.clj:150)
cascalog.logic.predicate/build-predicate (predicate.clj:224)
clojure.core/apply (core.clj:619)
clojure.core/partial/fn--445 (core.clj:2396)
clojure.core/map/fn--466 (core.clj:2485)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:60)
clojure.lang.RT.seq (RT.java:484)
clojure.core/seq (core.clj:133)
clojure.core.protocols/seq-reduce (protocols.clj:30)
clojure.core.protocols/eval2802/fn--2803 (protocols.clj:54)
clojure.core.protocols/eval2735/fn--2736/G--2726--2749 (protocols.clj:13)
clojure.core/reduce (core.clj:6177)
clojure.core/group-by (core.clj:6492)
cascalog.logic.parse/build-rule (parse.clj:637)
cascalog.logic.parse/parse-subquery (parse.clj:687)
ybot.analytics.donbot.jobs.language-model/eval13864 (NO_SOURCE_FILE:1)
clojure.lang.Compiler.eval (Compiler.java:6619)
clojure.lang.Compiler.eval (Compiler.java:6582)
clojure.core/eval (core.clj:2852)
clojure.main/repl/read-eval-print--6252/fn--6255 (main.clj:259)
clojure.main/repl/read-eval-print--6252 (main.clj:259)
clojure.main/repl/fn--6261 (main.clj:277)
clojure.main/repl (main.clj:277)
clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn--6321
(interruptible_eval.clj:56)
clojure.core/apply (core.clj:617)
clojure.core/with-bindings* (core.clj:1788)
clojure.tools.nrepl.middleware.interruptible-eval/evaluate
(interruptible_eval.clj:41)
clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn--6362/fn--6365
(interruptible_eval.clj:171)
clojure.core/comp/fn--409 (core.clj:2330)
clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn--6355
(interruptible_eval.clj:138)
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:615)
java.lang.Thread.run (Thread.java:724)
nil

—
Reply to this email directly or view it on GitHub
#190.

Sam Ritchie, Twitter Inc
703.662.1337
@sritchie

Quantisan commented

I tried :aot specific namespaces and also :aot :all already

Quantisan commented

let's try this again...I just pushed a patch that fixed #193, do you guys want to try it with this issue too?

sorenmacbeth commented

103d206 doesn't fix this one 😿

sorenmacbeth commented

WAIT! I was running on the wrong uberjar! I does seem to fix it 👍

sritchie commented

EL JOKESTERINO IN THE HOUSE! Yes, @Quantisan is a saint.

Now I need to read up on that Clojure thread to understand why the hell this worked.

Quantisan commented

you owe me 20 minutes of misery @sorenmacbeth! :)

sorenmacbeth commented

YESSSSSSSSSS fixed via 103d206

sorenmacbeth commented

@Quantisan hahahaha, I do I do. Thanks for getting this in Paul!