Application is picked up, but the view is empty

[aynroot]

Hi.

I am running a Spark 2.1.0 job on Amazon EMR cluster (yarn) and trying to set up sparkling in live mode.

I am providing the following settings in spark-submit:

--deploy-mode cluster
--master yarn
...
--conf spark.eventLog.enabled=true 
--conf spark.extraListeners=com.groupon.sparklint.SparklintListener 
--packages com.groupon.sparklint:sparklint-spark210_2.11:1.0.5 

The issue is: I can see the sparklint UI on the default port "connected" to my running application, but the view is basically empty — no executors or tasks are visible.

Am I missing some configuration? What should I do for sparklint to pick up the statistics?

[roboxue]

Have you tried pressing the button (like >| and >>|) on the sidebar? Does that work?

Also please forward related log on your drive node if you see any ERROR or WARN that might come from sparklint

I don't have an easy way to run on a spark 2.1.0 app right now (since my company is still on 2.0.1). I need some time to setup a suitable environment.

[aynroot]

yes, clicking on one of those buttons gives me an error:

com.groupon.sparklint.events.ListenerEventSource cannot be cast to com.groupon.sparklint.events.FreeScrollEventSource

[aynroot]

and the logs related to sparkling are the following:

17/05/19 08:03:26 INFO Sparklint: Server started on 23763
17/05/19 08:03:26 INFO SparkContext: Registered listener com.groupon.sparklint.SparklintListener
...
17/05/19 08:03:33 ERROR SparklintBackend: Failure to navigate.
java.lang.ClassCastException: com.groupon.sparklint.events.ListenerEventSource cannot be cast to com.groupon.sparklint.events.FreeScrollEventSource
	at com.groupon.sparklint.events.GenericEventSourceGroupManager.getFreeScrollEventSource(GenericEventSourceGroupManager.scala:45)
	at com.groupon.sparklint.SparklintBackend.fwdApp(SparklintBackend.scala:115)
	at com.groupon.sparklint.SparklintBackend.com$groupon$sparklint$SparklintBackend$$fwdApp(SparklintBackend.scala:111)
	at com.groupon.sparklint.SparklintBackend$$anonfun$backendService$1$$anonfun$applyOrElse$5.apply(SparklintBackend.scala:61)
	at com.groupon.sparklint.SparklintBackend$$anonfun$backendService$1$$anonfun$applyOrElse$5.apply(SparklintBackend.scala:61)
	at scala.util.Try$.apply(Try.scala:192)
	at com.groupon.sparklint.SparklintBackend$$anonfun$backendService$1.applyOrElse(SparklintBackend.scala:61)
	at com.groupon.sparklint.SparklintBackend$$anonfun$backendService$1.applyOrElse(SparklintBackend.scala:55)
	at org.http4s.package$HttpService$$anonfun$apply$1.apply(package.scala:56)
	at org.http4s.package$HttpService$$anonfun$apply$1.apply(package.scala:56)
	at scala.Function1$$anonfun$andThen$1.apply(Function1.scala:52)
	at scalaz.Kleisli$$anonfun$flatMap$1.apply(Kleisli.scala:51)
	at org.http4s.server.Router$$anonfun$define$2$$anonfun$apply$1.apply(Router.scala:34)
	at org.http4s.server.Router$$anonfun$define$2$$anonfun$apply$1.apply(Router.scala:29)
	at org.http4s.server.Router$$anonfun$define$2$$anonfun$apply$1.apply(Router.scala:34)
	at org.http4s.server.Router$$anonfun$define$2$$anonfun$apply$1.apply(Router.scala:29)
	at com.groupon.sparklint.server.RoutingMap$$anonfun$router$1.applyOrElse(RoutingMap.scala:44)
	at com.groupon.sparklint.server.RoutingMap$$anonfun$router$1.applyOrElse(RoutingMap.scala:41)
	at org.http4s.package$HttpService$$anonfun$apply$1.apply(package.scala:56)
	at org.http4s.package$HttpService$$anonfun$apply$1.apply(package.scala:56)
	at scalaz.Kleisli$$anonfun$flatMap$1.apply(Kleisli.scala:51)
	at org.http4s.server.blaze.Http1ServerStage$$anonfun$runRequest$2.apply(Http1ServerStage.scala:110)
	at org.http4s.server.blaze.Http1ServerStage$$anonfun$runRequest$2.apply(Http1ServerStage.scala:110)
	at scalaz.concurrent.Task$.Try(Task.scala:403)
	at scalaz.concurrent.Task$$anonfun$apply$25.apply(Task.scala:300)
	at scalaz.concurrent.Task$$anonfun$apply$25.apply(Task.scala:300)
	at scalaz.concurrent.Future$$anonfun$apply$15$$anon$4.call(Future.scala:380)
	at scalaz.concurrent.Future$$anonfun$apply$15$$anon$4.call(Future.scala:380)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

(there are similar exceptions thrown when I click on one of the buttons you've mentioned)

[roboxue]

Think I fixed it in #58 after rounds of testing. 1.0.7 will be released very shortly that contains this fix.

[aynroot]

works with the 1.0.7 release, thank you!