Some jobs remain in processing state forever

[sorabh89]

Hi Dibbhatt,

My application is running normally except that the UI shows some jobs remain in processing state forever.

Following is the screenshot of the same:

Regards,
Sorabh

[dibbhatt]

Hi,

I can see there are large scheduling delay of 8min or so. I guess all these jobs are still queued ? If you are using latest version of consumer , you can set props.put("consumer.backpressure.enabled", "true"); to see if that helps your scheduling delay and back-pressure issue .

Dibyendu

[dibbhatt]

hi @sorabh89 . this issue is still there once you use the consumer.backpressure.enabled flag ?

[dibbhatt]

Hi @sorabh89

I have created a JIRA to track the progress of contributing back this project to Apache Spark.

https://issues.apache.org/jira/browse/SPARK-11045

This project is now presently in spark-packages and I believe this is the correct time to contribute it to Apache Spark Project and give better options to larger community around Kafka Connectivity for Spark Streaming.

kindly Vote for this JIRA.