Bringing this thread back as I'm seeing this exception on a production kafka cluster.
I have two Spark streaming apps reading the same topic. App1 has batch interval 2secs and app2 has 60secs. Both apps are running on the same cluster on similar hardware. I see this exception only in app2 and fairly consistently.
Difference I see between the apps is App1 spark.streaming.kafka.maxRatePerPartition, 6000 batch interval 2 secs App2 spark.streaming.kafka.maxRatePerPartition, 10000 batch interval 60 secs
All other kafka/spark related configs are same for both apps. spark.streaming.kafka.consumer.poll.ms = 4096 spark.streaming.backpressure.enabled = true
Not sure if pre-fetching or caching is messing things up.
16/10/19 14:32:04 WARN TaskSetManager: Lost task 2.0 in stage 1780.0 (TID 12541, ip-10-150-20-200.ec2.internal): java.lang.AssertionError: assertion failed: Failed to get records for spark-executor-StreamingEventSplitProd mt_event 6 49091480 after polling for 4096 at scala.Predef$.assert(Predef.scala:170) at org.apache.spark.streaming.kafka010.CachedKafkaConsumer. get(CachedKafkaConsumer.scala:74) at org.apache.spark.streaming.kafka010.KafkaRDD$ KafkaRDDIterator.next(KafkaRDD.scala:227) at org.apache.spark.streaming.kafka010.KafkaRDD$ KafkaRDDIterator.next(KafkaRDD.scala:193) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at scala.collection.Iterator$$anon$21.next(Iterator.scala:838) On Wed, Sep 7, 2016 at 3:55 PM, Cody Koeninger <[EMAIL PROTECTED]> wrote:
60 seconds for a batch is above the default settings in kafka related to heartbeat timeouts, so that might be related. Have you tried tweaking session.timeout.ms, heartbeat.interval.ms, or related configs?
On Wed, Oct 19, 2016 at 12:22 PM, Srikanth <[EMAIL PROTECTED]> wrote:
Kakfa 0.10.1 release separates poll() from heartbeat. So session.timeout.ms & max.poll.interval.ms can be set differently. I'll leave it to you on how to add this to docs! On Thu, Oct 20, 2016 at 1:41 PM, Cody Koeninger <[EMAIL PROTECTED]> wrote: