Bringing this thread back as I'm seeing this exception on a production
kafka cluster.

I have two Spark streaming apps reading the same topic. App1 has batch
interval 2secs and app2 has 60secs.
Both apps are running on the same cluster on similar hardware. I see this
exception only in app2 and fairly consistently.

Difference I see between the apps is
      spark.streaming.kafka.maxRatePerPartition, 6000
      batch interval 2 secs
      spark.streaming.kafka.maxRatePerPartition, 10000
      batch interval 60 secs

All other kafka/spark related configs are same for both apps. = 4096
      spark.streaming.backpressure.enabled = true

Not sure if pre-fetching or caching is messing things up.

16/10/19 14:32:04 WARN TaskSetManager: Lost task 2.0 in stage 1780.0 (TID
12541, ip-10-150-20-200.ec2.internal): java.lang.AssertionError: assertion
failed: Failed to get records for spark-executor-StreamingEventSplitProd
mt_event 6 49091480 after polling for 4096
        at scala.Predef$.assert(Predef.scala:170)
        at org.apache.spark.streaming.kafka010.CachedKafkaConsumer.
        at org.apache.spark.streaming.kafka010.KafkaRDD$
        at org.apache.spark.streaming.kafka010.KafkaRDD$
        at scala.collection.Iterator$$anon$
        at scala.collection.Iterator$$anon$
        at scala.collection.Iterator$$anon$
On Wed, Sep 7, 2016 at 3:55 PM, Cody Koeninger <[EMAIL PROTECTED]> wrote: