Bringing this thread back as I'm seeing this exception on a production
I have two Spark streaming apps reading the same topic. App1 has batch
interval 2secs and app2 has 60secs.
Both apps are running on the same cluster on similar hardware. I see this
exception only in app2 and fairly consistently.
Difference I see between the apps is
batch interval 2 secs
batch interval 60 secs
All other kafka/spark related configs are same for both apps.
spark.streaming.kafka.consumer.poll.ms = 4096
spark.streaming.backpressure.enabled = true
Not sure if pre-fetching or caching is messing things up.
16/10/19 14:32:04 WARN TaskSetManager: Lost task 2.0 in stage 1780.0 (TID
12541, ip-10-150-20-200.ec2.internal): java.lang.AssertionError: assertion
failed: Failed to get records for spark-executor-StreamingEventSplitProd
mt_event 6 49091480 after polling for 4096
On Wed, Sep 7, 2016 at 3:55 PM, Cody Koeninger <[EMAIL PROTECTED]> wrote: