Hi,

Unfortunately we already did reset our offsets in order to continue our processing, so I'm unable to reproduce this again right now.

Here is log output with Debug level for KafkaSystemConsumer. You'll notice there is no stacktrace even in the debug line that should have it. Just an echo of the same message about the OffsetOutOfRange exception.

Thanks,
Thunder
2018-07-06 18:57:01 logback 5545 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 7598450 for SystemStreamPartition [kafka, my-topic-backfill, 3]
2018-07-06 18:57:01 logback 5545 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 671804914 for SystemStreamPartition [kafka, my-topic-new, 3]
2018-07-06 18:57:01 logback 5545 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 663279798 for SystemStreamPartition [kafka, my-topic-new, 9]
2018-07-06 18:57:01 logback 5545 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 7598982 for SystemStreamPartition [kafka, my-topic-backfill, 9]
2018-07-06 18:57:01 logback 5545 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 645599258 for SystemStreamPartition [kafka, my-topic-new, 1]
2018-07-06 18:57:01 logback 5545 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 7595444 for SystemStreamPartition [kafka, my-topic-backfill, 1]
2018-07-06 18:57:01 logback 5545 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 7596127 for SystemStreamPartition [kafka, my-topic-backfill, 7]
2018-07-06 18:57:01 logback 5546 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 671082000 for SystemStreamPartition [kafka, my-topic-new, 7]
2018-07-06 18:57:01 logback 5546 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 7596438 for SystemStreamPartition [kafka, my-topic-backfill, 5]
2018-07-06 18:57:01 logback 5546 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 675851134 for SystemStreamPartition [kafka, my-topic-new, 5]
2018-07-06 18:57:01 logback 5546 [main] INFO  o.a.samza.checkpoint.OffsetManager - Checkpointed offset is currently 7595266 for SystemStreamPartition [kafka, my-topic-backfill, 11]
2018-07-06 18:57:01 logback 5559 [main] INFO  o.a.samza.checkpoint.OffsetManager - Successfully loaded last processed offsets: {Partition 1={SystemStreamPartition [kafka, my-topic-new, 1]=645599258, SystemStreamPartition [kafka, my-topic-backfill, 1]=7595444}, Partition 3={SystemStreamPartition [kafka, my-topic-new, 3]=671804914, SystemStreamPartition [kafka, my-topic-backfill, 3]=7598450}, Partition 5={SystemStreamPartition [kafka, my-topic-backfill, 5]=7596438, SystemStreamPartition [kafka, my-topic-new, 5]=675851134}, Partition 7={SystemStreamPartition [kafka, my-topic-backfill, 7]=7596127, SystemStreamPartition [kafka, my-topic-new, 7]=671082000}, Partition 9={SystemStreamPartition [kafka, my-topic-new, 9]=663279798, SystemStreamPartition [kafka, my-topic-backfill, 9]=7598982}, Partition 11={SystemStreamPartition [kafka, my-topic-backfill, 11]=7595266}}
2018-07-06 18:57:01 logback 5561 [main] INFO  o.a.samza.checkpoint.OffsetManager - Successfully loaded starting offsets: Map(Partition 9 -> Map(SystemStreamPartition [kafka, my-topic-backfill, 9] -> 7598983, SystemStreamPartition [kafka, my-topic-new, 9] -> 663279799), Partition 3 -> Map(SystemStreamPartition [kafka, my-topic-backfill, 3] -> 7598451, SystemStreamPartition [kafka, my-topic-new, 3] -> 671804915), Partition 5 -> Map(SystemStreamPartition [kafka, my-topic-backfill, 5] -> 7596439, SystemStreamPartition [kafka, my-topic-new, 5] -> 675851135), Partition 7 -> Map(SystemStreamPartition [kafka, my-topic-backfill, 7] -> 7596128, SystemStreamPartition [kafka, my-topic-new, 7] -> 671082001), Partition 1 -> Map(SystemStreamPartition [kafka, my-topic-backfill, 1] -> 7595445, SystemStreamPartition [kafka, my-topic-new, 1] -> 645599259), Partition 11 -> Map(SystemStreamPartition [kafka, my-topic-backfill, 11] -> 7595267))
2018-07-06 18:57:01 logback 5562 [main] INFO  o.a.samza.container.SamzaContainer - Starting stores in task instance Partition 3
2018-07-06 18:57:01 logback 5566 [main] INFO  o.a.s.storage.TaskStorageManager - Validating change log streams: Map()
2018-07-06 18:57:01 logback 5568 [main] INFO  o.a.s.storage.TaskStorageManager - Got change log stream metadata: Map()
2018-07-06 18:57:01 logback 5571 [main] INFO  o.a.s.storage.TaskStorageManager - Assigning oldest change log offsets for taskName Partition 3: Map()
2018-07-06 18:57:01 logback 5577 [main] INFO  o.a.samza.container.SamzaContainer - Starting stores in task instance Partition 9
2018-07-06 18:57:01 logback 5577 [main] INFO  o.a.s.storage.TaskStorageManager - Validating change log streams: Map()
2018-07-06 18:57:01 logback 5578 [main] INFO  o.a.s.storage.TaskStorageManager - Got change log stream metadata: Map()
2018-07-06 18:57:01 logback 5578 [main] INFO  o.a.s.storage.TaskStorageManager - Assigning oldest change log offsets for taskName Partition 9: Map()
2018-07-06 18:57:01 logback 5578 [main] INFO  o.a.samza.container.SamzaContainer - Starting stores in task instance Partition 1
2018-07-06 18:57:01 logback 5578 [main] INFO  o.a.s.storage.TaskStorageManager - Validating change log streams: Map()
2018-07-06 18:57:01 logback 5578 [main] INFO  o.a.s.storage.TaskStorageManager - Got change log stream metadata: Map()
2018-07-06 18:57:01 logback 5578 [main] INFO  o.a.s.storage.TaskStorageManager - Assigning oldest change log offsets for taskName Partition 1: Map()
2018-07-06 18:57:01 logback 5578 [main] INFO  o.a.samza.container.SamzaContainer - Starting stores in task instance Partition 7
2018-07-06 18:57:01 logback 5578 [main] INFO  o.a.s.storage.TaskStorageManager - Validating change log streams: Map()
2018-07-06 18:57:01 logback 5579 [main] INFO  o.a.s.storage.TaskStor
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB