Hi all,
We've just run into a strange problem with samza 0.14.1. We had a job down for a bit, while kafka cleaned past our saved offsets. When starting the job now, we get repeated org.apache.kafka.common.errors.OffsetOutOfRangeException. And it just retries over and over again. We HAVE set

systems.kafka.consumer.auto.offset.reset=smallest as well. Has anyone else seen this? Our understanding from the documentation is that this setting says what to do if the offset is out of range.

systems.system-name.consumer.auto.offset.reset : This setting determines what happens if a consumer attempts to read an offset that is outside of the current valid range. This could happen if the topic does not exist, or if a checkpoint is older than the maximum message history retained by the brokers.

This is the set of messages that keeps repeating:

2018-07-06 18:32:15 INFO  kafka.utils.VerifiableProperties - Verifying properties

2018-07-06 18:32:15 INFO  kafka.utils.VerifiableProperties - Property client.id is overridden to samza_consumer-stg_apollo_crawler_stream_task-1

2018-07-06 18:32:15 INFO  kafka.utils.VerifiableProperties - Property metadata.broker.list is overridden to kafka-server.ntent.com:9092

2018-07-06 18:32:15 INFO  kafka.utils.VerifiableProperties - Property request.timeout.ms is overridden to 30000

2018-07-06 18:32:15 INFO  kafka.client.ClientUtils$ - Fetching metadata from broker BrokerEndPoint(0,kafka-server,9092) with correlation id 12 for 1 topic(s) Set(my-topic)

2018-07-06 18:32:15 INFO  kafka.producer.SyncProducer - Connected to kafka-server:9092 for producing

2018-07-06 18:32:15 INFO  kafka.producer.SyncProducer - Disconnecting from kafka-server:9092

2018-07-06 18:32:15 INFO  o.a.samza.system.kafka.GetOffset - Validating offset 6883929 for topic and partition [my-topic,10]

2018-07-06 18:32:15 WARN  o.a.s.s.kafka.KafkaSystemConsumer - While refreshing brokers for [my-topic,10]: org.apache.kafka.common.errors.OffsetOutOfRangeException: The requested offset is not within the range of offsets maintained by the server.. Retrying.