Subject: Deletion of topic from a pattern in KafkaConsumer if auto-create is true will always recreate the topic


Yep, but

"Consider this example: if you had a Kafka Consumer that was consuming from
topic A, you did a savepoint, then changed your Kafka consumer to instead
consume from topic B, and then restarted your job from the savepoint.
Before this change, your consumer would now consume from both topic A
and B because
it was stored in state that the consumer was consuming from topic A. With
the change, your consumer would only consume from topic B after restore
because it now filters the topics that are stored in state using the
*configured
topics*."

Yep, the issue is that our pattern is all inclusive and thus *configured
topics *implies all topics for the all inclusive pattern and thus all of
our 1500 plus topics . We use flink to push all data in all topics to
hdfs.  We now are in the process of pruning our topics ( and as you can
imagine, they do not follow a specific pattern on name )  and thus removal
of some 500 of them. We would have assumed that removing the topics from
the cluster would be an indication to flink that though it has state
associated with those topics and irrespective of pattern,  the topics
should no longer be consumed, ver akin to auto discovery of partitions (
and thus topics ) but opposite.

Does that make sense ?
On Fri, Jun 14, 2019 at 3:24 AM Fabian Hueske <[EMAIL PROTECTED]> wrote: