Subject: MirrorMaker 2.0 and Streams interplay (topic naming control)


Hey Paul, thanks for the kind words re MM2.

I'm not a Streams expert first off, but I think I understand your question:
if a Streams app can switch between topics with and without a cluster alias
prefix when you migrate between prod and pre-prod, while preserving state.
Streams supports regexes and lists of topics as input, so you can use e.g.
builder.stream(List.of("topic1", "prod.topic1")), which is a good place to
start. In this case, the combined subscription is still a single stream,
conceptually, but comprises partitions from both topics, i.e. partitions
from topic1 plus partitions from prod.topic1. At a high level, this is no
different than adding more partitions to a single topic. I think any
intermediate or downstream topics/tables would remain unchanged, since they
are still the result of this single stream.

The trick is to correctly translate offsets for the input topics when
migrating the app between prod and pre-prod, which RemoteClusterUtils can
help with. You could do this with external tooling, e.g. a script
leveraging RemoteClusterUtils and kafka-streams-application-reset.sh. I
haven't tried this with a Streams app myself, but I suspect it would work.

Ryanne
On Sun, Mar 24, 2019 at 12:31 PM Paul Whalen <[EMAIL PROTECTED]> wrote: