Subject: Running a stand alone Mirror Maker 2.0 cluster and Issues


Hey Ryanne,
           The test was on topics that had a 7 day retention. Which
generally implies that the batch size for flush is pretty high ( till the
consumption becomes current ). The  offset.flush.timeout.ms defaults to 5
seconds and the code will not send in the offsets if the flush is not
complete. Increasing that time out did solve the "not sending the offset to
topic" issue.

Two questions ( I am being greedy here :) )

1. It seems though that flushing the flushing the producer and setting the
offset to the compacting topic is not atomic  OR  do we use
transactions here  ?

2. I see

 WorkerSourceTask{id=MirrorHeartbeatConnector-0} flushing 956435

 WorkerSourceTask{id=MirrorSourceConnector-1} flushing 356251

 WorkerSourceTask{id=MirrorCheckpointConnector-2} flushing 0

 WorkerSourceTask{id=MirrorCheckpointConnector-3} flushing 0 outstanding
messages
I think these are 4 threads ( b'coz num.tasks=4 ), and I have 2 topics with
1 partition each. Do I assume this right, as in there are 4 consumer groups
( on CG per thread ) ...

THANKS A LOT
Vishal.

On Mon, Oct 14, 2019 at 3:42 PM Ryanne Dolan <[EMAIL PROTECTED]> wrote: