Faisal,

How much heap do you have allocated to your NiFi instance? In conf/bootstrap.conf
the default value is 512 MB. If you haven't changed that, you could be just running out of
heap.

Also, have you changed the maximum number of threads available to your NiFi instance?
In the top-right menu you can go to Controller Settings. The default for "Maximum Timer Driven
Thread Count" is 10, but you'll definitely want to increase that for your use case.

Also, how many cores does the VM/node that NiFi is running on have?

Thanks
-Mark
On Jun 14, 2018, at 3:39 AM, Faisal Durrani <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:

it appears that the error only comes when i increase the number of consumers. It seems to work fine with around 100 consumers but as soon as i bump it up at about 150 consumers all hell breaks lose with error such as below

"Commit cannot be completed due to group rebalance"
"Failed to retain connection due to No current assignment for partition TEST_KAFKA_TOPIC: "

Quite strangely the consumers start consuming messages that have already been read before. I cant wrap me head around why that is happening.
Can someone point to the right direction here? I tried wild carding all the topics in one consumer but then it becomes very slow. All the consumers are running on the primary node with single concurrent tasks .

On Mon, Jun 11, 2018 at 11:17 AM Joe Witt <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
So you have a unique instance of the ConsumeKafka proc for each topic rhen, right?

Id increase the flow controller thread pool size by quite a bit as well.

On Sun, Jun 10, 2018, 10:13 PM Faisal Durrani <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi,

Yes the kafka service is hosted on a single server while NIFI is on a cluster of 4 servers. I'm not entirely sure what wild carding of topics is but kafka is integrated with a Oracle golden gate and the topics are auto generated as soon as a new table is created in Oracle.

"If you want that in a single instance you may need to alter the timeout associated with any single kafka consumer. "
Can let me know which configuration is this?

The consumer kafka processors are scheduled(Timer driven ) to run on all four nodes with 4 concurrent task so i assume there will be 16 threads.( the Maximum timer driven thread count is set to 40)

We tested the whole data flow with about 15-20 consumers and everything worked fine with out any errors. We started getting all these wired errors as soon as we bumped up our load test with 150+ consume kafka processors.

On Mon, Jun 11, 2018 at 10:59 AM Joe Witt <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hello

Is this a single instance with wildcarding of topics?  Please share config details.

If you want that in a single instance you may need to alter the timeout associated with any single kafka consumer.  The assignment will be per topic per partion.  How many he threads for that processor?

Finally, consider using ConsumeKafkaRecord and if you are using kaka 1 or newer use the latest processor.

thanks

On Sun, Jun 10, 2018, 9:21 PM Faisal Durrani <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Does anyone know about this error from Kafka? I am using Nifi 1.5.0 with ConsumerKafka processor.

ConsumeKafka[id=34753ed3-9dd6-15ed-9c91-147026236eee] Failed to retain connection due to No current assignment for partition TEST_KAFKA_TOPIC:

This is the first time we are testing Nifi to consume from over 200 topics and its failing terribly so far. When this error goes the other one comes up which is as below

Was interrupted while trying to communicate with Kafka with lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@6cb8afba. Will roll back session and discard any partially received data.