Subject: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations


Hi Tom,

Thanks for the feedback.

>> If I understand the proposed throttling algorithm, an initial request
would
>> be allowed (possibly making K negative) and only subsequent requests
>> (before K became positive) would receive the QUOTA_VIOLATED. That would
>> mean it was still possible to block the controller from handling other
>> events – you just need to do so via making one big request.

That is correct. One could still create one big topic (not request) and
that would
create some load on the controller. All options suffer from this issue as
we can
stop clients from creating a very large topic. At least, when it happens,
the client
will have to wait to pay back its credits which guarantee that we control
the average
load on the controller.

>> While the reasons for rejecting execution throttling make sense given the
>> RPCs we have today that seems to be at the cost of still allowing harm to
>> the cluster, or did I misunderstand?

Execution throttling would also suffer from large topics being created. We
have
rejected it due to the current RPCs and also because it does not limit the
amount
of work queued up in the controller. Imagine a low quota, that would result
in a huge
backlog of pending operations.

Best,
David

On Mon, May 11, 2020 at 4:57 PM David Jacot <[EMAIL PROTECTED]> wrote:

> Hi Anna and Jun,
>
> Anna, thanks for your thoughtful feedback. Overall, I agree with what you
> said. If I summarize, you said that using time on server threads is not
> easier
> to tune than a rate based approach and it does not really capture all the
> load neither as the control requests are not taken into account.
>
> I have been thinking about using an approach similar to the request quota
> and
> here are my thoughts:
>
> 1. With the current architecture of the controller, the ZK writes
> resulting from a
> topic being created, expanded or deleted are spread among the api handler
> threads and the controller thread. This is problematic to measure the real
> thread
> usage. I suppose that this will change with KIP-500 though.
>
> 2. I do agree with Anna that we don't only care about requests tying up
> the controller
> thread but also care about the control requests. A rate based approach
> would allow
> us to define a value which copes with both dimensions.
>
> 3. Doing the accounting in the controller requires to attach a principal
> and a client id
> to each topic as the quotas are expressed per principal and/or client id.
> I find this a
> little odd. These information would have to be updated whenever a topic is
> expanded
> or deleted and any subsequent operations for that topic in the controller
> would be
> accounted for the last principal and client id. As an example, imagine a
> client that
> deletes topics which have partitions on a dead broker. In this case, the
> deletion would
> be retried until it succeeds. If that client has a low quota, that may
> prevent it from
> doing any new operations until the delete succeeds. This is a strange
> behavior.
>
> 4. One important aspect of the proposal is that we want to be able to
> reject requests
> when the quota is exhausted. With a usage based approach, it is difficult
> to compute
> the time to wait before being able to create the topics. The only that we
> could do is
> to ask the client to wait until the measure time drops to the quota to
> ensure that a new
> operations can be accepted. With a rate based approach, we can
> precisely compute
> the time to wait.
>
> 5. I find the experience for users slightly better with a rate based
> approach. When I say
> users here, I mean the users of the admin client or the cli. When
> throttled, they would
> get an error saying: "can't create because you have reached X partition
> mutations/sec".
> With the other approach, we could only say: "quota exceeded".
>
> 6. I do agree that the rate based approach is less generic in the long
> term, especially if
> other resource types are added in the controller.
>
> Altogether, I am not convinced by a usage based approach and I would