support for rate limiting something that takes non-trivial amount of time and throttles based on completion time

[snaar]

As a simplified example, imagine there is a service that does some slow computation on demand. Amount of time the computation takes is non-deterministic and is essentially random. That service also requires that you wait specific amount of time until you can submit another request. However that amount of time is small compared to how long the work takes.

E.g. let's say service requires 1 second downtime from when previous requests finishes (not starts! finishes!) and new one can be submitted. Requests take anywhere from 0 to 100 seconds to proceed. If you naively create governor with 1 cell that takes 1 second to replenish, then you can easily start submitting requests before previous ones are done. If you use 100 seconds as replenish interval - you will end up being suboptimal and service will be idle most of the time.

Note that in this simple case, governor crate is arguably an overkill, but there are more general cases of this.

It doesn't seem to be possible with the current API. Does this make sense? Is this in scope for this crate?

[antifuchs]

Hi @snaar, thanks for posting this question! I think that's a really interesting use case, even if I'm not sure how to properly support it with governor: Since the entire implementation of it is based on atomic writes without locks, it'll take a bit of work to get support for this.

One (potentially complicated) thought that comes to mind is that it's possible to "peek" at the state of a rate limiter; with that, and two limiters with the same parameters (let's call them "before" and "after"), it ought to be possible to to the following:

1. peek at the state of limiter "after" - if there's capacity available, proceed
2. check & consume an element of limiter "before"
3. do the work and finish
4. check & consume an element of limiter "after".

This should ensure that only if the "after" limiter allows elements through (based on the time the operation finished), another operation will be started, and that no two operations get started based on the "peek" outcome (by ensuring entry to the critical section is limited by the "before" limiter.

Since they both are configured with the same parameters, the "before" limiter will start allowing requests through before "after" - hence the peek at "after"; but if both "before" (by actually consuming capacite) and "after" (via the peek) allow a request, then that means that a request can be made according to "after".

Does that make sense? I could draft up an example (and extend the API where necessary) for that if you think this makes sense.

[snaar]

Yeah this actually make sense. Sounds reasonable to me!