Speed-based storage throttling

[westonpace]

We currently have some basic controls for storage throttling:

• We have a per-scan throttle (configured as part of the scan) that controls how much data we buffer to avoid running out of RAM
• We have a per-scan throttle (configured as part of the object store) that controls how many concurrent requests we make to storage
• We have a per-process throttle (configured with environment variables) that controls how many concurrent requests we make to storage (across the entire process)

Some storage systems (e.g. hugging face) have speed-based limits (3000 requests / five minutes). We should introduce a new throttling mechanism, configured at the object store, which can control this.

Alternatively, this could be an object_store feature (we could presumably move the concurrent requests throttle into object_store as well).

[westonpace]

@Xuanwo curious if you have any thoughts on who should own the throttling. I think Lance will always own the "how much data can we buffer" throttle but the object storage layer could own the "how many concurrent requests" and "how many requests per minute/second" throttles.

[Xuanwo]

I think Lance will always own the "how much data can we buffer" throttle but the object storage layer could own the "how many concurrent requests" and "how many requests per minute/second" throttles.

yep, makes sense to me.