rate_limits: Geek Sites Rate Limits
In jchrom/geeks: Geek API Client

Description Exponential back-off Throttling

Geek sites rate limits are not very well documented, and may even vary in relation to server load. This makes it hard to predict when the limit will be breached. Exponential back-off is used to mitigate this.

This means that after an unsuccessful request, for example one that returned a response with status 429 ("Too many requests"), R will pause for a time and then try again. It will try 3 times in total, including the initial time, increasing the pause duration before each-retry. This is implemented using httr::RETRY(), according to the formula 5 * 2 ^ attempt seconds, with the minimum of 5s pause.

This will make it more likely that a large amount of data is retrieved successfully, but will not help in other situations (e.g. badly formed request).

According to some observations, throttling can vary depending on the current server load.

This is relevant for some functions like get_games() which allow you to specify a vector of id numbers, and the API does not seem to place a hard limit on how long it can be. This allows you to request hundreds of records at a time, but if the server decides it is too busy, you will only get a 429 error back. Exponential back-off might not be enough to resolve this if the server remains busy for a longer period of time.

To mitigate this, if you are planning to make intensive requests, please pick a time of day when the traffic is lowest (basically night time in the US). To help distribute the load during paging (i.e. when fetching comments or plays), you might also use the delay_s argument to space out the requests.

jchrom/geeks documentation built on Oct. 24, 2020, 9:01 a.m.