Today, most of the applications are API-driven. Whether creating a weather forecast, a financial ticker, a sports score alert, or translating a local language, you’ll need to connect 3rd party APIs to access the data you need. APIs are typically metered and restricted in consumption through a mechanism called API Rate.

Setting up the scenario

Assume we want to retrieve the weather forecast for all 19000 (approximately) postal codes (PINs) in India. One request contains only one postal code’s prediction, and the maximum API call rate is 50 per second. We can’t hit 19000 requests at once to retrieve data. We need to build a mechanism for requesting at pace.

This post will demonstrate how to create a Pacer (in Typescript) that takes all requests for pin codes and hits API at the appropriate rate. Both the browser and node sides would be compatible with the code.

Building the Interface

Pacing logic might well be contained within a class. It is best to keep API call logic and pacing logic separated. Pacing can then be applied in different contexts.

Pacer Interface

class Pacer accepts requests and processes them at a given pace. A Request is a self executable code for fulfilling it, and Pacer decides when to run that code.

interface Request<T> {
  (): Promise<T>;
}

class Pacer<T> {
  private ratePerSecond: number;
  constructor(ratePerSecond: number) {
    this.ratePerSecond = ratePerSecond;
  }

  pace(request: Request<T>): Promise<T> {
     // implementation pending
    return request();
  }
}

Using Pacer

To use Pacer, we must create an instance of it with the required rate. After then, pace method can be used to add requests.

let pacer = new Pacer<Axios.Response>(50);

let resultPromises = [];
for(let pinCode of pinCodes) {
  resultPromises.push(pacer.pace(() => $axios.get(`https://api.weather.org/weather/${pinCode}`));
}

let responses = await Promise.all(results);

Implementing Pacer

The functionally correct basic implementation of pace method can be:

When using the above implementation, pace method would execute the request immediately after the reception, similar to code without pacing. We thus need to take this further.

Micro-batching

The requests must be batched and executed once per second for appropriate implementation. This technique is referred to as micro-batching: Smaller batch, Frequent Execution. To complete requests in a controlled manner, we need a Queue and an Executor.

Queue

A request will be queued as soon as it is received. We also need to return a Promise of the request result. But we can’t resolve the promise without executing the request.

pace(request: Request<T>): Promise<T> {
  this.q.push(request);
  // what to return ??
}

Proxy Promise

A Proxy Promise, which will capture the result of the request when executed, will be returned. We need to store its resolve and reject references along with the request.

pace(request: Request<T>): Promise<T> {

  let requestResolve, requestReject;
  let result = new Promise<T>((resolve, reject) => {
    requestResolve = resolve;
    requestReject = reject; 
  })
  this.q.push({request, requestResolve, requestReject});

   // execution code.. implementation pending;
  return result;
}

Execution

A solid design choice is keeping request execution in a separate function that executes multiple queued requests. Now, we need to work on triggering it once every second.

private executeRequests() {
  let els = this.q.splice(0, Math.min(this.ratePerSecond, this.q.length));
  for (let el of els) {
    el.request()
        .then(el.requestResolve)
        .catch(el.requestReject)
  }
}

setTimeout() is reasonable construct to use than setInterval(), as with setTimeout we can control the next execution.

Scheduling Logic

1. When a request is accepted, it may be processed immediately or after a few milliseconds of delay if no execution is scheduled.
2. On the execution side, after triggering requests, we check the queue size for the scheduling next execution with 1s delay.

Scheduling Logic | Image is partly created using nomnoml.com

private scheduleRequests() {
  if (this.exectutorId == null) {
    // MIN_WAIT_TIME = 0 means immediate or MIN_WAIT_TIME = 50 after 50 millis seconds
    this.exectutorId = setTimeout(() => this.executeAndScheduleNext(), Pacer.MIN_WAIT_TIME);
  }
}

private executeAndScheduleNext() {
  this.executeRequests();

  // clear schedule
  clearTimeout(this.exectutorId);
  this.exectutorId = null;

  // next schedule
  if (this.q.length > 0) {
    this.exectutorId = setTimeout(() => this.executeAndScheduleNext(), 1000);
  }
}

The implementation works if requests are completed within one second; it is hardly the case in the real world. To make it more effective, we must track the number of requests being processed and accordingly adjust the pace.

pace(request: Request<T>): Promise<T> {

  // rest code

  // tracking
  let requestWithExecutionTracker: Request<T> = () => {
    this.requestsInExecution++;
    return request();
  };
  this.q.push({
    request: requestWithExecutionTracker,
    requestResolve,
    requestReject,
  });

  // rest code
  return result;
 }

private executeRequests() {
  let els = this.q.splice(0, Math.min(this.ratePerSecond - this.requestsInExecution, this.q.length));
  for (let el of els) {
    el.request()
      .then(el.requestResolve)
      .catch(el.requestReject)
      // tracking 
      .finally(() => this.requestsInExecution--);
  }
}

The full code is accessible at:

GitHub — DM8tyProgrammer/api-pacer

What’s next?

It is practical implementation. Another option is to include a retry or failure mechanism. I have not yet thought of implementing this feature; I would appreciate your feedback to improve it.