Handling Roster API Rate Limits and Retries in Python

The exact task this guide solves is pulling crew roster data from an upstream API reliably — respecting its rate limit so you are never throttled, and recovering from transient failures without hammering it or losing data. Crew management APIs are rate-limited and occasionally flaky, and a naive client that fires requests as fast as the event loop allows will be throttled into failure exactly when a schedule change most needs to sync. The fix is a client-side limiter that paces requests, honours the server’s Retry-After, and backs off with jitter on transient errors while failing fast on permanent ones. This page builds that client. It is part of the crew roster API integration section and feeds the normalised events the flight data ingestion pipeline depends on.

Prerequisites

Pace first, retry second

Two distinct mechanisms keep an integration healthy, and conflating them is a common mistake. Rate limiting is proactive: it paces outbound requests so the client stays under the server’s limit and is never throttled in the first place. Retrying is reactive: it recovers from the failures that happen anyway — a transient 503, a dropped connection — by waiting and trying again. A client needs both, applied in that order: pace every request through a limiter, and wrap each request in a retry that distinguishes transient from permanent failure.

Step 1 — Pace requests with a token bucket

A token bucket refills at the allowed rate and each request spends a token, so bursts are smoothed to the sustainable rate without a fixed delay between calls. An async lock keeps the accounting correct under concurrency.

import asyncio
import time


class TokenBucket:
    def __init__(self, rate_per_sec: float, burst: int):
        self._rate = rate_per_sec
        self._capacity = burst
        self._tokens = float(burst)
        self._updated = time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self) -> None:
        async with self._lock:
            now = time.monotonic()
            self._tokens = min(self._capacity, self._tokens + (now - self._updated) * self._rate)
            self._updated = now
            if self._tokens < 1.0:
                await asyncio.sleep((1.0 - self._tokens) / self._rate)
                self._tokens = 0.0
            else:
                self._tokens -= 1.0

Verify: with a rate of 5 per second and a burst of 5, five immediate acquire() calls proceed at once, and the sixth waits roughly 200 ms — the bucket smooths the burst to the sustainable rate.

Step 2 — Retry transient failures with backoff and jitter

Transient failures are retried with exponentially growing waits and random jitter, so many clients recovering at once do not synchronise into a thundering herd. A server-supplied Retry-After always wins over the computed backoff, because the server knows best when to return.

import httpx

RETRYABLE = {429, 502, 503, 504}


async def get_with_retry(client: httpx.AsyncClient, bucket: TokenBucket, url: str,
                         *, max_attempts: int = 5) -> httpx.Response:
    import random
    for attempt in range(max_attempts):
        await bucket.acquire()
        resp = await client.get(url)
        if resp.status_code < 400:
            return resp
        if resp.status_code not in RETRYABLE:
            resp.raise_for_status()                       # permanent — fail fast
        retry_after = resp.headers.get("Retry-After")
        wait = float(retry_after) if retry_after else min(2 ** attempt, 30) + random.uniform(0, 1)
        await asyncio.sleep(wait)
    resp.raise_for_status()
    return resp

Verify: a 404 raises immediately without retrying; a 503 with Retry-After: 2 waits exactly two seconds; a 503 without the header waits roughly 1, 2, 4, 8 seconds across attempts, each plus jitter.

Step 3 — Fail fast on permanent errors

Retrying a 401 or a 400 wastes the rate budget and never succeeds. The client distinguishes permanent from transient by status code and raises immediately on the permanent ones, so a misconfigured credential surfaces as an error rather than five slow, doomed attempts.

def is_permanent(status: int) -> bool:
    return status in {400, 401, 403, 404, 422}
Rate-limited request with retry and backoff A request first passes through a token bucket that paces it under the rate limit. On a transient failure it retries with exponentially growing waits plus jitter, unless the server supplies a Retry-After header, which is obeyed exactly. A permanent error such as 401 fails fast without retrying. Request Token bucket pace to limit GET status? 2xx — return 429/5xx — backoff honour Retry-After 4xx — fail fast retry through the bucket
Every request is paced by the bucket, then routed by status: success returns, a transient failure backs off (honouring Retry-After) and retries through the bucket, and a permanent failure fails fast.

Failure modes and troubleshooting

Frequently Asked Questions

Why pace requests instead of just retrying on 429?

Because being throttled already cost you a failed request and a forced wait. A token bucket keeps you under the limit proactively, so 429s become rare exceptions rather than the normal control flow. Retrying is the safety net, not the pacing mechanism.

Should the backoff be exponential or fixed?

Exponential with jitter. A fixed delay either recovers too slowly or hammers a struggling server; exponential growth backs off proportionally to the trouble, and jitter prevents many clients from synchronising their retries into a herd.

What makes a status code retryable?

Transient server-side or throttling conditions — 429, 502, 503, 504 — are retryable because the same request may succeed shortly. Client errors — 400, 401, 403, 404, 422 — are permanent: the request itself is wrong, so retrying cannot help.

How does this interact with idempotency?

Roster fetches are reads and inherently idempotent, so retrying is always safe. Writes need explicit idempotency keys before they can be retried, which is covered by the idempotent-ingestion patterns in the async workflows section.

Back to Crew Roster API Integration.