Request limits and retries

Check the selected model and the current platform limit before sending a request.

Text requests use the selected model's verified context window. Estimated input, including tool definitions and history, plus requested max_tokens must fit that window. Check context_length and max_output_tokens in the catalogue. A null value means the provider has not published an exact limit; existing platform safeguards still apply.

A TOKEN_LIMIT error includes a stable failure_code, the limit, estimated input tokens and requested maximum output. Token estimation is approximate; the upstream tokenizer determines actual usage. Vision and staging retain their own limits.

Current platform limits

  • Request rate, counted per client IP address over one minute: 60 requests for POST /v1/chat/completions and POST /v1/decisions, 10 for POST /v1/images/generations, and 120 for GET /v1/models.
  • Concurrency: a workspace can have up to 4 requests in progress at once, across all of its keys.
  • Spending: each request reserves its maximum possible cost against the workspace balance and the key's remaining spending cap.
  • Upstream deadline: the gateway waits up to 90 seconds for the model service. Set your client timeout above that.

429 and 402 responses

  • 429 RATE_LIMITED: the per-minute rate was exceeded. The response includes a Retry-After header in seconds; wait at least that long.
  • 429 CONCURRENCY_LIMIT: four requests are already in progress. Wait for one to finish, and cap your worker pool at four.
  • 429 KEY_QUOTA: the reservation would exceed the key's remaining cap. Raise the cap by creating a new key, or lower max_tokens.
  • 402 INSUFFICIENT_CREDITS: the available balance, net of other reservations, cannot cover the reservation. Add credits or lower max_tokens.

These refusals happen before a request is recorded or reserved, so no model call was made. You can resend the same request, with the same Idempotency-Key, after waiting. When no Retry-After header is present, back off exponentially with random jitter and a maximum number of attempts, rather than retrying in a tight loop.

How max_tokens affects the reservation

When max_tokens is omitted, the chat endpoint uses 1024, or 4096 for a request that includes tools. The reservation covers the estimated input at the model's highest input rate plus max_tokens at its output rate, and is released down to the verified usage when the request settles. A large max_tokens can therefore trigger KEY_QUOTA or INSUFFICIENT_CREDITS even when the actual answer would be short. Request only the output you need.

Uncertain outcomes

If the API reports a rate limit, reduce concurrency and retry later. If a request has an uncertain outcome, inspect its status in your workspace before sending new work with a new idempotency key. Reusing a reserved key returns IDEMPOTENCY_CONFLICT without another upstream call or debit; it does not replay the previous response. A new key starts a separate, potentially billable request. The current response format and error handling are described in the API reference.

A 502 UPSTREAM_UNCERTAIN response, including a gateway timeout, means the credit reservation stays held for reconciliation. On chat completions it also reports a failure_code and a billing_status, normally reservation_held. Do not retry it automatically with a new key. The guide LLM API timeouts: reconcile usage before retrying explains the workflow.

Usage is prepaid. Review pricing and reservations before production traffic.