Token-Aware Admission

Token-aware admission is an optional local quota smoother for provider request and token budgets. It controls when an attempt may start; it does not change provider concurrency, transport capacity, or retry policy.

Four controls, four jobs

Control	Bounds	Typical owner
Worker concurrency	Framework tasks doing useful work	`concurrency` / `max_workers`
Provider concurrency	Calls holding a provider-capacity slot	`max_provider_concurrency` or the strategy/model
Requests per minute (RPM)	Physical attempts admitted per quota scope	`max_requests_per_minute`
Tokens per minute (TPM)	Estimated then reconciled token load per quota scope	`max_tokens_per_minute` plus an estimator

Equal request counts are not equal load. Ten classification requests may fit inside one long generation's token budget. Raising workers or connection-pool size cannot overcome RPM or TPM; it only creates more tasks waiting at the quota gate.

The live-attempt order is:

coordinated cooldown → token estimation → atomic RPM+TPM reservation
→ provider-capacity wait → final cooldown check → provider start

No provider-capacity slot is held during estimation or quota waiting. WorkItemResult.quota_wait_seconds is therefore separate from admission_wait_seconds, which measures provider-capacity wait.

Configure RPM and TPM

from async_batch_llm import CharacterTokenEstimator, ProcessorConfig

config = ProcessorConfig(
    concurrency=32,
    max_requests_per_minute=500,
    max_tokens_per_minute=200_000,
    token_estimator=CharacterTokenEstimator(
        characters_per_token=4.0,
        expected_output_tokens=400,
    ),
)

TPM is opt-in. When max_tokens_per_minute is configured, every live provider attempt needs a TokenEstimate. Supply token_estimator on the config, on CallableStrategy, or through a strategy's estimate_tokens() hook. A missing estimator fails before provider work with TokenEstimatorRequired. An individual estimate larger than the configured per-minute token limit fails immediately with TokenEstimateExceedsLimit; it cannot become admissible by waiting.

Configuration failures

These failures describe the run's configuration, not the item. Each is non-retryable and happens before any provider call:

Exception	`error_category`
`TokenEstimatorRequired`	`ErrorCategory.TOKEN_ESTIMATOR_REQUIRED`
`TokenEstimationError` (estimator raised or returned an invalid estimate)	`ErrorCategory.TOKEN_ESTIMATION_ERROR`
`TokenEstimateExceedsLimit`	`ErrorCategory.TOKEN_ESTIMATE_EXCEEDS_LIMIT`
`QuotaScopeError` (a strategy's `quota_scope` could not be resolved)	`ErrorCategory.QUOTA_SCOPE_ERROR`

Since v0.27 they are never replayed from artifacts, even under ResumePolicy.REUSE_ALL. The first three are checkpointed with replay_eligible=False; quota_scope_error fails before an artifact key exists and is not checkpointed. Fix the configuration and resume, and those items execute again. Because a missing estimator or a broken quota scope usually affects every item, TOKEN_ESTIMATOR_REQUIRED and QUOTA_SCOPE_ERROR are good abort_on_error_categories candidates. See ErrorCategory for every category.

CharacterTokenEstimator uses a character-ratio heuristic and a fixed expected output allowance. It is approximate and is never enabled automatically. Prefer a provider tokenizer and a workload-specific output estimate when accuracy matters.

Reservation and reconciliation

Each physical attempt atomically reserves one RPM unit and its estimated input plus output tokens from one FIFO gate. It never reserves RPM and then waits separately for TPM.

After provider start:

Known usage below the estimate refunds the difference.
Known usage above the estimate records debt. Availability may go negative; later attempts wait until refill pays the debt.
Explicit known zero refunds the full token reservation but still consumes the RPM unit because a provider attempt started.
Unknown usage retains the estimate conservatively. This is distinct from known zero.
Recoverable failed-attempt usage is reconciled before the retry. Every physical retry then receives a fresh estimate and reservation.

Cancellation before provider start refunds both the RPM unit and token reservation. Cancellation after provider start follows the same known/unknown usage rules as any other started attempt. An item or batch deadline may expire while waiting; the waiting attempt makes no provider call and leaves no live reservation. A deadline reached during the provider-capacity wait records timing.timeout_category="admission_timeout" and keeps the elapsed wait in admission_wait_seconds; a deadline during the RPM/TPM quota wait does not set admission_timeout. See guardrails.

Dry-run, compatible artifact replay, and middleware-filtered items bypass live quota admission. They emit no quota-admission events and mutate no RPM/TPM state.

Reported usage and canonical totals

Usage knowledge is independent of the compatibility dictionary returned by exception extraction. Missing usage may expose zero-filled counters, but those placeholders do not authorize a token refund.

Provider usage	Overall usage known?	Reported total
Absent, `None`, empty, or unrelated fields only	No	`None`
`cached_input_tokens` only	No	`None`
`total_tokens: 0`	Yes	0
`input_tokens: 7`, `output_tokens: 3`	Yes	10
Input 7, output 3, optional exception total `None`	Yes	10
`input_tokens: 7`, output absent	Yes	7
Recognized counter explicitly `None` or invalid, without a valid total	No	`None`
Input 7, output 3, explicit total 12	Yes	12

A valid explicit total wins over the component sum. In exception usage, an absent total or total_tokens=None permits derivation from input/output counters when at least one is present and both present counters are valid. A non-None invalid total remains unknown. An absent component counts as zero; an invalid component does not. Successful strategy mappings still reject total_tokens=None under their strict validation contract. The derived total is written into the normalized total_tokens field used by results, events, statistics, and new checkpoints. Caller-owned mappings and historical artifacts are not modified. Cached tokens are telemetry, not an additional charge or an automatic deduction from the total. An optional cached_input_tokens=None allows fallback to cache_read_tokens or prompt_tokens_details.cached_tokens; a valid explicit zero remains zero.

Successful strategy mappings require non-negative integers: None, booleans, negative, fractional, and non-finite values fail validation. Exception extraction is best effort and treats invalid counters as unknown rather than known zero. It retains compatibility with integral numeric values and integer strings in provider exception usage.

Exception usage supports mappings, attribute objects, modern/legacy provider field names, and synchronous .usage() accessors. A valid framework _failed_token_usage report takes precedence, followed by cause-result usage and direct exception usage. Empty or invalid framework stamps do not hide a valid lower-priority report. Async accessors are not started or awaited; ordinary accessor failures are best effort, while cancellation and process-control exceptions propagate.

The executor observes each failed physical attempt before recovery hooks or guardrail error replacement. A synchronous processor _extract_token_usage override can supply a valid positive total (including a derived one); calling super() reuses the observation without calling the provider accessor again. Zero-filled legacy overrides cannot turn unknown into known zero or erase a known positive total. Report explicit unbilled usage on the provider exception when a zero refund is intended.

If strategy.on_error adds or changes a valid _failed_token_usage stamp, the executor uses that later report for the failed attempt's result accounting, including when the hook raises or is interrupted by a deadline or batch abort. It does not call the provider accessor again or revise the completed quota reconciliation. A later report replaces that attempt's earlier result usage; it is not added as another attempt. Empty/invalid stamps do not erase previously observed usage. Explicit known zero is a valid later result report.

For example, if a failed attempt reserved 20 tokens and its usage was unknown at reconciliation, admission retains 20. If on_error then reports 10, the item records 10 tokens while its quota timing still shows unknown usage. This preserves usage supplied during recovery without delaying reservation finalization behind hooks that may block, fail, or be cancelled.

Quota timing and QUOTA_RECONCILED describe individual physical attempts. An item that fails with 10 reported tokens and then succeeds with 15 reports 25 item tokens, while its reconciliations report 10 and 15 separately. ITEM_COMPLETED reports the final attempt's tokens. Batch live counters add newly executed usage; replay keeps the historical item usage but adds zero live consumption. Reservation estimates for unknown attempts never become invented provider-reported tokens.

Quota scopes

quota_scope identifies the account or upstream budget shared by strategies. Object identity defines sharing. By default it follows concurrency_scope, so existing strategies preserve their established ownership. Override it when one account quota spans multiple models or when one shared client serves independent accounts.

Quota scope and concurrency scope answer different questions:

quota_scope: which calls spend the same RPM/TPM budget and share cooldown?
concurrency_scope: which calls contend for the same client/provider capacity?

Two models sharing one account quota:

from async_batch_llm import ArtifactIdentity, CallableStrategy, TokenEstimate

shared_quota = object()


async def call_model_a(prompt, *, attempt, timeout, state):
    ...  # call model A and return a CallOutcome


async def call_model_b(prompt, *, attempt, timeout, state):
    ...  # call model B and return a CallOutcome


def estimate_a(prompt, *, strategy, attempt, state):
    return TokenEstimate(input_tokens=count_model_a(prompt), output_tokens=300)


def estimate_b(prompt, *, strategy, attempt, state):
    return TokenEstimate(input_tokens=count_model_b(prompt), output_tokens=600)


strategy_a = CallableStrategy(
    call_model_a,
    quota_scope=shared_quota,
    token_estimator=estimate_a,
    identity=ArtifactIdentity(
        provider="example",
        model="model-a",
        prompt_version="v1",
        parser_version="v1",
        application_version="v1",
    ),
)
strategy_b = CallableStrategy(
    call_model_b,
    quota_scope=shared_quota,
    token_estimator=estimate_b,
    identity=ArtifactIdentity(
        provider="example",
        model="model-b",
        prompt_version="v1",
        parser_version="v1",
        application_version="v1",
    ),
)

For independent account budgets, use distinct identities:

account_a_quota = object()
account_b_quota = object()

strategy_a = CallableStrategy(
    call_model_a,
    quota_scope=account_a_quota,
    token_estimator=estimate_a,
    identity=identity_a,
)
strategy_b = CallableStrategy(
    call_model_b,
    quota_scope=account_b_quota,
    token_estimator=estimate_b,
    identity=identity_b,
)

Do not derive scope identities from credentials or place arbitrary scope representations in logs or metrics. Events expose only run-local ordinal scope IDs.

Retry-aware estimators

The estimator receives the effective middleware-replaced strategy and prompt, the logical attempt number, and the item's RetryState. This allows validation recovery or model escalation to change the estimate:

def estimate(prompt, *, strategy, attempt, state):
    escalated = state is not None and state.get("model") == "large"
    output_allowance = 1_200 if escalated else 300
    return TokenEstimate(
        input_tokens=provider_tokenizer(prompt),
        output_tokens=output_allowance,
    )

Synchronous estimators run off the event loop. Asynchronous estimators are awaited directly. Estimator failures are redacted framework errors and are not sent through strategy or middleware recovery hooks.

Visibility boundaries

Token accounting covers attempts visible to ABL, including recoverable failed-attempt usage. Retries hidden inside an upstream gateway require gateway-reported usage to be visible. If a gateway returns only aggregate usage, ABL cannot reconstruct its internal attempts or exact reservation deltas.

Providers implement quotas differently: fixed windows, rolling windows, weighted model budgets, cached-token rules, and account-level policies all exist. ABL's gate is a local continuously refilled smoother. It reduces bursts but does not claim to reproduce a provider's enforcement exactly.

TPM admission also does not schedule active GPU sequences, KV-cache occupancy, or instantaneous decode-token capacity. Use a serving system or gateway that owns those resources when that is the real constraint.

FIFO trade-off

Admission is FIFO within each quota scope. This prevents later small requests from repeatedly bypassing an earlier large request, but a large head item can delay small items behind it. Split unrelated workloads into intentional quota scopes only when they truly have independent upstream budgets; otherwise the head-of-line behavior reflects a real shared limit.

Observe and troubleshoot

Attempt timing reports estimate components, reserved and reported tokens, reconciliation delta, quota wait, and run-local scope ordinal. Processor stats and MetricsObserver aggregate bounded quota wait percentiles, reservations, reported tokens, refunds, debt, known-zero/unknown attempts, estimator failures, and scope count. QUOTA_ADMITTED and QUOTA_RECONCILED events expose attempt-level evidence without using item or scope IDs as metric labels.

If one estimate exceeds max_tokens_per_minute, raise the limit to the real account budget, reduce expected output, or route that workload to its real independent quota scope. Waiting can't fix it, and quota_burst_seconds doesn't affect this check. If timeouts or 429s

make admission look conservative, inspect unknown_usage_attempts: without a reliable provider total, retaining the reservation is intentional.

See Choosing Your Limits for sizing order and Troubleshooting for symptom-based guidance.

Exception identity and attempt ownership

The executor keeps classification, timing, and cumulative usage in private per-attempt and per-item records. It does not add or update _abl_error_info, _abl_work_item_timing, _abl_admission_wait_seconds, or cumulative _failed_token_usage on provider exceptions. A reused exception is classified against the current strategy on every physical attempt. call() and call-pool raising APIs still raise the original exception instance.

Provider or custom strategy _failed_token_usage stamps remain supported, including read-only mapping views of exception attributes. Such stamps describe provider usage; item-wide totals and timing belong to WorkItemResult. Framework guard replacements carry usage knowledge internally rather than creating zero-filled stamps. An explicit valid zero supplied by a hook remains known zero. A custom hook that writes an ordinary four-key zero dictionary is asserting known zero: the framework cannot distinguish it from a copied unknown-usage compatibility dictionary once that provenance has been discarded.

Burst size and admission races

ProcessorConfig.quota_burst_seconds=None preserves a full-minute bucket. A positive finite value b sets fixed capacity C=max(1, limit*b/60) in each configured dimension, with refill at limit/60 per second. RPM starts with C units. TPM also starts full; a head estimate larger than C can start when the bucket is full and leaves debt that must refill before later requests. Capacity never grows permanently. An estimate larger than the per-minute TPM limit still fails before provider work.

Without refunds, at most C+limit RPM units are admitted in any rolling minute. For TPM reservations the bound is max(C, E_max)+limit, where E_max is the largest admitted estimate in that interval. This exception matters for large requests: the simpler limit*(1+b/60) bound applies only when neither the one-unit floor nor oversized estimates apply. Reconciliation can refund tokens and allow more reservations; these bounds are not guarantees about actual tokens reported by a provider. Provider windows, SDK retries and other processes remain outside this local limiter.

A cooldown may begin while an attempt waits for quota or capacity. The final synchronous start check preempts that attempt, releases capacity, refunds both quota dimensions exactly once, then re-enters admission at the FIFO tail. Preemption consumes no attempt or rate-limit retry budget. Quota, cooldown and capacity waits accumulate on the same attempt timing. Admission/finalization events can repeat for the same physical try; they describe reservations, not provider calls. Cancellation, item/batch deadlines and aborts during re-entry retain the same refund and cleanup rules.

Each provider start snapshots its scope's cooldown generation. A late 429 from an older generation still consumes the item's rate-limit retry budget but does not start another cooldown or escalate its consecutive counter. An exhausted item reports failure without waiting for the shared cooldown it just triggered; that coordinator-owned cooldown continues to pause its peers until completion or host shutdown.

If the rate-limit strategy raises or returns a non-finite or negative duration, ABL logs a warning and peers wait for at least the configured cooldown_seconds (and a longer server hint, subject to its configured cap). Bare ExponentialBackoffStrategy() now uses the same 300-second initial cooldown and 1.5 multiplier as RateLimitConfig. Both retry backoffs are overflow-safe.

Scope registry entries remain strongly owned until ordered host cleanup. Weak entries are deferred: dropping a scope must not orphan its quota wake or cooldown task or bypass a failed cleanup retry.