Production checklist

Tuning knobs that matter when you move from a 10-item test to a real bulk run. Each item links to the deeper reference where one exists.

When a run is already failing or appears stuck, use the Troubleshooting and FAQ alongside this preflight list.

1. Concurrency (`concurrency` / `max_workers`)

LLM calls are I/O-bound, so concurrency is "how many calls in flight at once", not a CPU count. Do not use os.cpu_count().

Prefer the single concurrency=N knob. It sets max_workers and max_provider_concurrency to N and, for factory-built OpenAI-compatible models, sizes the connection pool too (see Choosing Your Limits). Set max_workers separately only when you want the framework wider or narrower than the provider; an explicit value wins.

Situation	Starting point
General I/O-bound (most providers)	`5`–`10`
Rate-limited / low-quota endpoint	`3`–`5`, lean on the coordinated cooldown
High-concurrency provider (e.g. DeepSeek, self-hosted vLLM)	`50`–`200` — with `concurrency=N` the pool is sized to match (below)

Throughput from added workers flattens out well before you exhaust sockets/fds, so raising max_workers past the point where you're latency-bound just adds contention. Measure with examples/benchmark_worker_overhead.py (no network).

2. Connection pool (`max_connections`) vs `max_workers`

With concurrency=N and a factory-built OpenAI-compatible model (llm("...") or from_api_key() without max_connections), the pool is sized for you: N connections, all kept alive, before the first request.

Otherwise the openai SDK default applies to OpenAIModel / OpenRouterModel / DeepSeekModel: 1000 connections, of which 100 are kept alive. Above 100 concurrent requests, extra connections are opened and torn down for each request (a fresh handshake per call). Above 1000, extra workers wait inside httpx for a connection; that wait counts against attempt_timeout but isn't reported as a capacity wait. When you set max_workers yourself, or want a pool size other than concurrency, set max_connections explicitly (it sets both the connection and keep-alive limits):

model = DeepSeekModel.from_api_key(
    "deepseek-v4-flash",
    max_connections=200,   # match your ProcessorConfig(max_workers=...)
)

Models built with from_api_key(max_connections=N) advertise that capacity to their strategy. ParallelBatchProcessor and LLMCallPool emit a UserWarning when max_workers > N; the shared executor holds excess attempts in ABL admission before strategy.execute() and before attempt_timeout starts. Matching values avoids unnecessary admission wait:

model = DeepSeekModel.from_api_key("deepseek-v4-flash", max_connections=32)
strategy = DeepSeekStrategy(model)
config = ProcessorConfig(max_workers=32)

ABL cannot reliably inspect a caller-supplied AsyncOpenAI/httpx transport, so models constructed directly with OpenAIModel(model, client) advertise unknown capacity and do not warn. Set max_provider_concurrency to protect a known custom-client limit:

config = ProcessorConfig(
    max_workers=100,
    max_provider_concurrency=32,  # outside attempt_timeout
)

When both the config and strategy advertise a capacity, the lower value wins. Limits are scoped to the underlying model, so multiple ModelStrategy instances sharing one model also share one capacity semaphore.

See OpenAI integration → connection-pool sizing.

3. Open-file limit (`RLIMIT_NOFILE`)

Each in-flight request holds a socket (a file descriptor). A high max_workers plus the connection pool plus your app's own fds can hit the OS open-file limit — OSError: [Errno 24] Too many open files. This bites hardest on macOS (default soft limit ~256). The processor emits a UserWarning at construction when max_workers is close to the soft limit; it does not raise the limit for you. Fix by raising it (ulimit -n 8192, or resource.setrlimit early in the process) or lowering max_workers. Full guidance: Troubleshooting → throughput and connection pools.

4. Timeout and concurrency semantics

attempt_timeout is per attempt, enforced via asyncio.wait_for around each execute() — it is not a total budget across retries. With retry.max_attempts=3, a single item can spend up to ~3 × attempt_timeout in calls, plus backoff waits.

The timeout boundary is deliberately narrow:

Phase	Counts against `attempt_timeout`?
Batch queue wait / streaming backpressure	No
Worker or shared-call semaphore admission	No
Provider-capacity admission	No
Coordinated cooldown / post-cooldown slow-start	No
Proactive RPM/TPM quota wait	No
`strategy.execute()`	Yes
httpx pool wait occurring inside `strategy.execute()`	Yes
Retry backoff between attempts	No

The transport-pool row is the common trap: ABL cannot distinguish provider time from a lower-level wait once strategy.execute() begins. Advertised or explicit capacity prevents that hidden wait by gating attempts first:

# Safe: 100 workers may do middleware/post-processing, but only 32 attempts
# enter strategy.execute() at once.
model = DeepSeekModel.from_api_key("deepseek-v4-flash", max_connections=32)
config = ProcessorConfig(max_workers=100, attempt_timeout=30)

Aligning the worker count avoids admission queues when the extra workers provide no other benefit:

model = DeepSeekModel.from_api_key("deepseek-v4-flash", max_connections=32)
config = ProcessorConfig(max_workers=32, attempt_timeout=30)

Rate limits are exempt from max_attempts (a 429 is "wait and retry", not a failed attempt — see below), so a throttled item can sit through many cooldowns without consuming its attempt budget. Bound that separately with retry.max_rate_limit_retries (default 20). Net: size attempt_timeout for one slow call, and use the two retry budgets to bound total effort.

ProcessorConfig(
    attempt_timeout=60.0,          # per attempt
    retry=RetryConfig(
        max_attempts=3,              # content/transport failures
        max_rate_limit_retries=20,   # throttling retries (separate budget)
    ),
)

For LLMCallPool, semaphore wait is outside attempt_timeout, while submit_timeout wraps the full caller path: both admission waits, cooldown, all attempts, and backoff. Use submit_timeout for an end-to-end request latency budget and attempt_timeout for one provider attempt.

Each WorkItemResult.admission_wait_seconds reports cumulative provider-capacity wait across attempts. get_stats() exposes total/max item wait, and MetricsObserver exposes attempt-level count, sum, max, and average wait. WorkItemResult.timing further separates execution, built-in provider-call, cooldown, retry-backoff, and startup-ramp time for every physical try. Processor stats retain the latest 10,000 attempt samples and report admission and execution p50, p95, and p99 values.

5. Rate-limit configuration

When one worker hits a rate limit (HTTP 429 or an equivalent such as RESOURCE_EXHAUSTED, error_category="rate_limit"), the framework runs a coordinated cooldown — all workers pause, then slow-start back up — instead of each worker hammering a throttled endpoint. Overload (503) and other 5xx errors use per-item retry backoff instead. Billing and daily-quota exhaustion (insufficient_balance, quota_exhausted) fail without retry; see ErrorCategory. Tune the cooldown via RateLimitConfig:

Field	What it does
`cooldown_seconds`	Base pause after a rate limit (a server `Retry-After` raises it as a floor)
`backoff_multiplier`	Grows the cooldown on consecutive rate limits
`slow_start_items` / `slow_start_initial_delay` / `slow_start_final_delay`	Ramp delays as workers resume after a cooldown
`max_cooldown_seconds`	Cap on the escalated cooldown (default 600s)

Pair with proactive limiting (ProcessorConfig(max_requests_per_minute=...)) to stay under quota before you trip a 429 at all.

For cold-start burst protection, configure StartupRampConfig separately. It limits initial concurrency and raises it by a fixed step on each interval; unlike the fields above, it applies before the first provider call and does not require a preceding rate limit. Ramp wait remains outside attempt_timeout.

See the OpenAI-compatible high-throughput guide for owned/custom client recipes and troubleshooting.

6. Token-aware admission

Before enabling max_tokens_per_minute, verify all of the following:

Identify the provider/account that owns the quota and intentionally share a stable quota_scope across every strategy spending that budget.
Keep independent account quotas in distinct scopes. Do not use a credential, credential-derived string, or arbitrary repr() as a scope label.
Use a provider tokenizer where possible and measure estimator accuracy on representative small, medium, and large prompts.
Reserve expected output as well as input. Check that the largest single estimate fits inside max_tokens_per_minute.
Monitor refunded_tokens, underestimated_tokens, unknown_usage_attempts, and known_zero_usage_attempts; sustained debt means the estimate is too low.
Confirm whether an upstream gateway retries internally and whether it reports aggregate usage. Hidden attempts cannot be reconstructed by ABL.
Test one recoverable failed-usage retry, one unknown-usage timeout/transport failure, and cancellation both before and after provider start.
Verify replay and dry-run emit no quota events and do not consume live admission state.
Know that configuration failures (token_estimator_required, token_estimation_error, token_estimate_exceeds_limit, quota_scope_error) are never replayed on resume; fix the configuration and they re-execute. See configuration failures.

See Token-Aware Admission for reservation, reconciliation, FIFO, and visibility semantics.

7. Bounded streaming for large inputs

For a very large (or unbounded) input, don't buffer all the work up front. Use streaming mode with bounded max_queue_size and max_result_queue_size: workers run while you feed, so full queues apply backpressure instead of deadlocking or accumulating work in proportion to input size.

from async_batch_llm import ProcessorConfig, process_stream

config = ProcessorConfig(
    max_workers=50,
    max_queue_size=200,
    max_result_queue_size=100,
)

async for result in process_stream(strategy, huge_prompt_source, config=config):
    if result.success:
        await save(result.item_id, result.output)   # completion order

huge_prompt_source can be any sync or async iterable (e.g. a generator reading a file lazily). The low-level equivalent is processor.start() / add_work() / finish() / results().

process_prompts() retains every result in its returned BatchResult, and process_all() requires work to be added before workers start. Neither is a bounded-memory result path for an unbounded workload. process_stream() avoids retaining results itself; set max_result_queue_size so a slow consumer applies backpressure to provider workers instead of accumulating completed items. See Bounded Work and Backpressure for incremental database input, low-level streaming, and separate input/output/provider/shared-call limits.

8. Single calls and the shared call pool (request paths)

For a web service's request path — where work arrives one call at a time, not as a batch — use LLMCallPool instead of standing up a processor per request:

One long-lived pool per app. Create it once at startup (e.g. a FastAPI lifespan handler) and share it across all request handlers. A single pool means one shared rate-limit cooldown — when one caller hits a 429, all callers briefly pause and then slow-start, instead of a thundering herd.
Set max_pending and submit_timeout for web paths. max_pending caps how many requests may wait beyond the max_workers running ones (total in flight ≤ max_workers + max_pending), so an overload sheds load instantly (rejecting with a failed result) rather than growing an unbounded waiter list; submit_timeout bounds per-caller latency so a request stuck behind a cooldown returns instead of hanging the handler. Both are off by default.
Do not create unbounded outer tasks. max_pending bounds pool admission, but one large asyncio.gather() still materializes every caller task. Use the bounded batch pattern for ingestion jobs or an explicitly bounded task window.
Shutdown drains admitted requests. aclose() (the async with exit) stops accepting new work, then waits for already-admitted requests to finish before cleaning up the shared strategy, so in-flight calls aren't cut off mid-flight. Set submit_timeout to bound how long shutdown waits for that drain; with no timeout it waits as long as the admitted work takes.

For a single ad-hoc call, call() / call_result() run one prompt through the same resilience pipeline with no pool at all.

9. Spend limits

Set GuardrailConfig(max_total_tokens=...) on large or unattended runs, and max_total_cost with your own cost_function when a dollar ceiling matters. Caps are soft: calls already running when the cap is reached can overshoot it by at most one call per worker, so pair a cap with a per-call output limit (max_tokens) if the overshoot must stay small. The budget API is provisional and may change in a minor release (API stability). See Token and cost budgets.

10. Cleanup

Use the processor as an async with context manager so workers, caches, and HTTP clients are released. If you can't, call await processor.shutdown() when done.

11. Artifact storage (checkpoint/resume runs)

Backend choice. JSONL for portable, human-inspectable audit logs; SqliteArtifactStore for 100k+ restartable runs needing indexed replay. See the comparison table.
Disk space and sidecars. Budget from a measured bytes-per-record on your payloads. SQLite runs also create -wal/-shm sidecar files next to the database; the WAL plateaus during healthy writes and is truncated on clean close (an -shm file may harmlessly remain).
Durability mode. SqliteDurability.BALANCED (default) survives process crashes; a power failure can lose recently committed transactions. Choose FULL when power-loss durability matters more than write throughput. Neither mode encrypts the file — use filesystem permissions and volume encryption for sensitive outputs.
Commit batch latency. Results are published only after their transaction commits; commit_interval_seconds (default 0.01 s) is added result latency. Lower it for latency-sensitive streaming, raise the batch size for maximum append throughput.
Write permissions. The store creates parent directories and needs write access for the database and its sidecars.
Single-writer deployment. One writable store instance per artifact file, in one process. Sequential reopen is supported; concurrent external readers obey busy_timeout_seconds. There is no distributed work claiming or cross-process exactly-once execution.
Before million-item claims. Run make scale-100k (and scale-1m for seven digits) on a representative environment and review the report — don't extrapolate from profile existence. See Benchmarks.