Results, Audit Artifacts, and Resume

Production runs often need two different outputs: a convenient batch summary and an append-only checkpoint that survives interruption. BatchResult serialization serves the first use case; an artifact store (JsonlArtifactStore or SqliteArtifactStore) serves the second.

Completion order and input order

Batch processing and process_stream() publish results in completion order by default. This keeps fast items visible while a slow or retrying item is still in flight and preserves existing behavior.

Use preserve_order=True when collecting a batch in input order:

result = await process_prompts(
    strategy,
    [("slow", "slow prompt"), ("fast", "fast prompt")],
    preserve_order=True,
)

The processor assigns an internal submission_index when each item is accepted. It does not derive order from item_id, so duplicate IDs are safe (they're accepted, with a one-time UserWarning). To reorder an existing batch without mutating it, call ordered = result.in_input_order(). This raises ValueError if any result predates submission indexes; it never guesses.

Each LLMWorkItem object is a single submission. Duplicate IDs on distinct objects are supported, but submitting the identical mutable object twice raises ValueError; create a new work item for each queue entry.

process_stream() intentionally remains completion ordered. Ordered streaming would block all later results behind one slow early item and require a reorder buffer that can grow without a safe bound.

JSON result serialization

Result mappings use the named async-batch-llm-result schema and an integer schema version. They include terminal status, submission index, error category, replay state, token usage, complete attempt/item timing, and batch termination metadata.

from pathlib import Path
from async_batch_llm import BatchResult

value = result.to_json()
Path("summary.json").write_text(value, encoding="utf-8")

restored = BatchResult.from_json(
    Path("summary.json").read_text(encoding="utf-8")
)

result.to_jsonl("summary.jsonl")
restored_lines = BatchResult.from_jsonl("summary.jsonl")

JSONL result files contain one complete versioned WorkItemResult record per line. A zero-result batch uses one versioned batch-metadata record so termination state still round-trips. These are summary exports, not resumable checkpoints; use an artifact store for checkpoint/resume.

The safe encoder accepts normal JSON primitives plus dataclasses, Pydantic models, enums, datetime/date/time, UUIDs, filesystem paths, tuples, and sets. Tuples and sets normalize to JSON lists; dataclasses and Pydantic models normalize to mappings; dates, UUIDs, and paths normalize to strings. Sets are ordered deterministically. Without explicit decoders, those normalized values remain JSON-native mappings, lists, and strings after deserialization.

Application-specific values need an explicit encoder:

payload = result.to_json(
    encoder=lambda value: {"widget_id": value.id}
)

WorkItemResult.from_dict() and the BatchResult.from_*() methods accept output_decoder and context_decoder hooks for trusted type reconstruction. Output decoding runs only for successful results whose output was included and is not None; failed, omitted, and null outputs bypass it. Context decoding runs only for non-null context. Artifact inspection and replay use the same rules. Deserialization never imports a class named by untrusted data. Exceptions are stored only as module name, class name, and redacted message; the restored runtime exception is None. Tracebacks and raw exception objects are never persisted. Framework-controlled error text and values under structured authorization/API-key/token keys are redacted. Other user-controlled strings round-trip without text rewriting, so applications must keep credentials out of prompts, outputs, context, and metadata.

Unsupported values, malformed input, and future schema versions raise ResultSerializationError; values are never silently replaced with repr(). A reader on the same schema version can also reject a value that a newer release added; see Compatibility across releases.

Middleware and replay

For each accepted item, the processor establishes its total deadline and runs before_process once before preparing artifact identity or looking up an old result. Retries use this effective request without repeating preprocessing. Current filtering therefore always wins over historical success.

A replacement may change prompt, context, and strategy, but cannot change the accepted item ID. Submission indexes are preserved. Fingerprints, inferred provider/model identity, strategy class, lookup, and audit fields describe the effective item; append uses the exact fingerprint prepared before execution. Explicit ArtifactIdentity values still override inferred identity. Automatic stores still require one consistent inferred identity per run, regardless of worker preparation order. Identity and input serialization errors produce a failed artifact_preparation_error result for that item during processing, after middleware. Other accepted items continue; store I/O, format, and checkpoint-write failures still propagate for ordinary execution checkpoints. To stop immediately on identity or input errors, include artifact_preparation_error in GuardrailConfig.abort_on_error_categories; it remains opt-in so isolated bad inputs do not discard valid neighboring results.

Replay retains historical output, usage, and timing, binds current effective context and the current submission index, and neither calls strategy prepare() nor appends a duplicate record. Capacity configuration and admission-scope binding still run once per effective strategy. A completed lookup is used even if a deadline or abort arrives concurrently. after_process runs once for a newly executed success, before append, and does not rerun on replay. Changes to after_process that leave effective inputs unchanged require an application_version or parser_version bump in ArtifactIdentity. Middleware functions are not automatically hashed.

Filtered items produce a current-run middleware_filtered result. Preprocessing filtering and invalid middleware requests do not open the store or append records. Aborts, budget stops, and total-item/batch deadlines are checkpointed for audit when an artifact key can be prepared, with replay_eligible=False. Interrupted artifact preparation is not restarted solely for audit, and unrepresentable inputs cannot be checkpointed. Artifact errors during batch-abort, batch-budget, or batch-deadline audit preparation or append are logged, preserving the controlled stop and its results even when that audit record cannot be persisted. A per-item deadline is an ordinary failed result in an otherwise running batch: its checkpoint errors still propagate, although its record remains ineligible for replay.

An abort can append one audit row per accepted queued item before the run returns. These writes use the store's normal locking and persistence path, so large queues can add substantial abort-drain time. Audit preparation and append are not bounded by the expired guardrails; a store that blocks on open or write can delay draining indefinitely.

Both backends exclude legacy abort/deadline records during replay as well, allowing an older compatible success to be reused. A filter record explicitly appended via the store API has replay_eligible=False, including under REUSE_ALL. Both JSONL and SQLite also reject legacy filter records bearing the exact historical Skipped by middleware error, even when those records were marked replayable. Configuration failures (token_estimator_required, token_estimation_error, token_estimate_exceeds_limit and quota_scope_error) are likewise checkpointed with replay_eligible=False, and legacy records carrying those categories are excluded, so fixing the configuration and resuming re-executes the item even under REUSE_ALL. Unlike abort and deadline audit records, their checkpoint errors propagate. quota_scope_error fails before an artifact key is prepared, so it is not checkpointed at all. Both stores fall back to an older compatible result when a newer record is excluded. Both retain the existing version-1 schema. Newly executed results are still checkpointed before publication or post-processing callbacks.

Resumable JSONL artifacts

An artifact begins with a version-1 manifest followed by versioned item records. The manifest records UTC creation time, package version, canonical identity, its SHA-256 fingerprint, and optional user metadata. Each terminal item record contains input fingerprints, current submission index, strategy class, identity/provenance, safe result data, timing, token use, error category, replay eligibility, and optional caller-calculated cost.

The minimal form needs only a path (v0.20.0):

from async_batch_llm import JsonlArtifactStore, ResumePolicy, process_prompts

result = await process_prompts(
    strategy,
    prompts,
    artifact_store=JsonlArtifactStore("runs/customer-tagging.jsonl"),
    resume=ResumePolicy.REUSE_SUCCESSES,
)

The run closes the store when it finishes: process_prompts(), process_stream() and processor context exit all close the store passed in, and a closed store raises ArtifactIOError if used for another run. Construct a new store instance, on the same path, for each run. Path-based inspection (read_results) still works afterward.

When no ArtifactIdentity is given, provider and model are inferred from ordinary model-backed strategies at run start (built-in models map to their provider names; custom models use their class name) and the remaining identity fields default to "unversioned". Prompt — and, by default, context — still participate in the per-item compatibility fingerprint, so a changed prompt or a changed model never silently replays a stale result.

Automatic identity covers the provider and the model. OpenAIModel adds its API surface when it is the Responses API, and DeepSeekStrategy adds its API surface and strict response schema. Sampling and request parameters (temperature, reasoning effort, extra_body, and other request kwargs) are not part of identity or the item fingerprint, so changing them does not invalidate replay. For example, v0.27 changed the default temperature from 0.0 to None (the provider default); a v0.26 checkpoint resumed with REUSE_SUCCESSES replays successes produced at temperature=0.0 without warning, except for OpenAIModel on the new Responses default, whose identity changed. When a request parameter must invalidate replay, pass an explicit ArtifactIdentity that records it (for example extra={"temperature": 0.0}) or bump prompt_version.

Automatic identity is homogeneous within one live store instance/run. The first prepared item pins its inferred identity; every later item is inferred and must match before artifact lookup, append, or provider work begins. Use separate stores for heterogeneous strategies, or pass one deliberate explicit ArtifactIdentity that describes the mixed/routed run. A later store instance may pin a different automatic identity and append to the same physical artifact—older identities remain audit history and only compatible records participate in replay.

CallableStrategy is intentionally stricter: an arbitrary function cannot safely reveal its provider, model, route, parser, or application version. Pass identity=ArtifactIdentity(...) to CallableStrategy or directly to JsonlArtifactStore. Omitting both fails before the invocation callback runs; ABL never derives identity from a lambda, closure, object ID, memory address, or unstable repr(). An explicit store identity takes precedence over the strategy's identity.

When to use the full identity: versioned production pipelines. An explicit ArtifactIdentity lets a prompt-template change, parser change, or application release invalidate replay even when the literal prompt text is unchanged (e.g. context assembled outside the prompt, or a new parser reading the same outputs):

from async_batch_llm import (
    ArtifactIdentity,
    JsonlArtifactStore,
    ResumePolicy,
    process_prompts,
)

store = JsonlArtifactStore(
    "runs/customer-tagging.jsonl",
    identity=ArtifactIdentity(
        provider="openai",
        model="example-model",
        prompt_version="v3",
        parser_version="v2",
        application_version="2026.07",
    ),
)

result = await process_prompts(
    strategy,
    prompts,
    artifact_store=store,
    resume=ResumePolicy.REUSE_SUCCESSES,
)

An explicit identity is the caller-owned compatibility boundary. It may intentionally describe a mixed-strategy run, but the caller must change it whenever routing, provider, model, parser, prompt policy, or application semantics should invalidate replay.

A terminal record is flushed before its result is returned or yielded. Set fsync=True for an operating-system durability barrier after every record; flush-only is the default. Readers ignore every final line without a newline, even if it contains complete JSON, and never modify the file during inspection. Writable preparation validates retained records, truncates the incomplete tail, and builds replay state only from retained records. A dropped valid item is executed again; its zero-based stored sequence may be reused, remaining unique among retained records. Warnings report dropped bytes, a SHA-256 and the item ID when available, without logging raw payloads. Repair is also fsynced when enabled.

On macOS, fsync() flushes host buffers to the drive but does not force the drive to write its cache to stable storage (that needs F_FULLFSYNC, which the store does not issue). fsync=True there does not guarantee against data loss from an OS crash or power loss.

A file containing only a recognizable canonical manifest prefix can be repaired and initialized again. Unrecognized files, unsupported versions, and malformed complete or middle records fail without truncation. After any write failure, the JSONL instance rejects later appends; close it before opening a new instance for recovery. JSONL still supports only one writer per file. Cross-process writer locking remains outside this release; do not resume a file another process is currently writing.

Compatibility matching

Replay requires all of the following to match:

item ID;
SHA-256 prompt fingerprint;
context fingerprint when context_in_identity=True (the default);
combined input fingerprint; and
complete ArtifactIdentity fingerprint and supported artifact schema.

Matching only an item ID is never sufficient. Changing provider, model, prompt/parser/application version, identity extra, prompt, or participating context invalidates the old record. When several records match, the newest complete record wins.

Context is canonically JSON-encoded before a provider call. Supply encoder= or context_fingerprinter= when an application context is not supported; the store raises ArtifactSerializationError rather than silently excluding that identity component.

Sensitive structured values are redacted from persisted identity, context, output, and metadata mappings. Their original values still feed the one-way context/identity fingerprint, so a credential change invalidates replay without writing the credential itself to the artifact.

Compatibility across releases

The test suite keeps artifacts and serialized results written by published releases readable. tests/test_legacy_fixtures.py checks that files written by v0.18, v0.21, v0.24.1 and v0.26 (JSONL and, from v0.21, SQLite artifacts, plus BatchResult JSON and JSONL) still load, replay, and accept new records.

Two caveats apply:

Replay exclusions apply retroactively. A newer release that stops replaying a category (for example the configuration failures above) also ignores older records carrying it, and re-executes those items.
Forward compatibility is not guaranteed. An older release on the same schema version can reject a value that a newer release added. For example, a BatchResult serialized with termination.kind == "budget_exceeded" (v0.27) can't be read by v0.26, although RESULT_SCHEMA_VERSION is still 1.

Resume policies

ResumePolicy.NONE never reuses old results but still checkpoints new ones.
ResumePolicy.REUSE_SUCCESSES reuses the newest compatible success and reruns failures, missing items, and stale items.
ResumePolicy.REUSE_ALL also reuses the newest compatible terminal failure.

Replayed items do not call the provider and are not appended a second time. They retain historical output, timing, error, and token use, receive the current run's submission_index and current work-item context, and set replayed_from_artifact=True. Persisted historical context is never decoded or substituted during replay. Historical tokens remain visible on the item for audit but are excluded from newly consumed provider-token statistics returned by processor.get_stats(). In contrast, BatchResult aggregate token fields are computed from all returned results and therefore include replayed historical usage. This makes live processor stats a "spent this run" view and the collected batch an auditable result-history view.

For a callable, compatible replay likewise bypasses the invocation callback. Changing its explicit identity invalidates replay and executes it again.

When a billed response fails parsing or validation, raise TokenTrackingError(token_usage=...) from the callback. That attempt's usage is retained and added to later successful or terminal usage. ABL cannot account for transport retries hidden inside an upstream gateway unless the gateway reports them.

Setting include_output=False makes a success record audit-only and therefore ineligible for replay. An ordinary provider failure remains reusable under REUSE_ALL because it does not need a successful output. Guardrail audit, filtered, and artifact serialization failures are never replayed, and neither are configuration failures (see Middleware and replay).

A JSONL iterator started before the same store's first writable preparation can overlap tail truncation and subsequent appends. In that narrow case, it may see new records within its original byte range. Finish inspection before starting writable use if a strict snapshot is required.

Item-local artifact serialization failures

Built-in JSONL and SQLite stores validate and UTF-8 encode records before writing. An invalid output, metadata, persisted context, or cost-calculator result makes that item fail with artifact_serialization_error while neighboring items finish. The returned result retains its output for salvage and its token usage and timing. The checkpoint is a minimal failure record: output, metadata, raw prompt, raw context, and calculated cost are omitted. It cannot replay even under REUSE_ALL, so fixing an encoder or persistence configuration allows the item to run again.

Error messages identify the failing JSON path where applicable. Lone surrogates are rejected in application data. Stored diagnostic error and exception text escapes them as \ud800-style sequences; in-memory error text is preserved. Canonical encoding and fingerprints for valid Unicode do not change.

Serialization failures do not invoke middleware on_error or retry the provider within the same run. abort_on_error_categories={"artifact_serialization_error"} can stop a run after that item's failure checkpoint. Existing guardrail audit results keep their original category if their payload needs the same minimal projection. Only batch-abort, batch-budget, and batch-deadline audit writes remain best-effort; a failed total-item-deadline checkpoint remains fatal. I/O errors, invalid artifact format, unprepared store keys, custom-store failures, and failure to serialize or write even the minimal checkpoint still propagate under those same rules.

Subclassing the built-in stores is not a supported extension interface. In particular, overriding append does not intercept their private minimal-failure write path. Implement the public ArtifactStore protocol for custom write or redaction policies.

A provider success event may already have reached lifecycle observers before a checkpoint serialization failure is discovered. Until terminal-event handling is unified, observer success counts can differ from final result and processor counts on this path. No second terminal event is emitted.

Inspection decode-error messages identify one-based record positions on both backends. JSONL's stored record_sequence still starts at zero; SQLite's starts at one. This changes diagnostics only, not stored keys or replay ordering.

Privacy controls

Prompt and context hashes are always stored for matching, but raw prompts and raw context are excluded by default. Raw provider responses are never stored. The independent options are:

include_output=True (default);
include_metadata=True (default);
include_prompt=False (default); and
include_context=False (default).

Outputs and metadata can themselves contain sensitive application data. Review their shape before enabling artifacts, or disable either field. Raw prompt and context persistence must be explicitly enabled.

Generate, review, apply

Artifacts support a provider-free review/apply phase:

from async_batch_llm import JsonlArtifactStore

review = JsonlArtifactStore.read_results(
    "runs/customer-tagging.jsonl",
    successes_only=True,
)
for item in review.results:
    await apply_approved_output(item.item_id, item.output)

Inspection restores raw context only when the writer explicitly used include_context=True. Without a context_decoder, the restored value remains JSON-native; a supplied decoder is applied only when stored context exists. Context omitted by the default privacy policy, including an original None, inspects as None.

Replay never substitutes that historical audit context for the live input and does not invoke the stored-context decoder. It always attaches the current work item's context and submission index to the replayed result.

For asynchronous inspection through an open store, use async for item in store.iter_results(successes_only=True).

Costs are optional and caller-supplied because this package does not maintain a provider price database:

store = JsonlArtifactStore(
    "run.jsonl",
    identity=identity,
    cost_calculator=lambda item: calculate_cost_from_current_prices(item),
)

The artifact stores null when there is no calculator.

Indexed SQLite artifacts for large restartable runs

SqliteArtifactStore (v0.21) stores the same logical version-1 records in a standard-library SQLite database with indexed replay lookup and bounded iteration. It exists for 100k–1M-item restartable runs where scanning or retaining a JSONL history on open becomes the bottleneck; JSONL remains the recommended portable, human-inspectable audit format.

from async_batch_llm import ResumePolicy, SqliteArtifactStore, process_stream

store = SqliteArtifactStore("runs/customer-tagging.sqlite")
async for item in process_stream(
    strategy,
    prompts,
    artifact_store=store,
    resume=ResumePolicy.REUSE_SUCCESSES,
):
    await sink(item)

Everything documented above for JSONL applies unchanged: identity resolution and explicit ArtifactIdentity, the complete compatibility fingerprint, the three resume policies, replay accounting, privacy controls (include_output / include_metadata / include_prompt / include_context), custom encoders/decoders and context_fingerprinter, and optional cost_calculator. The two backends share one codec, so a record that would replay from JSONL replays from SQLite under the same conditions. Context-free items match through a NULL-safe indexed predicate. There is no automatic JSONL→SQLite import; opt in by creating a new store, and existing JSONL files remain fully supported.

Backend	Best fit	Reopen behavior	Write behavior	Human inspection
JSONL	Small/medium runs, portable audit log	Resume scans/indexes complete history; `iter_results()` streams line pages	One flushed line per record	Excellent
SQLite	100k–1M restartable runs	Indexed replay and bounded keyset iteration — no history decode on open	Batched transactions with WAL checkpoints	Use SQLite tools/API

“100k+” is a recommendation, not a cliff — the crossover depends on record size and environment.

Batched commits and durability

Appends are grouped into transactions (commit_batch_size, default 100, within commit_interval_seconds, default 0.01). An append() — and therefore the processor's publication of that item's result — completes only after its transaction commits, preserving checkpoint-before-result-publication. A transaction failure rolls back without exposing partial rows.

SqliteDurability selects the synchronization policy:

BALANCED (default) — WAL with synchronous=NORMAL. A normal process crash retains a consistent database and committed history. A sudden OS or power failure may lose transactions committed since the last WAL sync; the loss window can span more than one transaction.
FULL — WAL with synchronous=FULL; stronger power-loss durability at lower throughput. Exact guarantees always depend on SQLite, the OS, filesystem, and storage hardware. On macOS, SQLite syncs with plain fsync() unless its fullfsync pragma is enabled (off by default, and the store does not set it), so neither policy guarantees power-loss durability there.

WAL auto-checkpointing keeps the log at a bounded plateau during healthy writes; close() drains accepted appends, attempts a truncating checkpoint, and terminates the store's worker thread. An external long-lived reader can make that final checkpoint report “busy” — recorded, not an error and not data loss. SQLite may leave an -shm sidecar after a healthy close, and the database directory will transiently contain -wal/-shm files during a run. ABL does not encrypt SQLite artifacts; use filesystem permissions and volume encryption as your deployment requires.

Cleanup finishes before close() reports an error. A previously unobserved writer or cancelled-caller operation error is delivered first; a distinct checkpoint or connection-close error is retained for the next close(). Each store-level error is delivered once, and later closes return silently. Observing an error never restores a failed store to usability.

Inspection

Inspecting an existing SQLite store before its first run does not pin its model identity. Its first prepared item registers the resolved identity before appending, even if the connection was already opened by inspection. Automatic stores still reject mixed effective identities within one run.

Live store.iter_results() first settles detached errors, prepares the writable store, and flushes its accepted writes. It then streams a finite committed high-water snapshot in keyset-paged batches (read_batch_size), decoding one row at a time. No cursor or read transaction is held across an async yield, so a slow consumer never blocks the writer or WAL checkpointing.

SqliteArtifactStore.read_results(path) is a separate path-based, materializing convenience. It opens an internal SQLite mode=ro connection, never creates a schema or identity, never starts a writer, never changes WAL or synchronization policy, and never checkpoints an active writer. Normal reads participate in SQLite locking and change detection, allowing a writer to start before or during inspection; SQLite may create empty -wal/-shm coordination sidecars in a writable directory. A clean database in a non-writable directory uses SQLite's immutable fallback, under the filesystem guarantee that no writer can start while the read is active. The method captures a committed high-water sequence and excludes rows committed later; a subsequent read sees them. Its connection and owned reader thread close after success, error, or ordinary cancellation. Repeated cancellation remains prompt and lets the already-running reader thread clean itself up without synchronously blocking the event loop. Unlike the synchronous JSONL counterpart, it is an async classmethod because SQLite I/O stays off the event loop:

from async_batch_llm import SqliteArtifactStore

review = await SqliteArtifactStore.read_results(
    "runs/customer-tagging.sqlite",
    successes_only=True,
)

Opening a writable store validates schema and version markers, records the current run's distinct identity when needed (a sequential identity history for provenance), and never decodes stored history before a lookup needs it. The read-only path validates the same application ID, versions, tables, columns, required index names, and manifest without recording an identity. Each item row consumed by either path is checked against the supported logical artifact schema before its result or persisted context JSON is decoded; unsupported or malformed row versions fail loudly instead of falling back to an older compatible row.

Process-safety boundary

Both stores are single-process, single-writer designs. JsonlArtifactStore serializes writes with an asyncio.Lock, guaranteeing complete, non-interleaved JSON records for concurrent workers sharing one store instance in one process. SqliteArtifactStore owns one connection, one worker thread, and one writer task per instance; sequential reopen is supported and a concurrently open external reader obeys the configured busy timeout. Neither implements distributed work claiming, leasing, or cross-process exactly-once execution — use one writer per artifact.

On open, the JSONL store still builds an in-memory replay index over complete item records for constant-time resume lookup (its iter_results() streams bounded line pages instead of materializing history, as of v0.21). Very large audit histories are exactly what SqliteArtifactStore is for.