Batch inference needs workflow semantics, not bulk HTTP

Batch inference is a durable workflow, not a large HTTP request. The moment submission and completion are separated by hours, the system inherits identity, state, timeout, cancellation, reconciliation, and recovery obligations. Treating a batch API as “send JSONL, poll until done, download JSONL” hides the architecture precisely where production failures accumulate.

The first obligation is stable item identity. Results may arrive in arbitrary order, some requests may succeed while others error, expire, or are canceled, and a retried batch may overlap an earlier attempt. A client-generated identifier is therefore not convenience metadata; it is the join key between source work, provider execution, returned output, billing, and downstream effects. I want it derived from durable business identity and an attempt number, stored before submission, and unique enough that reconciliation never depends on line position.

The second obligation is an explicit lifecycle. Providers expose states such as validating, in progress, finalizing, completed, failed, expiring, and canceling because a batch is already a state machine. The application should own its corresponding states and transitions rather than collapsing everything into submitted and done. Validation failure may require repair. Partial expiry may require selective resubmission. Cancellation requested is different from cancellation confirmed. Completion means outputs are available, not that they have been streamed, validated, joined to source records, or committed downstream.

This is why checkpoints and replay semantics matter even when no autonomous agent is choosing the next step. A worker can die after downloading results but before recording them; a parser can commit half a file; a retry can encounter the provider’s original job after the local timeout fired. Each boundary needs an idempotent commit and enough recorded evidence to resume from the last confirmed transition. Streaming result files rather than loading them wholesale also makes backpressure and progress observable instead of turning a large response into one memory spike.

The economic promise reinforces the workflow shape. Asynchronous capacity can cost materially less than real-time inference, and prompt caching can reduce repeated-prefix cost further. But cost is an architectural property: savings vanish when expired items are blindly resubmitted, cache prefixes drift between jobs, or failed records trigger whole-batch reruns. Cost per accepted output should include validation failures, expired work, reconciliation, and downstream rejects—not just the provider’s discounted token rate.

Compliance belongs in admission too. Some batch services retain inputs and outputs differently from real-time endpoints or do not offer the same zero-data-retention terms. Workspace-level visibility may also be broader than the submitting credential suggests. A scheduler must select batch mode only after data classification, residency, retention, and deadline policies pass. Cheap capacity is not permission to change the handling contract.

There is one precise concession: a small, disposable offline experiment can use a thin upload-and-download script when every item is safe to rerun, no downstream effect occurs, and loss is cheaper than orchestration. That boundary excludes recurring evaluations, dataset generation, customer content, regulated inputs, and any pipeline whose outputs become production data. Once another system depends on the result, the script has become an undocumented workflow engine.

I would design batch inference with the same discipline as any long-running job: durable identities, explicit state transitions, deadline-aware scheduling, idempotent reconciliation, typed terminal outcomes, and an auditable connection from source record to accepted result. Deterministic workflows are cheap where paths can be enumerated, and batch inference is exactly such a path. Agents are workloads; batches are workloads too. Bulk transport changes the price and latency envelope. It does not repeal workflow semantics.