A workflow can be accurate in testing and still fail in production because a larger room, longer document, or richer search result pushes one request beyond its useful boundary.
Bound the unit of work
I prefer small, independently retryable batches over one enormous prompt. A room, a bounded group of line items, or a single document section provides a natural unit. If batch seven fails, the system retries batch seven—not the entire report.
Track the reason, not just the total
Total monthly tokens do not explain waste. Usage records should include the application, task type, model, claim or job, input tokens, output tokens, latency, retry count, and final status. That makes it possible to find expensive prompts and failures that repeatedly consume context.
Design the fallback explicitly
Retries need limits. Truncated JSON needs validation. A fallback value must never masquerade as a real result. If pricing fails, the UI should say pricing failed—not quietly write zero dollars. If evaluation is incomplete, the report should remain incomplete.
Optimize after observability
Once usage is attributable, optimization becomes concrete: reduce repeated reference material, cache stable descriptions, select only relevant examples, compress search results, or move deterministic transformations out of the model. Cost control is then an engineering exercise rather than guesswork.