Should we use an LLM framework or call the API directly?

Direct API calls plus a thin internal abstraction, for most product work. Frameworks are useful for prototyping and become a liability in production, because when output is wrong the first thing you need is the exact payload that was sent — and that's precisely what the abstraction hides. Keep a swappable interface around the provider so a migration is a day's work, and stop there.

How do we handle it when the model is unavailable or slow?

Decide the degraded behaviour deliberately and build it at the same time as the feature. Write-time enrichment can simply retry. Retrieval answering can fall back to returning the search results without a generated summary, which is still useful. Assistive generation should fail visibly and leave the user's own text untouched. What should never happen is a page that hangs on a third-party timeout.

Is prompt injection actually solvable?

Not at the prompt layer. It's mitigated architecturally: least-privilege tools scoped to the calling user, human confirmation on irreversible actions, full logging, and never treating model output as a command your code executes. Assume it will occasionally succeed and make sure that when it does, the model wasn't holding anything dangerous.

How much does an LLM abstraction layer cost to build?

A day or two, and it should stay that small. What you want is a single function that takes a request, applies your retry and timeout policy, logs the payload, and returns typed output — with the provider behind an interface. Anything larger starts hiding the thing you'll need to see when something breaks.

LLM Integration Patterns for Existing Software

Most language model integrations fall into five architectural patterns: enrichment at write time, retrieval-augmented answering, structured extraction, a constrained tool-using agent, and assistive generation inside an existing interface. The pattern is chosen by two properties of the problem — whether the work can be done before the user asks, and whether the model needs to take actions or only produce text. Choosing the wrong one is the most expensive mistake available, because it is discovered late.

Almost every article about adding a language model to a product jumps straight to prompts. Prompting is the last decision and the cheapest to change. The architecture around the model is the expensive one, and it's usually chosen by accident — somebody builds a chat interface because chat is what a language model looks like, and eight weeks later discovers the problem never wanted a chat interface.

Five patterns cover most production work. Here's how to tell which one you have.

The two questions that pick the pattern

Before comparing patterns, answer these. They eliminate three of the five almost every time.

01Can the work happen before the user asks for it? If the input exists at write time — a document uploaded, a ticket created, a transcript finished — you can do the model work then. That changes everything: latency stops mattering, cost becomes predictable, and failures can be retried without anyone watching.
02Does the model need to act, or only to produce text? Producing text is a well-understood engineering problem. Taking actions in your systems is a permissions and blast-radius problem, and it is a different order of difficulty.

	Text only	Takes actions
Before the user asks	Write-time enrichment · Structured extraction	Rare — usually a scheduled job, not an agent
When the user asks	Retrieval-augmented answering · Assistive generation	Constrained agent

The two answers pick the pattern.

Pattern 1 — Enrichment at write time

When something enters your system, a background job runs the model over it and stores the result as ordinary columns. Tickets get a category and a sentiment. Uploaded documents get a summary and extracted entities. Transcripts get action items.

The user-facing feature then involves no model call at all. It's a database read, and it is instant, free at read time, and reliably available. This is the most underrated pattern in the list, and it's the right answer far more often than teams assume, because it doesn't *look* like AI while you're building it.

Cost: one model call per item, ever. Predictable and bounded by ingest volume, not by traffic.
Latency: zero at read time.
Failure handling: trivial. Retry the job. Nobody is waiting.
Watch for: re-processing when the prompt improves. Version the enrichment so you know which rows came from which prompt, or you'll be unable to tell whether a change helped.

Pattern 2 — Retrieval-augmented answering

The user asks a question; you search your own content, hand the model the best passages, and it answers with citations. The pattern everyone means when they say RAG.

The engineering is almost entirely in the retrieval, not the generation — chunking, hybrid keyword-and-vector search, re-ranking and permission filtering. Adding RAG to a product that already has customers covers the specifics.

Cost: per query, and dominated by how much context you pass. Trimming retrieved context is the biggest lever.
Latency: retrieval plus generation. Stream, or it will feel slow.
Failure handling: must include a clean refusal when the corpus doesn't contain the answer.
Watch for: tuning the prompt when the real problem is retrieval. Measure recall before touching wording.

Pattern 3 — Structured extraction

Unstructured input in, typed record out. Invoices to line items, CVs to candidate fields, emails to CRM records, contracts to key terms. The output is validated against a schema, so a malformed response fails loudly rather than propagating.

This is consistently the highest return on effort of anything in this list, and the least discussed, because it replaces manual data entry with something measurable. It's also the easiest to evaluate — you have a schema, so correctness is largely field-by-field comparison rather than a judgement call.

const InvoiceSchema = z.object({
  invoiceNumber: z.string(),
  issuedOn: z.iso.date(),
  currency: z.string().length(3),
  lineItems: z.array(
    z.object({ description: z.string(), amount: z.number() }),
  ),
});

const parsed = InvoiceSchema.safeParse(await extract(document));

if (!parsed.success) {
  // Defined behaviour: one retry with the validation errors fed back,
  // then a human review queue. Never a silent partial record.
  return queueForReview(document, parsed.error);
}

The schema is the contract. Parsing failures are a defined path, not an exception nobody catches.

Pattern 4 — The constrained agent

The model is given tools — your API, a database query, a third-party service — and the authority to call them in sequence until a task is done. This is the pattern with the most attention and the highest failure rate in production, and the difficulty is not the loop.

It's blast radius. Any agent that reads untrusted text — a web page, an uploaded file, an inbound email — can have its instructions overwritten by that text. Prompt injection is not fully solvable at the prompt layer, so the containment has to be architectural.

Least privilege per tool. The agent gets the narrowest possible capability, scoped to the calling user's own permissions. Not an admin key.
Irreversible actions need confirmation. Sending, paying, deleting and publishing get a human, or they get an undo window.
Every call logged with its inputs, so an incident can be reconstructed rather than guessed at.
Hard iteration limits. An agent that has looped fifteen times is not converging, and the loop is billable.
Never treat model output as a command. It's input to your code, and your code decides.

Pattern 5 — Assistive generation in place

Drafting, rewriting, summarising and translating, embedded where the user already works rather than in a chat panel bolted onto the corner of the screen. The model proposes and the human decides, which is what makes it forgiving: a wrong suggestion costs a click.

Stream always. The output is being read as it arrives.
Make it cancellable. Users abandon generations constantly, and an uncancelled generation is billed.
Never overwrite silently. Propose alongside; let the human accept.
Watch for: cost from abandoned generations. It's invisible in testing and material at scale.

Four antipatterns

In rough order of how much they cost when they surface.

01A chat interface for a non-conversational problem. Chat is the most expensive interface to build well and the hardest to evaluate, because there's no defined correct output. If the task is "categorise this" or "extract that", the interface is a field, not a conversation.
02Model calls in the request path that didn't need to be. If the input existed an hour ago, the work could have happened an hour ago. This single reframing removes most latency and cost problems.
03A framework abstraction thick enough to hide the prompt. When output is wrong you need to see exactly what was sent. Abstractions that obscure the final payload turn a ten-minute debug into an afternoon.
04No eval suite, so nothing can be changed. The system works, nobody dares touch it, and it slowly becomes a fossil. Evals and guardrails before you ship covers what to build instead.

Picking, in one paragraph

If the input arrives before the user needs the output, use enrichment at write time — it's cheaper, faster and more reliable than anything else here. If users ask questions of your content, it's retrieval. If you're turning documents into records, it's extraction, and you should probably do it first because it's the easiest to prove value with. If the model must act on your systems, it's an agent and the work is permissions rather than prompting. And if you're helping someone write, it's assistive generation, where the human staying in the loop is the feature and not a limitation.

Most real products end up with two or three of these. What they should not end up with is one chat box asked to be all five.

Five patterns for putting an LLM into software that already exists