Prompt Engineering Patterns for Production Systems

Prompt Engineering Is Software Engineering

In prototypes, prompt engineering means experimenting in a playground until the output looks reasonable. In production, it means writing prompts with the same rigor as code: versioned, tested, evaluated, and iterated systematically.

The gap between a demo and a production system often comes down to prompt discipline. This guide covers patterns that work at scale.

Pattern 1: Structured Output Contracts

Never parse free-form LLM text in production code. Use JSON mode or structured output features to enforce a contract:

from openai import OpenAI
from pydantic import BaseModel
from typing import Optional

client = OpenAI()

class ExtractedOrder(BaseModel):
    customer_name: str
    product_ids: list[str]
    quantity: int
    delivery_date: Optional[str]
    notes: Optional[str]

response = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {
            "role": "system",
            "content": "Extract order details from customer messages. Be precise."
        },
        {
            "role": "user",
            "content": "I need 3 units of SKU-4421 and 1 of SKU-8832 delivered by Friday. — James"
        }
    ],
    response_format=ExtractedOrder,
)

order = response.choices[0].message.parsed
# order.customer_name == "James"
# order.product_ids == ["SKU-4421", "SKU-8832"]

client.beta.chat.completions.parse enforces the schema. You get a typed ExtractedOrder object or a parse error — never ambiguous text to pattern-match against.

For models without structured output, use JSON mode plus Pydantic validation:

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_object"},
    messages=[...]
)

raw = json.loads(response.choices[0].message.content)
order = ExtractedOrder.model_validate(raw)  # Raises ValidationError on bad output

Pattern 2: Chain-of-Thought with Scratchpads

For complex reasoning tasks, prompt the model to reason before answering. Crucially: keep the reasoning separate from the output, so downstream code only parses the final answer.

ANALYSIS_PROMPT = """
You are a credit risk analyst. Analyze the following application.

<application>
{application_data}
</application>

First, think through the key risk factors in a <reasoning> block.
Then provide your decision in a <decision> block with this exact JSON structure:
{
  "approved": boolean,
  "risk_score": number (0-100),
  "reason": string (max 100 chars)
}

Format:
<reasoning>
[Your detailed analysis here]
</reasoning>

<decision>
[JSON here]
</decision>
"""

def parse_credit_decision(response: str) -> dict:
    decision_match = re.search(r'<decision>(.*?)</decision>', response, re.DOTALL)
    if not decision_match:
        raise ValueError("No decision block found in response")
    return json.loads(decision_match.group(1).strip())

The reasoning block improves decision quality by forcing the model to consider evidence before concluding. You parse only the structured <decision> block.

Pattern 3: Few-Shot Examples as Documentation

Few-shot examples are the most reliable way to encode format requirements. They also document intent for the humans maintaining the prompt:

CLASSIFICATION_PROMPT = """
Classify customer support tickets into exactly one category.

Categories: billing | technical | account | shipping | other

Examples:
Input: "My invoice shows $99 but I should be on the $49 plan"
Output: billing

Input: "The app crashes when I try to upload files larger than 5MB"
Output: technical

Input: "I need to change the email on my account"
Output: account

Input: "My package was supposed to arrive yesterday, tracking shows it's still in transit"
Output: shipping

Input: "{ticket_text}"
Output:"""

Three to five examples outperforms two pages of instruction text. Pick examples that cover the ambiguous edge cases — the ones your system gets wrong in staging.

Pattern 4: Prompt Versioning and Registry

Treat prompts as code. Store them in a registry with version tracking:

# prompts/registry.py
PROMPTS = {
    "credit-analysis": {
        "v1": {
            "template": "...",
            "model": "gpt-4",
            "temperature": 0.1,
            "deployed_at": "2026-03-01",
        },
        "v2": {
            "template": "...",  # Added chain-of-thought
            "model": "gpt-4o",
            "temperature": 0.0,
            "deployed_at": "2026-05-15",
        }
    }
}

def get_prompt(name: str, version: str = "latest") -> dict:
    versions = PROMPTS[name]
    if version == "latest":
        return max(versions.values(), key=lambda v: v["deployed_at"])
    return versions[version]

When you change a prompt, increment the version. Keep old versions in the registry. This makes A/B testing and rollback trivial.

Pattern 5: Evaluation-Driven Iteration

Never ship a prompt change without measuring it. Build an eval suite:

# evals/credit_analysis_evals.py
TEST_CASES = [
    {
        "input": "Application data for John Smith...",
        "expected_approved": True,
        "expected_risk_range": (20, 40),
    },
    {
        "input": "Application data for suspicious LLC...",
        "expected_approved": False,
        "expected_risk_range": (75, 100),
    },
    # 50+ cases covering edge cases
]

def run_eval(prompt_version: str) -> dict:
    results = []
    for case in TEST_CASES:
        output = run_credit_analysis(case["input"], prompt_version)
        
        results.append({
            "correct_decision": output["approved"] == case["expected_approved"],
            "risk_in_range": case["expected_risk_range"][0] <= output["risk_score"] <= case["expected_risk_range"][1],
        })
    
    return {
        "decision_accuracy": sum(r["correct_decision"] for r in results) / len(results),
        "risk_calibration": sum(r["risk_in_range"] for r in results) / len(results),
    }

# Before deploying v3:
v2_score = run_eval("v2")  # {"decision_accuracy": 0.91, "risk_calibration": 0.87}
v3_score = run_eval("v3")  # {"decision_accuracy": 0.94, "risk_calibration": 0.90}
assert v3_score["decision_accuracy"] >= v2_score["decision_accuracy"]

This eval harness catches regressions before production. Build it early. Even 20 hand-labeled test cases are infinitely better than eyeballing outputs.

Pattern 6: Temperature and Sampling Parameters

# Deterministic classification, extraction, structured output
{"temperature": 0.0, "top_p": 1.0}

# Factual Q&A requiring some flexibility
{"temperature": 0.2, "top_p": 0.9}

# Creative writing, brainstorming
{"temperature": 0.8, "top_p": 0.95}

# Never use high temperature for structured output — it breaks JSON

Match temperature to task type. For any task where you need consistent, parseable output, use temperature 0.

Pattern 7: Context Window Management

Long prompts are expensive and sometimes counterproductive. Monitor prompt token usage:

import tiktoken

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

def build_prompt(context_docs: list[str], max_context_tokens: int = 4096) -> str:
    used_tokens = count_tokens(SYSTEM_PROMPT)
    selected_docs = []
    
    for doc in context_docs:
        doc_tokens = count_tokens(doc)
        if used_tokens + doc_tokens > max_context_tokens:
            break
        selected_docs.append(doc)
        used_tokens += doc_tokens
    
    return SYSTEM_PROMPT + "\n\nContext:\n" + "\n---\n".join(selected_docs)

Stay within budget. Exceeding the context window silently truncates (or raises an error, depending on the API). Neither outcome is what you want.

Production prompt engineering is a discipline of measurement: write prompts explicitly, evaluate them systematically, version every change, and treat regressions as bugs.