LLM Architect

Purpose

Provides expert large language model system architecture for designing, deploying, and optimizing LLM applications at scale. Specializes in model selection, RAG (Retrieval Augmented Generation) pipelines, fine-tuning strategies, serving infrastructure, cost optimization, and safety guardrails for production LLM systems.

When to Use

Designing end-to-end LLM systems from requirements to production
Selecting models and serving infrastructure for specific use cases
Implementing RAG (Retrieval Augmented Generation) pipelines
Optimizing LLM costs while maintaining quality thresholds
Building safety guardrails and compliance mechanisms
Planning fine-tuning vs RAG vs prompt engineering strategies
Scaling LLM inference for high-throughput applications

Quick Start

Invoke this skill when:

Designing end-to-end LLM systems from requirements to production
Selecting models and serving infrastructure for specific use cases
Implementing RAG (Retrieval Augmented Generation) pipelines
Optimizing LLM costs while maintaining quality thresholds
Building safety guardrails and compliance mechanisms

Do NOT invoke when:

Simple API integration exists (use backend-developer instead)
Only prompt engineering needed without architecture decisions
Training foundation models from scratch (almost always wrong approach)
Generic ML tasks unrelated to language models (use ml-engineer)

Decision Framework

Model Selection Quick Guide

Requirement	Recommended Approach
Latency <100ms	Small fine-tuned model (7B quantized)
Latency <2s, budget unlimited	Claude 3 Opus / GPT-4
Latency <2s, domain-specific	Claude 3 Sonnet fine-tuned
Latency <2s, cost-sensitive	Claude 3 Haiku
Batch/async acceptable	Batch API, cheapest tier

RAG vs Fine-Tuning Decision Tree

Need to customize LLM behavior?
│
├─ Need domain-specific knowledge?
│  ├─ Knowledge changes frequently?
│  │  └─ RAG (Retrieval Augmented Generation)
│  └─ Knowledge is static?
│     └─ Fine-tuning OR RAG (test both)
│
├─ Need specific output format/style?
│  ├─ Can describe in prompt?
│  │  └─ Prompt engineering (try first)
│  └─ Format too complex for prompt?
│     └─ Fine-tuning
│
└─ Need latency <100ms?
   └─ Fine-tuned small model (7B-13B)

Architecture Pattern

[Client] → [API Gateway + Rate Limiting]
              ↓
         [Request Router]
          (Route by intent/complexity)
              ↓
    ┌────────┴────────┐
    ↓                 ↓
[Fast Model]    [Powerful Model]
(Haiku/Small)   (Sonnet/Large)
    ↓                 ↓
[Cache Layer] ← [Response Aggregator]
    ↓
[Logging & Monitoring]
    ↓
[Response to Client]

Core Workflow: Design LLM System

1. Requirements Gathering

Ask these questions:

Latency: What's the P95 response time requirement?
Scale: Expected requests/day and growth trajectory?
Accuracy: What's the minimum acceptable quality? (measurable metric)
Cost: Budget constraints? ($/request or $/month)
Data: Existing datasets for evaluation? Sensitivity level?
Compliance: Regulatory requirements? (HIPAA, GDPR, SOC2, etc.)

2. Model Selection

def select_model(requirements):
    if requirements.latency_p95 < 100:  # milliseconds
        if requirements.task_complexity == "simple":
            return "llama2-7b-finetune"
        else:
            return "mistral-7b-quantized"
    
    elif requirements.latency_p95 < 2000:
        if requirements.budget == "unlimited":
            return "claude-3-opus"
        elif requirements.domain_specific:
            return "claude-3-sonnet-finetuned"
        else:
            return "claude-3-haiku"
    
    else:  # Batch/async acceptable
        if requirements.accuracy_critical:
            return "gpt-4-with-ensemble"
        else:
            return "batch-api-cheapest-tier"

3. Prototype & Evaluate

# Run benchmark on eval dataset
python scripts/evaluate_model.py \
  --model claude-3-sonnet \
  --dataset data/eval_1000_examples.jsonl \
  --metrics accuracy,latency,cost

# Expected output:
# Accuracy: 94.3%
# P95 Latency: 1,245ms
# Cost per 1K requests: $2.15

4. Iteration Checklist

Latency P95 meets requirement? If no → optimize serving (quantization, caching)
Accuracy meets threshold? If no → improve prompts, fine-tune, or upgrade model
Cost within budget? If no → aggressive caching, smaller model routing, batching
Safety guardrails tested? If no → add content filters, PII detection
Monitoring dashboards live? If no → set up Prometheus + Grafana
Runbook documented? If no → document common failures and fixes

Cost Optimization Strategies

Strategy	Savings	When to Use
Semantic caching	40-80%	60%+ similar queries
Multi-model routing	30-50%	Mixed complexity queries
Prompt compression	10-20%	Long context inputs
Batching	20-40%	Async-tolerant workloads
Smaller model cascade	40-60%	Simple queries first

Safety Checklist

Content filtering tested against adversarial examples
PII detection and redaction validated
Prompt injection defenses in place
Output validation rules implemented
Audit logging configured for all requests
Compliance requirements documented and validated

Red Flags - When to Escalate

Observation	Action
Accuracy <80% after prompt iteration	Consider fine-tuning
Latency 2x requirement	Review infrastructure
Cost >2x budget	Aggressive caching/routing
Hallucination rate >5%	Add RAG or stronger guardrails
Safety bypass detected	Immediate security review

Quick Reference: Performance Targets

Metric	Target	Critical
P95 Latency	<2x requirement	<3x requirement
Accuracy	>90%	>80%
Cache Hit Rate	>60%	>40%
Error Rate	<1%	<5%
Cost/1K requests	Within budget	<150% budget

Additional Resources

Detailed Technical Reference: See REFERENCE.md
- RAG implementation workflow
- Semantic caching patterns
- Deployment configurations
Code Examples & Patterns: See EXAMPLES.md
- Anti-patterns (fine-tuning when prompting suffices, no fallback)
- Quality checklist for LLM systems
- Resilient LLM call patterns

llm-architect