Articles for people who
ship the stack
Rewrites on React, Node.js, TypeScript and AI. A translated body is shown when it exists. If it does not, you see the English text and a note that says so.
Tagged: llm-evaluation
Why Agent Accuracy Scores Mislead: pass^k, Perturbations and Fault Injection
Why pass@1 hides unreliable AI agents, and how pass^k, input perturbation sets and fault injection reveal consistency, stability and fault-tolerance gaps.
9 min readDiagnosing RAG Failures: A Stage Map and a Claim-Level Groundedness Judge
Localize where a RAG pipeline broke, then build an LLM groundedness judge that checks claims one by one and is validated against human raters with kappa.
9 min readAgentic Finetuning: Training a Knowledge Wiki Instead of Model Weights
Run the ML training loop on a wiki of learnings mined from company documents: benchmarks built from raw data, secret questions, and a certificate with honest limits.
9 min readRAG or Knowledge Graph? 2,160 Graded Answers and a Model Effect
RAG vs knowledge graph, measured: 2,160 graded answers across three model families and 120 questions. The winner flips by model, and the model matters most.
9 min readTesting a Rubric Gate That Retries an Agent's Wrong Final Action
A 30-case experiment: an LLM judge plus one retry cut wrong support actions from 6 to 2, and why detecting a violation doesn't always mean the agent fixes it.
7 min readFrom Evals to Self-Improving Agents: Five Eras of Agent Evaluation
Agent evaluation now means auditing traces. See how deterministic checks, LLM and agent judges, diagnosis and a meta-harness build a safe improvement loop.
7 min readAgent Evaluation and Optimisation: DSPy, TextGrad, Reflexion and OPRO Compared
How DSPy, TextGrad, Reflexion and OPRO fit together as feedback loops for agents, which to adopt first, and how to separate evaluation from optimisation in an enterprise.
12 min readEval-Driven Development for AI Agents: Contracts, Traces and Judges
Treat evals as CI/CD for agents: version prompts as contracts, trace runs with OpenTelemetry and Tempo, then test with deterministic checks and an LLM judge.
13 min readClosed-Loop Evals for a Production Image Agent: What Uber Eats Presented
A case study of the evaluation design behind Uber Eats' photo enhancement agent: router recall, pass@K, drift diagnosis, autotuning and layered QA gates.
7 min read
About these articles
Request a 24h estimate
Need the same stack in a production operator layer? Send the brief — estimate within 24 hours.