# HyperAgility

> What 571 research papers found about building LLM agents: which methods win, on which tasks and models, what they cost and how they fail.

HyperAgility lines up the results from 571 research papers on building LLM agents, so you can see which techniques beat the alternatives, on which tasks and models, what they cost, and how they fail. Data as of 2026-11-09.

## How to read these results

- Won means a method scored better than whatever that paper compared it against, on the same model, task and metric. Each paper counts once. Results within half a percent count as tied.
- Typical change is the median relative change in score across papers.
- Independent tests exclude papers introducing that method. Most published results come from papers testing their own new technique, so compare both columns.
- Many results rest on few papers. Treat anything under 5 papers as a lead, and check the source paper before relying on a specific result.

## Methods, best supported first

| Method | Won (all results) | Won (independent tests) | Won (same budget) | Typical change |
|---|---|---|---|---|
| Harness search & evolution | 18 of 20 | 5 of 6 | 1 of 2 | +14% |
| Training data & environment synthesis | 16 of 20 | 6 of 7 | none | +14% |
| World-model simulation & lookahead | 8 of 9 | 1 of 3 | none | +15% |
| Reinforcement learning (verifiable / judged rewards) | 65 of 99 | 21 of 31 | 3 of 7 | +8% |
| Skill / playbook self-evolution | 19 of 26 | 10 of 13 | 5 of 6 | +6% |
| Parallel sampling & selection (Best-of-N, voting) | 47 of 72 | 33 of 43 | 8 of 15 | +2% |
| Tree search over actions or solutions | 22 of 32 | 15 of 18 | 2 of 5 | +5% |
| Trained reward / process reward models | 12 of 16 | 6 of 6 | 2 of 3 | +5% |
| Trained memory / context managers | 12 of 16 | 6 of 9 | 2 of 2 | +23% |
| Automated agentic workflow / MAS search | 18 of 26 | 12 of 15 | 2 of 2 | +6% |
| Skills, playbooks & rule files | 28 of 44 | 14 of 20 | 7 of 10 | +5% |
| Experience & lesson memory | 25 of 42 | 18 of 28 | 3 of 3 | +4% |
| Runtime safety guardrails | 5 of 6 | 2 of 3 | none | +29% |
| Hand-written instructions, policies & personas | 31 of 55 | 18 of 33 | 18 of 25 | +3% |
| Supervised fine-tuning on agent trajectories | 64 of 124 | 30 of 49 | 5 of 9 | +2% |
| Context compaction & history management | 23 of 40 | 12 of 20 | 2 of 4 | +2% |
| Structured indexes & knowledge navigation | 10 of 15 | 1 of 4 | 4 of 4 | +19% |
| Role-specialised multi-agent pipeline | 40 of 76 | 14 of 27 | 5 of 9 | +2% |
| Runtime control policies (gates, reminders, budgets) | 18 of 31 | 4 of 10 | 4 of 7 | +4% |
| Automated prompt optimisation | 11 of 18 | 7 of 9 | 0 of 1 | +3% |
| Communication topology & protocol design | 13 of 23 | 6 of 12 | 3 of 6 | +3% |
| Action space & tool-surface design | 22 of 43 | 14 of 22 | 10 of 13 | +1% |
| Standard ReAct-style tool loop | 42 of 97 | 29 of 53 | 7 of 20 | -2% |
| Heterogeneous model composition & routing | 19 of 39 | 8 of 14 | 0 of 1 | +0% |
| Self-reflection & iterative self-refinement | 27 of 59 | 20 of 35 | 4 of 11 | +0% |
| External critic / verifier feedback | 18 of 37 | 11 of 19 | 2 of 6 | +0% |
| Orchestrator with dynamic sub-agents | 17 of 35 | 12 of 22 | 2 of 6 | +0% |
| Chain-of-thought & reasoning elicitation | 30 of 71 | 22 of 44 | 4 of 9 | +0% |
| Plan-then-execute & task decomposition | 21 of 47 | 9 of 20 | 1 of 6 | +0% |
| Persistent external memory stores | 14 of 31 | 8 of 20 | 0 of 5 | -3% |
| Multi-agent debate & consensus | 11 of 24 | 11 of 16 | 1 of 2 | -1% |
| Code-as-action / programmatic tool calling | 13 of 31 | 11 of 16 | 0 of 4 | -3% |
| Observation representation & perception tools | 18 of 46 | 11 of 21 | 0 of 7 | -2% |
| Fixed few-shot demonstrations | 8 of 17 | 5 of 9 | 3 of 5 | +0% |
| Adaptive re-planning | 4 of 7 | 1 of 2 | 1 of 1 | +10% |
| Single-shot retrieval-augmented generation | 13 of 35 | 13 of 23 | 1 of 2 | -8% |
| Agentic search / retrieval as a tool | 9 of 23 | 2 of 10 | 3 of 5 | -6% |
| Production / open coding-agent harness | 21 of 74 | 14 of 31 | 3 of 13 | -5% |
| Tool / skill retrieval & selection | 4 of 9 | 3 of 6 | 3 of 4 | +0% |
| Preference optimisation | 5 of 14 | 4 of 6 | 0 of 1 | -1% |
| Model-level efficiency & parameter editing | 3 of 7 | 2 of 3 | 1 of 1 | +0% |
| Direct / zero-shot prompting (no scaffold) | 9 of 62 | 7 of 25 | 1 of 2 | -17% |

## Approach families

| Approach family | Won (all results) | Won (independent tests) | Won (same budget) | Typical change |
|---|---|---|---|---|
| Workflow optimisation | 62 of 79 | 31 of 38 | 7 of 9 | +7% |
| Training & adaptation | 114 of 161 | 47 of 62 | 4 of 8 | +15% |
| Memory & context | 91 of 142 | 48 of 81 | 13 of 22 | +5% |
| Verification & selection | 88 of 138 | 60 of 80 | 13 of 25 | +3% |
| Planning & search | 49 of 81 | 24 of 38 | 4 of 11 | +6% |
| Multi-agent | 79 of 144 | 39 of 68 | 10 of 19 | +2% |
| Single-agent harness | 87 of 200 | 56 of 98 | 15 of 39 | +0% |
| Retrieval | 31 of 64 | 15 of 33 | 9 of 12 | +0% |
| Prompting | 53 of 149 | 41 of 83 | 20 of 30 | -2% |

## Model product lines, within each maker

Each line against the same maker's other lines, only in experiments that ran both. Newer models tend to beat older ones.

### OpenAI

| Product line | Scored higher | Typical change |
|---|---|---|
| GPT-5 | 48 of 54 | +20% |
| o1 | 8 of 11 | +14% |
| GPT-4o | 24 of 48 | +1% |
| GPT-4 | 9 of 15 | +5% |
| GPT-5 Codex | 2 of 2 | +13% |
| GPT Sol | 4 of 6 | +5% |
| GPT Terra | 2 of 3 | +11% |
| GPT Astra | 1 of 1 | +8% |
| GPT-4.5 | 1 of 1 | +5% |
| o3 | 5 of 12 | +0% |
| GPT-4.1 | 6 of 16 | -10% |
| o4 mini | 3 of 9 | -19% |
| GPT-4.1 nano | 1 of 2 | -1% |
| GPT-4o mini | 5 of 25 | -11% |
| GPT-5 mini | 6 of 33 | -8% |
| o3 mini | 2 of 8 | -1% |
| GPT Luna | 1 of 4 | -4% |
| GPT-OSS | 1 of 5 | -25% |
| GPT-4.1 mini | 1 of 6 | -32% |
| GPT-3.5 | 2 of 19 | -16% |
| GPT-5 nano | 1 of 8 | -13% |
| o1 mini | 0 of 5 | -3% |
| GPT-5 Pro | 0 of 1 | -48% |

### Anthropic

| Product line | Scored higher | Typical change |
|---|---|---|
| Claude Opus | 28 of 42 | +5% |
| Claude Sonnet | 23 of 47 | +0% |
| Claude Fable | 1 of 2 | +0% |
| Claude Haiku | 1 of 18 | -12% |

### Google

| Product line | Scored higher | Typical change |
|---|---|---|
| Gemini Pro | 24 of 33 | +21% |
| Gemini Flash-Lite | 4 of 8 | -9% |
| Gemini Flash | 10 of 30 | -11% |
| Gemma | 2 of 6 | -27% |

### Alibaba

| Product line | Scored higher | Typical change |
|---|---|---|
| Qwen Max | 3 of 3 | +57% |
| Qwen VL | 3 of 3 | +32% |
| Qwen Plus | 2 of 2 | +13% |
| Qwen 3 | 8 of 15 | +4% |
| Qwen Coder | 2 of 6 | -18% |
| Qwen 2 | 1 of 9 | -24% |
| Qwen Flash | 0 of 2 | -8% |

### DeepSeek

| Product line | Scored higher | Typical change |
|---|---|---|
| DeepSeek V4 Pro | 6 of 7 | +12% |
| DeepSeek V3 | 5 of 11 | +0% |
| DeepSeek R1 | 4 of 10 | -2% |
| DeepSeek V4 Flash | 1 of 5 | -11% |
| DeepSeek R1 Distill | 0 of 1 | -64% |

### Z.ai

| Product line | Scored higher | Typical change |
|---|---|---|
| GLM 5 | 2 of 2 | +7% |
| GLM 4 | 1 of 2 | -1% |
| GLM Flash | 0 of 1 | -9% |
| GLM Air | 0 of 1 | -3% |

### xAI

| Product line | Scored higher | Typical change |
|---|---|---|
| Grok 4 | 2 of 2 | +12% |
| Grok 3 | 1 of 1 | +1% |
| Grok Fast | 0 of 2 | -8% |
| Grok mini | 0 of 1 | -1% |

### Meta

| Product line | Scored higher | Typical change |
|---|---|---|
| Llama 3 | 1 of 1 | +84% |
| Llama 4 | 0 of 1 | -46% |

## Failure modes

| Failure | Papers | Cases |
|---|---|---|
| Base-model capability ceiling | 208 | 297 |
| Over-action: tool overuse, over-exploration, over-engineering | 129 | 166 |
| Scaffold exceeds weak model's capacity | 129 | 174 |
| Transfer brittleness across models, environments or configurations | 121 | 168 |
| Redundant scaffolding for strong models | 118 | 156 |
| Context overload / attention dilution | 91 | 117 |
| Loops, stalling and budget exhaustion | 76 | 89 |
| Misleading or mismatched retrieval | 74 | 129 |
| Error compounding across steps | 54 | 68 |
| Ineffective or harmful self-reflection / self-refinement | 53 | 69 |
| Verifier / critic false positives (false pass) | 52 | 63 |
| Lossy compression / summarization drops needed detail | 51 | 90 |
| Verifier false rejection / over-intervention | 49 | 74 |
| Overfitting of optimized prompts, harnesses or fine-tuning | 48 | 62 |
| Catastrophic forgetting / alignment tax | 48 | 77 |
| Premature termination / false completion | 44 | 55 |
| Over-decomposition / context fragmentation | 41 | 54 |
| Self-evolution drift and unvalidated edits | 36 | 67 |
| Coordination overhead and handoff failures | 36 | 84 |
| Context window exhaustion / truncation | 34 | 42 |
| Interface / format / protocol mismatch | 30 | 39 |
| Reward hacking / shortcut exploitation | 27 | 32 |
| Safety regressions and harmful side effects | 24 | 40 |
| Social dynamics: herding, persuasion, premature consensus, adversarial rhetoric | 13 | 33 |

## Costs

| Cost | Papers | Rated highest |
|---|---|---|
| Engineering complexity | 565 | Harness search & evolution |
| Training compute | 551 | Reinforcement learning (verifiable / judged rewards) |
| Extra model calls | 480 | Tree search over actions or solutions |
| Token overhead | 429 | Parallel sampling & selection (Best-of-N, voting) |
| Data requirements | 407 | Trained reward / process reward models |
| Variance or brittleness | 360 | Single-shot retrieval-augmented generation |
| Human effort | 340 | Trained reward / process reward models |
| Requires stronger model | 318 | Chain-of-thought & reasoning elicitation |
| Latency overhead | 278 | Parallel sampling & selection (Best-of-N, voting) |
| Offline setup cost | 259 | Trained reward / process reward models |
| Maintenance | 89 | Structured indexes & knowledge navigation |
| Cache and pricing sensitivity | 42 | Model-level efficiency & parameter editing |

## Data

- [Paper list (CSV)](https://hyperagility.org/papers.csv)
- [Full data set (gzipped JSON)](https://hyperagility.org/kg.json.gz)

## License and credit

Please cite the original paper when using a specific result or quote, and respect its license. Please credit HyperAgility with a link when reusing the rankings, categories, statistics or data: "Source: HyperAgility (hyperagility.org), based on results from the cited papers." The data is CC BY 4.0, except paper titles, abstracts and quotes, which belong to their authors. The code is MIT.