InfraBench

A benchmark for infrastructure agents.

InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk

Twelve real operational incidents — from IPMI power recovery to silent data corruption — spanning hardware, local systems, distributed systems, and user applications. Beyond pass/fail: durable state, invariants, cleanup, and risk.

View leaderboard →arXiv →GithubGitHub →Contribute a task →

Task coverage by layer

L4 User Applications3 tasks
L3 Distributed Systems7 tasks
L2 Local Systems1 task
L1 Hardware1 task
12 tasks18 configs3 backends

01Leaderboard

Updated Aug 24, 2026

Mean effective score across 12 tasks. Every configuration runs each task three times, so all rows carry Mean ± SEM and Attempt Pass rates.

#AgentModelMeanPass@1Pass@0.5Best-of-N@1
1Claude CodeFable 589.2 ± 5.677.8% (28)91.7% (33)83.3% (10)
2Cursor CLIGrok 4.584.3 ± 7.975.0% (27)86.1% (31)83.3% (10)
3Claude CodeClaude Opus 583.1 ± 7.572.2% (26)86.1% (31)75.0% (9)
4Qoder CLIQwen3.8 Max81.4 ± 8.669.4% (25)77.8% (28)75.0% (9)
5Claude CodeClaude Opus 4.880.2 ± 7.563.9% (23)86.1% (31)75.0% (9)
6Qoder CLIKimi K377.3 ± 8.466.7% (24)77.8% (28)75.0% (9)
7Claude CodeClaude Sonnet 577.2 ± 7.463.9% (23)80.6% (29)75.0% (9)
8Gemini CLIGemini 3.6 Flash75.5 ± 9.566.7% (24)72.2% (26)75.0% (9)
9Qoder CLIGLM 5.274.0 ± 8.261.1% (22)77.8% (28)75.0% (9)
10Cursor CLIComposer 2.572.6 ± 9.358.3% (21)75.0% (27)66.7% (8)
View full leaderboard →

02Per-task results

■ pass■ partial□ fail

Full 12 × 18 score matrix across published model configurations.

Task
Anthropic
Fable 5
xAI
Grok 4.5
Opus 5
Alibaba
Qwen3.8 Max
Anthropic
Opus 4.8
Kimi K3
Sonnet 5
Google
Gemini 3.6 Flash
Zhipu
GLM 5.2
Cursor
Composer 2.5
Google
Gemini 3.5 Flash
Google
Gemini 3.1 Pro
Moonshot
Kimi K2.7 Code
DeepSeek
DS V4 Flash
Xiaomi
MiMo V2.5
Anthropic
Sonnet 4.6
Google
Gemini 3.1 Flash-Lite
DeepSeek
DS V4 Pro
IPMI Power Recovery
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Cassandra Dead Node
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
0.67
✓
✓
0.58
0.33
✓
✓
Cassandra NIC Split Brain
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
0.92
✓
0.92
✓
✓
Cassandra Hung Recovery
✓
✓
✓
✓
✓
0.67
0.67
0.67
✓
0.33
0.67
0.67
0.33
0.33
0.67
✓
✗
0.33
Cassandra CORDS
✓
✓
✓
✓
0.33
0.67
✓
✓
✓
✓
✓
✓
✓
✓
0.67
0.67
0.33
0.33
Ceph Pool Degraded
0.85
0.85
✓
0.69
0.69
0.85
0.54
✓
0.54
0.85
✓
0.54
0.69
0.39
0.54
0.54
0.50
0.54
Ceph Bootstrap
0.87
0.28
0.47
0.17
✓
0.11
0.74
0.09
0.35
0.42
0.15
0.29
0.54
0.03
0.29
0.24
0.03
0.03
Fileserver RAID10
✓
✓
0.67
✓
✓
✓
✓
✓
0.67
✓
✓
✓
✓
0.33
✗
✗
✗
✗
SLURM/Puppet Cascade
0.49
0.33
0.34
0.49
0.46
0.49
0.49
0.49
0.49
0.19
0.49
0.49
0.33
0.23
0.12
0.36
0.14
0.06
DB WAL Recovery
✓
✓
✓
✓
0.64
✓
0.33
0.33
0.33
0.43
✗
0.43
0.21
0.55
✗
✗
0.43
✗
PgBouncer Drift
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
0.67
✓
✓
0.67
0.67
0.67
Pelican Key Mismatch
0.50
0.67
0.50
0.41
0.50
0.50
0.50
0.48
0.50
0.50
0.35
0.44
0.50
0.50
0.50
0.59
0.44
0.50

Fractions show verifier checks passed. Full methodology is in the paper.

03Failure patterns

Seven recurring failure modes from verifier traces. The gray field is incidence-weighted severity across all published configs; several configurations are overlaid by default, and you can swap in any of the 18 to compare exposure.

Destructive Dx
72.2
Hidden Config
54.2
Side Effects
3.6
Non-Persistent
3.6
Cleanup Miss
40.0
Fix Regression
0.0
Deploy Residue
40.0

Radius = severity when exposed (critical 100 · high 65 · medium 40). Field = incidence-weighted severity across all 18 models. Overlay up to 5 configurations.

04Task taxonomy

Contribute a task →

12 tasks across four infrastructure layers and 3 execution backends.

TaskLayerBackendDifficultyCharacteristics
IPMI Power RecoveryL1 HardwareBM Cluster●○○Physical-level IPMI power cycling
Cassandra NIC Split BrainL2 Local SystemsBM Cluster●●○NIC-caused network partition
Cassandra Dead Node RemovalL3 Distributed SystemsBM Cluster●●○Dead node removal & ring repair
Cassandra Node Hung RecoveryL3 Distributed SystemsBM Cluster●●○Hung node recovery via IPMI/network
Cassandra CORDS PropagationL3 Distributed SystemsVM Cluster●●●Silent data corruption; adapted from CORDS (FAST '17)
Ceph Pool DegradedL3 Distributed SystemsVM Cluster●●●Degraded pool repair & CRUSH fix
Ceph BootstrapL3 Distributed SystemsVM Cluster●●●End-to-end Ceph cluster deployment
Fileserver RAID10 Silent DiskL3 Distributed SystemsVM Cluster●●●RAID10 silent disk latency injection
SLURM/Puppet Config CascadeL3 Distributed SystemsVM Cluster●●●Unattended-upgrades + Puppet config cascade
DB WAL RecoveryL4 User ApplicationsContainer●●●Encrypted WAL recovery; adapted from Terminal-Bench 2.0
PostgreSQL/PgBouncer DriftL4 User ApplicationsVM Cluster●●●Long-term persistent config drift
Pelican Namespace/Key MismatchL4 User ApplicationsVM Cluster●●●Real-world CHTC incident: JWKS/namespace mismatch