Docs
Introduction
InfraBench evaluates AI agents on realistic infrastructure operations — provisioning, diagnosis, and repair — across the full system stack. Scoring goes beyond binary pass/fail to cover durable state, invariants, cleanup, and risk.
Published at HotInfra '26 (co-located with ISCA '26). The project site and leaderboard live at infraben.ch.
Why InfraBench
Many agent benchmarks grade a patch against a frozen repository. Infrastructure work is different: agents act on live state, failures cross layer boundaries, and a green checkmark can hide a masked symptom or a degraded system.
InfraBench asks a harder question: when an agent operates a real system under a real failure, does it leave that system better, safe, and clean?
Stack layers
Tasks are situated across four layers:
| Layer | Scope | Examples |
|---|---|---|
| L1 | Hardware | IPMI power recovery, baseboard management |
| L2 | Local systems | Single-node misbehavior and recovery |
| L3 | Distributed systems | Cassandra hang, dead-node removal, NIC split-brain |
| L4 | User applications | Failures that surface to dependent services |
Twelve seed tasks span these layers and several execution backends (Docker, VM cluster, bare-metal cluster).
Beyond pass/fail
For each task we care about outcomes a single reward bit cannot capture:
- Durable state — what is true after the agent claims done
- Invariants — properties that must never break (e.g. no data loss)
- Cleanup — orphaned processes, half-applied config, latent blast radius
- Risk — whether the path was reversible and bounded
Paper and artifacts
- Paper (PDF): Beyond Pass/Fail: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk
- Leaderboard: /leaderboard
- Hugging Face: InfraBench org (dataset + org card)
- GitHub: XuanmiaoG/InfraBench (public task packages)
Evaluation runtime
The closed evaluation runtime (Syscraft) is not open-sourced yet. You can still contribute by authoring well-structured task packages (instruction, environment, verifier) that match the public layout. See Contribute and Submit a task.
When a public runner ships, these docs will grow installation and agent-harness sections. Until then, treat this site as the place for overview, leaderboard, and contribution guidance.
Source: CONTRIBUTING.md · Docs home · Introduction. What InfraBench evaluates, stack layers, and scoring beyond pass/fail.