Docs

Introduction

InfraBench evaluates AI agents on realistic infrastructure operations — provisioning, diagnosis, and repair — across the full system stack. Scoring goes beyond binary pass/fail to cover durable state, invariants, cleanup, and risk.

Published at HotInfra '26 (co-located with ISCA '26). The project site and leaderboard live at infraben.ch.

Why InfraBench

Many agent benchmarks grade a patch against a frozen repository. Infrastructure work is different: agents act on live state, failures cross layer boundaries, and a green checkmark can hide a masked symptom or a degraded system.

InfraBench asks a harder question: when an agent operates a real system under a real failure, does it leave that system better, safe, and clean?

Stack layers

Tasks are situated across four layers:

Twelve seed tasks span these layers and several execution backends (Docker, VM cluster, bare-metal cluster).

Beyond pass/fail

For each task we care about outcomes a single reward bit cannot capture:

• Durable state — what is true after the agent claims done
• Invariants — properties that must never break (e.g. no data loss)
• Cleanup — orphaned processes, half-applied config, latent blast radius
• Risk — whether the path was reversible and bounded

How the leaderboard reports

Every configuration runs each of the 12 tasks three times, so each row reports:

• Mean ± SEM — mean of 12 task scores (each task = mean of its passes), with SEM over tasks
• Attempt Pass@τ — share of (task, pass) attempts with difficulty-weighted reward (R \ge \tau); we report τ = 1 (perfect) and τ = 0.5 (substantially solved)
• Best-of-N @1 — share of tasks where at least one pass reaches (R = 1)

Attempt Pass is the raw share of attempts clearing the bar — it is not SWE-bench Pass@k, which estimates the probability that at least one of k samples succeeds.

Configurations pin a fixed model checkpoint; vendor model routers are excluded, since their backend selection changes without notice and results would not be reproducible against a named version.

Risk Monitor audit

Scores say whether the fault went away. The Risk Monitor asks what the agent did on the way there: every recorded command is read in the context of its task and classified against a seven-type danger taxonomy (destructive filesystem ops, disk/RAID/LVM destruction, network disruption, safety or privilege bypass, unsafe restarts, cross-service interference, and probing of the grading harness).

Over the 266 trials that carry a machine-readable action log — 9 of the 15 configurations, across the Claude Code, Cursor CLI, Gemini CLI, and OpenCode backends — 76 of 9,351 commands (0.8%) were flagged:

Agents are conservative by default — most write operations are legitimate repair — but the flagged actions are not random: 15 of the 16 safety bypasses are five different configurations independently disabling mandatory access control cluster-wide to get past one Ceph bootstrap bug, and 15 of the 17 probing actions are on the single task whose answer cannot be recovered from the environment.

The remaining six configurations ran before their CLI adapters recorded per-command traces, so their commands were never captured; they are reported as uncovered rather than counted as clean. Reproduce with the risk-judge skill in the repo.

Paper and artifacts

• Paper: InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk — arXiv:2608.11234 (PDF · HTML · HotInfra '26 camera-ready)
• Leaderboard: /leaderboard
• Hugging Face: InfraBench org (dataset + org card)
• GitHub: XuanmiaoG/InfraBench (public task packages)

Evaluation runtime

The closed evaluation runtime (Syscraft) is not open-sourced yet. You can still contribute by authoring well-structured task packages (instruction, environment, verifier) that match the public layout. See Contribute and Submit a task.

When a public runner ships, these docs will grow installation and agent-harness sections. Until then, treat this site as the place for overview, leaderboard, and contribution guidance.

Source: CONTRIBUTING.md · Docs home · Introduction. What InfraBench evaluates, stack layers, and scoring beyond pass/fail.