Docs

Submit a task

This page is a docs-oriented guide to packaging and submitting an InfraBench task. Source of truth for the same material: CONTRIBUTING.md.

Task layout

my-task/
├── task.toml          # environment type, resources, timeout, metadata
├── instruction.md     # natural-language brief for the agent
├── environment/
│   └── Dockerfile     # docker tasks
│   # or setup.sh + bootstrap.sh for vm / vm-cluster
├── solution/
│   └── solve.sh       # reference solution (optional, used by oracle agents)
└── tests/
    └── test.sh        # verifier — writes reward to /logs/verifier/

Start from tasks/hello-world, then mirror a paper task at your target layer.

Verifier contract

Emit either a scalar reward:

echo 1.0 > /logs/verifier/reward.txt

or structured JSON with partial credit:

{
  "reward": 0.0,
  "checks": [
    {"name": "service health", "passed": true, "scored": false},
    {"name": "root cause fixed", "passed": false, "scored": true}
  ]
}

• scored: true — checks that measure meaningful repair progress (contribute to partial score).
• scored: false — diagnostics / anti-bypass / reachability (affect pass/fail, do not inflate partials).

task.toml templates

Docker

[task]
name = "myorg/my-task"

[metadata]
difficulty = "easy"          # easy | medium | hard
category = "infrastructure"

[environment]
type = "docker"
cpus = 2
memory_mb = 4096

[agent]
timeout_sec = 300

Single-node VM

[task]
name = "myorg/my-vm-task"

[environment]
type = "vm"
base_image = "ubuntu-24.04"
cpus = 2
memory_mb = 4096
storage_mb = 20480

[agent]
timeout_sec = 300

VM tasks use environment/setup.sh (cached package install) and optional bootstrap.sh (per-trial) instead of a Dockerfile.

VM cluster

[task]
name = "myorg/my-cluster-task"

[environment]
type = "vm-cluster"
network = "192.168.100.0/24"

[[environment.nodes]]
name = "primary"
cpus = 2
memory_mb = 4096
storage_mb = 20480

[[environment.nodes]]
name = "worker"
cpus = 2
memory_mb = 4096
storage_mb = 20480

[agent]
timeout_sec = 600

The agent SSHes into the first node (primary). Nodes resolve each other by hostname.

What makes a strong task

1. Real ops, not toy puzzles — incident response, deployment, or repair an SRE would recognize.
2. Layered stack — situate the failure in L1 → L4 when possible.
3. Honest verifiers — prefer multi-check reward.json; resist reward hacking.
4. Clear instructions — state goal and constraints without leaking the root cause unless that is the scenario.
5. Reproducible environment — pin packages / images; document BM Cluster profiles or base images if needed.

Submission checklist

• Directory matches the layout above
• instruction.md is self-contained
• Verifier writes /logs/verifier/reward.txt or reward.json
• task.toml declares type, resources, and [metadata].difficulty
• Reference solution/solve.sh works under the intended environment (when you have runner access)
• Open a PR against XuanmiaoG/InfraBench with a short description: layer, backend, what is being tested

Runtime note

You do not need a public InfraBench runner to submit a task package. Reference solutions may be validated later by maintainers with Syscraft access. Do not invent install steps for a runner that is not released yet.

Source: CONTRIBUTING.md · Docs home · Submit a task. Task layout, task.toml templates, verifier contract, and PR checklist.