SLURM experiments

Run HAI-Co² against a model you host yourself on a SLURM cluster, driven by a scripted batch client through the real HTTP API. Developed and verified on the UKP Lab cluster at TU Darmstadt, so the concrete values throughout (partition names, GPU types, storage paths) are that site’s. They are examples. Everything site-specific lives in one file — see Adapting to another cluster. This is not how haico.gr is deployed — see Deployment for that, and Local installation for a laptop stack. This page is about running hundreds of scripted conversations against open weights instead of a paid API, so a study is cheap, reproducible, and pinned to an exact backend image — and to exact model weights when you pin those too (see MODEL below). Everything described here lives in deployment/slurm/, and the design rationale is ADR-0009.

Prerequisites

On the cluster

An account with access to GPU nodes, and whatever network access the site requires — a VPN, at many.
Apptainer or Singularity on the compute nodes. Postgres and the backend run as containers; only vLLM does not. Check before anything else:
srun --partition=<your-gpu-partition> --pty apptainer --version
If that fails, this stack will not work here as designed — see Adapting to another cluster.
Python 3.9+ as python3 on the compute nodes (srun --pty python3 --version). Verified on 3.12.
Roughly 15 GB of quota for the serving environment, plus whatever model weights you download if the site has no shared cache.

On your laptop

An SSH host alias for the login/submit node. The tooling never prompts for a password, so key auth must already work:
# ~/.ssh/config Host mycluster HostName login.cluster.example.edu User your-username IdentityFile ~/.ssh/your-key ServerAliveInterval 60
Verify with ssh mycluster hostname, then put the alias in cluster.env. The User line is load-bearing, not cosmetic: the laptop-side tooling reads it with ssh -G and substitutes it for {user} in CLUSTER_WORK to decide where the venv, the secrets file and the results live, while the job substitutes its own $USER on the node. Put your cluster username there even if it matches your laptop login.
make, bash, rsync, curl and python3 (used by make secrets, make fetch and submit.sh’s local validation).
The GitHub CLI (gh), logged in, for make digest only: it resolves an image tag through your gh session, not through the token stored by make secrets. The target refuses to run when gh is missing or logged out.
A classic GitHub token with the read:packages scope, to pull the private backend image. Fine-grained tokens are rejected by GHCR with an opaque 401.

The backend image is private. Your GitHub account needs read access to the haico-backend package, which is granted on the repository, not by the token — a token with the right scope still returns 401 without it. make secrets verifies this up front and refuses a token that cannot pull, so you find out in seconds rather than twenty minutes into a job. If it fails, ask a maintainer for package read access.

From zero to a first result

git clone https://github.com/petrosrapto/HAICO.git
cd HAICO/deployment/slurm

$EDITOR cluster.env              # 1. point it at your cluster  (skip at UKP)
make env                         # 2. queue the venv build job; wait for it  (once)
make secrets                     # 3. store the GHCR pull token   (once)

make gate   EXP=qwen3-32b        # 4. can this model drive the agent at all?
make submit EXP=qwen3-32b        # 5. run the batch
make status                      # 6. watch it
make fetch  EXP=qwen3-32b-pilot  # 7. pull results into ./runs/

Steps 1–3 are once per account. Everything after is per experiment. make help lists every target.

Step 1 first, or steps 2 and 3 build in the wrong place. cluster.env holds the submit host and the work directory, and both make env and make secrets write to paths derived from it. At the UKP cluster the shipped values are already correct and you can skip straight to step 2.

On any cluster other than UKP, step 4 needs an arm of your own first. The shipped qwen3-32b arm hardcodes PARTITION=gpu, QOS=gpu, GRES=gpu:a180:1 and a model path under UKP’s shared weights. cp arms/qwen3-32b.env arms/<name>.env, set PARTITION, QOS (empty if your site has none), GRES (your site’s GPU type name) and MODEL (a HuggingFace id such as Qwen/Qwen3-32B if SHARED_WEIGHTS is empty, or an absolute path to a flat model directory), then make gate EXP=<name>. See Adapting to another cluster.

`make env`

Submits a CPU-partition job (CLUSTER_CPU_PARTITION / CLUSTER_CPU_QOS in cluster.env) that creates the venv named by VLLM_VENV and installs env/requirements.txt. A pip install needs no GPU, and holding an A100 to download wheels denies it to someone who does. make env only queues the job and returns at once. Watch it with make logs EXP=haico-env and wait for the line environment ready: … before running make gate: the build takes a few minutes once it starts, plus whatever the CPU queue wait is. make secrets does not depend on it and can run meanwhile. A gate submitted too early dies at source …/bin/activate after its own queue wait. It is idempotent — re-running when the environment is already healthy is a no-op, so it is safe as the first line of any runbook. make env FORCE=1 rebuilds from scratch; make env VENV=/path/to/other builds elsewhere, which is how to test a version bump without disturbing the environment your campaigns use. The build verifies rather than trusting pip’s exit code: it imports vLLM, httpx and torch, and reports whether the tool-call parser registry loads (advisory — make gate is the real test). That matters because a subtly broken install otherwise surfaces much later as “the model cannot emit tool calls” at the top of a campaign. It then writes haico-requirements.lock.txt into the venv — the full resolved closure from pip freeze. requirements.txt states intent; the lock file states what a given result is actually attributable to.

Versions are pinned deliberately. A campaign’s results belong to a model and the stack that served it, so a floating range means a rebuild months later silently serves the same weights through different sampling or a different parser set. Upgrading vLLM is a decision, not housekeeping — the tool-call parser registry changes between releases and every arm file’s TOOL_PARSER names one of its entries, so re-run make gate after any bump.torch is deliberately absent from requirements.txt. vLLM ships its own pin and installs a matching build; adding a torch line fights that and is how you get a venv that imports but cannot see the GPU. It is also why this must stay separate from any training environment — trl/peft pull a different torch.

Another researcher on the same cluster runs make env once and changes nothing else: VLLM_VENV is per-user by default. Point it at a shared project directory if a group wants one environment — accepting that everyone then shares its pins, and an upgrade breaks everyone at once.

`make secrets`

Prompts for your GitHub username and the token, verifies the token against GHCR without downloading anything, and writes both to $WORK/haico-cluster/secrets/ghcr.env with mode 600. submit.sh checks that this file exists before queueing a batch, because the job dies without it — after the queue wait and the Postgres pull. It deliberately does not offer to reuse your gh CLI token. That credential carries whatever scopes the account granted it — typically repo, workflow, admin:org — and storing it on multi-user cluster storage for the length of a campaign, to save one token mint, trades a broad credential for a small convenience. A dedicated classic token with only read:packages can do nothing but pull the image. It is stored under $WORK and not $HOME for a reason: $HOME is not mounted on compute nodes, so a credential there is unreadable exactly where the job needs it. That is the only long-lived secret. JWT_SECRET, the admin password, the HAI-Co² API token and the vLLM key are all generated inside each job and die with it. HF_TOKEN is optional — needed only for gated repositories such as Llama, and irrelevant for Qwen or Mistral, which are what the shipped arms use. Add one later with bash scripts/bootstrap_secrets.sh --hf: the script rewrites the file as a whole, but it reads the stored values first and Enter at a prompt keeps them, so you are not asked to dig out the GHCR token again; a blank HF answer keeps whatever is stored. Re-running without --hf (to rotate the GHCR token) keeps a stored HF token too.

Run your first experiment

make gate   EXP=qwen3-32b        # can this model drive the agent at all?
make submit EXP=qwen3-32b        # run the batch
make status                      # squeue --me
make logs   EXP=qwen3-32b        # tail the newest .out and .err log for that arm's RUN_ID
make cancel EXP=qwen3-32b        # stop it: graceful for a running batch, removed if still queued
make resume EXP=qwen3-32b        # continue a RUN_ID: ok conversations kept, the rest re-run
make fetch  EXP=qwen3-32b-pilot  # pull results into ./runs/  (takes the RUN_ID, not the arm)

Run make gate first for any model or parser you have not used before. It costs minutes and catches a failure that would otherwise burn the entire allocation silently — see Choosing a model.

Where things land on the cluster. $WORK/haico-cluster/ is this directory, rsynced by every make submit / make gate / make env, plus logs/<jobname>_<jobid>.out|.err (the job logs) and secrets/ghcr.env. $WORK/haico-runs/<RUN_ID>/ holds a run’s results and is what make fetch copies. $WORK/envs/haico-serving is the venv from make env; $WORK/hf_cache, $WORK/.apptainer and $WORK/.cache are the model, image and pip caches shared across jobs. Jobs are named haico-exp-<RUN_ID> (batch), haico-gate-<RUN_ID> (gate) and haico-env (venv build). Two log files per job. Progress goes to .out; every FATAL: diagnostic, the driver’s progress notes and the gate’s GATE FAILED verdict go to .err. make logs tails both. make fetch does not copy the job logs — read them with make logs or ssh while you need them. Reading the gate. The gate job serves the model, makes GATE_N (default 25) tool-calling requests at temperature 0.7 (the arm’s TEMPERATURE is not used) with a 2048-token output cap and a 180 s timeout per call, and passes when at least 95 % return a parseable tool call carrying a non-empty action_and_reasoning. GATE PASSED appears in the .out log, GATE FAILED in .err. The per-prompt report is $WORK/haico-runs/<RUN_ID>/gate-report.json ({model, base_url, n, passed, pass_rate, threshold, results: [{prompt, passed, reason}]}, with gate-vllm.log beside it); make fetch EXP=<RUN_ID> pulls both, and a second gate run on the same arm overwrites them. Read reason before trusting a failure: no tool_calls emitted, action_and_reasoning missing and action_and_reasoning empty are model results; a reason beginning ConnectError, ReadTimeout or HTTPStatusError means the call never reached the model, and the gate says so with THIS IS NOT A MODEL RESULT.

Stopping a run

make cancel EXP=<arm> acts on the jobs named after the arm’s RUN_ID. A queued batch is removed from the queue. A running batch receives TERM on its batch shell only, so the job’s own teardown runs: the driver stops launching turns, records every unfinished conversation as interrupted, writes ledger.json, captures what it can with one short attempt per panel, and the database is dumped while Postgres is still up — allow a few minutes. make cancel EXP=<arm> FORCE=1 is a plain scancel: the whole cgroup is killed at once and nothing newer than the last periodic dump (every DUMP_INTERVAL seconds) survives. Gate jobs are always cancelled outright. Do not scancel a batch by hand unless you mean FORCE=1: it kills Postgres together with the driver, so the final dump fails and the ledger is not written. Afterwards make submit refuses the RUN_ID (it has a ledger); make resume EXP=<arm> continues it. sacct shows a cancelled run as FAILED with exit code 15:0 (Slurm’s rendering of 143, 128 + SIGTERM) on purpose, so a stopped run is never mistaken for a completed one; the results up to the signal are complete.

Changing the experiment

Everything is in one arm file, arms/<name>.env. Every key that shapes the result is required and has no default: submit.sh refuses an arm that omits one. Six keys are optional and fall back to a fixed value when absent — HAICO_READY_TIMEOUT (2400 s), PHOENIX_ENABLED (0), DUMP_INTERVAL (600 s), GATE_N (25), and REASONING_PARSER / EXTRA_VLLM_ARGS (empty). Keep them in the arm anyway so the file records what ran. The driver’s grace period after a signal (150 s) is a job-script constant, not an arm key. The arm file is what you edit; run-config.env in the run directory is what actually ran. Every attempt appends the job id, node, GPU, image digest, every knob, and for a model given as a directory the SHA-256 of its config.json and safetensors index, so a result stays attributable after the arm file has been edited for the next run.

cp arms/_template.env arms/my-run.env
$EDITOR arms/my-run.env
make dry-run EXP=my-run     # validate and print the sbatch line; submit nothing
make submit  EXP=my-run

submit.sh validates locally before anything is queued, because a typo caught on your laptop costs a second and the same typo caught by the scheduler costs a queue wait plus a model load. It rejects: missing values, a TENSOR_PARALLEL_SIZE that disagrees with GRES, a request above the ceilings in cluster.env, an output cap that leaves no room for the prompt, a tag where a digest belongs, a prompts file that does not parse (every line must be a JSON object with id and non-empty string turns, ids unique and usable as file names), a missing secrets file on the cluster, a QOS your account does not have, and a RUN_ID that already has a ledger (make resume is the way to continue one). Every make target wraps ./submit.sh <arm> [--gate] [--resume] [--dry-run] [--no-sync]; --no-sync submits without re-syncing the tree to the cluster, for when nothing but the arm has changed.

Arm-file reference

Identity

Key	Notes
`RUN_ID`	Names the run directory and ledger row. Must be unique; results are evidence, so a re-run cannot overwrite them.
`MODEL`	HuggingFace id, or an absolute cluster path to a flat model directory (`$SHARED_WEIGHTS/models--Qwen--Qwen3-32B` at UKP; a bare id re-fetches tens of GB into `$HF_HOME`). vLLM serves bf16/fp16 only — merge a QLoRA adapter first, a bnb-4bit checkpoint will not serve. The image is pinned by digest; the weights are pinned only if you pin them: with an id, vLLM serves the repository’s current `main`, so add `EXTRA_VLLM_ARGS=--revision <commit sha>`; with a directory, `run-config.env` records the SHA-256 of its `config.json` and safetensors index.
`BACKEND_DIGEST`	Pin the backend image by digest, never a tag: a tag can be repointed, silently changing what a “reproduced” run ran. Resolve one with `make digest TAG=e1023eb` (needs a logged-in `gh`; `IMAGE=<owner>/<name>` for a fork).

GPU and node selection — all passed straight through to sbatch.

Key	Notes
`PARTITION`	Site-specific. `sinfo -o "%P %a %l %D %G"`. At UKP: `gpu` (`PreemptMode=OFF`) or `yolo` (every node, but preemptible with `GraceTime=0` — killed mid-token, no warning, requeued from the start). Leave empty to omit the flag.
`QOS`	*Site- and* account-specific**, and optional — many sites need none. Leave empty to omit it. `submit.sh` checks it against your account before queueing, because Slurm otherwise rejects a bad one with a terse `Invalid qos specification`.
`GRES`	`<type>:<count>`. `a180` is A100-80GB (penelope, rubeus) and the workhorse; also `h200nvl` 140 GB, `b6000` 96 GB, `a6000`/`l40` 48 GB, `a100` 40 GB.
`CONSTRAINT`, `NODELIST`, `EXCLUDE`	Optional. Feature constraint, host pinning, host avoidance.
`CPUS`, `MEM`, `TIME`	`CPUS` caps at `CLUSTER_MAX_CPUS_PER_JOB` (16 at UKP). Keep `MEM` honest: memory, not GPUs, is what limits how many array tasks co-schedule against the 512 GB/user cap. `TIME` is the whole allocation: budget it as model load (bounded by `HAICO_READY_TIMEOUT`, ~15–25 min for a 32B on a cold JIT cache) plus the campaign plus ten minutes. The driver stops launching turns ten minutes before the end and cancels every conversation still in flight (each becomes a `campaign_timeout` row, re-run by `make resume`), so the capture pass and the final `pg_dump` run with Postgres alive; Slurm’s own `TERM` follows five minutes before the end, and the driver gets 150 s to write the ledger. If fewer than twenty minutes remain when the batch starts, no ceiling is armed and only the `TERM` path protects the ledger.

A QOS the site defines is not necessarily one your account may use. At UKP, gpu-large and gpu-small exist but are absent from a plain ukp-researcher association. Find out what is actually yours:

ssh <cluster> 'sacctmgr -nP show assoc user=$USER format=Account,Partition,QOS'

CONCURRENCY=2 is measured, not chosen. The backend serialises requests — vLLM reports Running: 1 reqs in the large majority of samples regardless — so raising concurrency does not raise throughput; it queues conversations inside the backend, where they emit no bytes and the driver eventually abandons them. On identical 8-conversation runs:

`CONCURRENCY`	completed	failures	wall clock	slowest conversation
4	6/8	2 × `client_timeout`	53m32s	1047s
2	8/8	0	50m27s	609s

Lower concurrency was both more reliable and marginally faster. Raise it only once the backend genuinely serves requests in parallel.

vLLM server

Key	Notes
`TENSOR_PARALLEL_SIZE`	Must equal the GPU count in `GRES`; `submit.sh` enforces it.
`MAX_MODEL_LEN`	Total context. The agent’s prompt carries the whole working document every turn, so this must comfortably exceed prompt + `MAX_TOKENS`.
`MAX_TOKENS`	Output cap per turn. Budget roughly a quarter of the window on a 32k model. The deployed environments use 32768 because their hosted models have far larger contexts.
`TOOL_PARSER`	How vLLM turns the model’s output into OpenAI `tool_calls`. See below.
`REASONING_PARSER`	For models that emit `<think>` blocks. Empty to disable.
`GPU_MEMORY_UTILIZATION`, `MAX_NUM_SEQS`, `DTYPE`, `EXTRA_VLLM_ARGS`	Passed through; `EXTRA_VLLM_ARGS` is appended verbatim.
`HAICO_READY_TIMEOUT`	Seconds to wait for the server (default 2400). Dominated by `torch.compile` and CUDA-graph capture, not weight loading; the JIT caches are per job, so every run compiles cold.

The batch — what the driver sends, and how it waits.

Key	Notes
`PROMPTS`	File under `driver/prompts/`; see Prompt sets. Required; validated locally.
`TEMPERATURE`	Sent in every request’s `config.args`, with `MAX_TOKENS`. Part of the record of a run: two arms with different values are different experiments. Not used by `make gate`, which samples at 0.7. Required.
`CONCURRENCY`	Conversations in flight at once; `2` is the measured optimum on this stack (see the Note above). Each open turn also holds one unpooled Postgres connection. Required.
`READ_TIMEOUT`	Seconds of socket silence before the driver abandons a turn as `client_timeout`; also passed to the backend as the provider call’s timeout. Heartbeats only start once the agent streams, so the window before the first frame is what this bounds. Required.
`DUMP_INTERVAL`	Seconds between periodic mirrors of the transcripts plus a `pg_dump` to `$WORK/haico-runs/<RUN_ID>` (default 600). These are what survive a preemption or a `FORCE=1` cancel; lower it for short validation runs.
`GATE_N`	Tool-calling probes `make gate` sends before scoring (default 25); the pass threshold is 95 %. Ignored by `make submit`.
`PHOENIX_ENABLED`	`1` runs Phoenix (`arizephoenix/phoenix:13.23.0`, the deployed pin) alongside the stack and traces every turn into it over OTLP/HTTP on the UI port (`/v1/traces`); its database is copied out at teardown.

Two ceilings are constants in driver/run_batch.py, not arm keys: a turn is abandoned as turn_deadline after 1800 s in total, and as stream_idle after 900 s of keep-alives with no data frame. Raising READ_TIMEOUT does not move them; a legitimately slow model at high CONCURRENCY can hit the first — lower concurrency, or change the constant and record it with the run.

Prompt sets

PROMPTS names a file under driver/prompts/, one conversation per line: {"id": "pilot-01-report", "turns": ["first user message", "second", …]}. Blank lines and lines starting with # are ignored, so a set can carry notes or have a conversation commented out. id becomes transcripts/<id>.json and the key make resume uses, so ids must be unique within a set and usable as file names; submit.sh checks both before queueing. Keys other than id and turns are ignored. Every turn is sent verbatim as the user message; see driver/prompts/README.md for why each turn must be answerable without a clarifying question. Two sets ship: pilot.jsonl, 8 conversations of 4 turns covering the document, plan, preferences and objective panels, and stress.jsonl, 20 conversations of 6 turns that apply three turn policies (grow: enlarge the document every turn; plan: churn todos and preferences; revise: rewrite repeatedly) across seven topics, built to push on MAX_MODEL_LEN, tool-call volume and snapshot-chain length respectively. Ids encode set, index and policy (stress-04-grow-tutorial), so outcomes can be grouped by policy.

Adapting to another cluster

Everything specific to one site lives in deployment/slurm/cluster.env. In the common case that file is the only thing you change.

CLUSTER_HOST=ukp                                  # ssh alias for the submit node
CLUSTER_WORK=/storage/ukp/work/{user}             # readable from COMPUTE nodes
VLLM_VENV={work}/envs/haico-serving               # venv containing vLLM
SHARED_WEIGHTS=/storage/ukp/shared/shared_model_weights   # optional model cache
CLUSTER_MAX_GPUS_PER_JOB=4                        # enforced by submit.sh, see below
CLUSTER_MAX_CPUS_PER_JOB=16
CLUSTER_CPU_PARTITION=cpu                         # where `make env` builds the venv
CLUSTER_CPU_QOS=cpu                               # its QOS; always passed, so name one your account has

CLUSTER_WORK is the one that most often bites. On many clusters $HOME is not mounted on compute nodes, so a credential or cache under it is unreadable exactly where the job runs. It is the single source of truth for where things live: the laptop side resolves the venv, the secrets file and the results from it (with {user} taken from the SSH config’s User line), and the job resolves the same paths from it with its own $USER. $SLURM_STORAGE_HOME is used only when CLUSTER_WORK is left empty. CLUSTER_MAX_GPUS_PER_JOB and CLUSTER_MAX_CPUS_PER_JOB are ceilings submit.sh enforces on your laptop — an arm asking for more is refused before anything is queued. They are not read from the scheduler; set them to what sacctmgr show qos format=Name,MaxTRESPerJob reports for the QOS you use. CLUSTER_CPU_PARTITION / CLUSTER_CPU_QOS are used only by make env, which deliberately builds on a CPU partition; if the site has none, point both at the GPU partition and its QOS and accept holding a GPU for a pip install. Then, in your arm file, replace the scheduler names — these differ at every site:

Key	How to find your site’s value
`PARTITION`	`sinfo -o "%P %a %l %D %G"`
`QOS`	`sacctmgr show qos format=Name,MaxTRESPerJob,MaxWall`
`GRES`	`sinfo -o "%n %G"` — GPU type names are site-defined (`a180`, `a100`, `h100`, …)
`CPUS`, `MEM`, `TIME`	whatever the QOS above permits

Three assumptions worth re-checking on a new site, because they shaped the design rather than merely configuring it:

Apptainer runs OCI images unprivileged. If your site has Docker on compute nodes, the job could be simpler. If it has neither, this approach does not port.
Compute nodes accept no inbound connections. That is why everything is co-located in one job. Where inbound is allowed, a split deployment becomes possible — though a requeued job landing on a different node still breaks every client endpoint.
Compute nodes have outbound internet. The job pulls container images at start. Without it, pre-stage the SIFs on shared storage and point the job at them.
The scheduler exports a per-job, node-local $TMPDIR. The batch job keeps Postgres, its socket, the container images and the working results there and relies on it being wiped with the allocation; the checkpointer writes on every turn, and a shared filesystem degrades for everyone under that pattern. A site that does not set it stops the job at once with TMPDIR is not set: … — after a green make gate, which never touches it. Check with srun --pty printenv TMPDIR; if it is empty, export node-local scratch at the top of scripts/_env.sh.
Single-node PCIe GPUs without a CUDA toolkit. scripts/_env.sh exports NCCL_P2P_DISABLE=1 and NCCL_IB_DISABLE=1 (PCIe A100s without NVLink; an InfiniBand probe that hung) and VLLM_USE_FLASHINFER_SAMPLER=0 (no nvcc on compute nodes). They cost nothing here but are wrong for NVLink nodes or multi-node jobs. They are not arm keys — submit.sh forwards only the arm’s own knobs — so change them in _env.sh, or export them in the login-node shell, which --export=ALL carries into the job. Downloaded weights land in $WORK/hf_cache (HF_HOME), shared across jobs.

Nothing else is site-specific: the driver, the capability gate, the arm-file format and the ledger are all plain HTTP and JSON.

Choosing a model

Every HAI-Co² tool schema carries an injected, required action_and_reasoning argument. A model that omits it does not fail cleanly: _is_blank_turn treats any message bearing a tool call as a real response, so the retry wrapper never sees the problem and the turn loops. This is #184, and it is why the server default moved off gpt-4o-mini. Two things contain it:

agent.recursion_limit (default 50) bounds the loop. Note that LangGraph does not raise when the limit is reached: create_react_agent substitutes a canned reply carrying no tool calls, which ends the graph normally. The backend detects that sentinel and emits an error frame instead, or a runaway turn would be reported as a success.
make gate EXP=<arm> tells you before a campaign whether a model can satisfy the schema at all. It serves the model, calls it GATE_N times (25 by default, 15 fixed prompts cycled) with a miniature of the real tool schema, and asserts every returned tool call parses and carries a non-empty action_and_reasoning. It exits 0 only when at least 95 % pass — at GATE_N=25 one failure still passes and two fail — so it can gate a pipeline. Where the verdict and the report land is described under Run your first experiment.

If a model fails, try a different TOOL_PARSER before abandoning it; if it still fails, prefer structured decoding on a model you have already validated over swapping in a new architecture.

--enable-auto-tool-choice requires --tool-call-parser, and without both vLLM never emits tool_calls at all. If the gate reports no tool_calls emitted for every prompt, suspect the flags before the model. The job always passes both.

Valid TOOL_PARSER values (vLLM 0.23.0 registers 42). The ones you are likely to want: hermes (Qwen2.5/Qwen3 — start here), qwen3_xml, qwen3_coder, llama3_json, llama4_json, llama4_pythonic, mistral, openai (gpt-oss), deepseek_v3/v31/v32/v4, glm45, glm47, granite, granite4, pythonic, xlam, kimi_k2, minimax, seed_oss, olmo3, gemma4, internlm, jamba, phi4_mini_json, step3. Already staged on the cluster under $SHARED_WEIGHTS (cluster.env), so no download if you point MODEL at the directory: MODEL=$SHARED_WEIGHTS/models--Qwen--Qwen3-32B (62 GB bf16), …/models--Qwen--Qwen3-32B-FP8, …/models--Qwen--Qwen3-4B-Thinking-2507, …/models--mistralai--Mistral-7B-Instruct-v0.2; check for others with ls $SHARED_WEIGHTS/. A bare HuggingFace id such as Qwen/Qwen3-32B also works but re-fetches the weights into $HF_HOME. Note that Qwen3.5-27B is Qwen3_5ForConditionalGeneration — a hybrid-linear vision model, not a drop-in swap for Qwen3-32B.

How it works

One sbatch job on one GPU node holds everything, all on 127.0.0.1:

GPU NODE — ONE JOB ──────────────────────────────────────────────────────┐
 [1] vllm serve   $WORK/envs/haico-serving         127.0.0.1:$VPORT      │
 [2] postgres 15  apptainer                        127.0.0.1:$PGPORT     │
 [3] uvicorn      backend SIF, 1 worker            127.0.0.1:$APIPORT    │
 [4] driver       run_batch.py, Bearer haico_pat_…                       │
                                                                         │
 /etc/hosts bind-mount: 127.0.0.1 vllm localhost                         │
 results → $TMPDIR → rsync + pg_dump every DUMP_INTERVAL → $WORK/haico-runs/$RUN_ID │
─────────────────────────────────────────────────────────────────────────┘

Why one node. Compute nodes accept no inbound connections, so a backend anywhere else would need a redir forwarder on the login node — restricted to ports 5000–5100, unreachable from the TU/HRZ VPN, and killed daily at 03:00. Co-locating removes that entire failure class. Splitting backend-onto-CPU-node buys nothing either: a gpu allocation already grants 16 CPUs and 256 GB, and the backend needs ~2 CPUs and a few GB. Why Apptainer, and why not for vLLM. Docker exists nowhere on the cluster; Apptainer 1.4.5 is on compute nodes and runs the existing OCI images directly, giving parity with what is deployed. vLLM stays outside a container because it is already installed in $WORK/envs/haico-serving (VLLM_VENV in cluster.env) and needs the GPUs, which would mean --nv passthrough for no parity benefit. (podman on the login node is a dead end: there is no /etc/subuid entry for the account, so rootless UID mapping — and apptainer build --fakeroot — is impossible.) Apptainer does not namespace the network, so the four processes reach each other on loopback with no compose-style network. The flip side is that loopback is shared with co-tenant jobs, which is why ports derive from $SLURM_JOB_ID and are probed immediately before binding. How traffic reaches vLLM. Entirely through the per-request config block on POST /api/query/stream_steps/sse — no application change:

{
  "query": "…",
  "thread_id": null,
  "config": {
    "API": "openai",
    "model_id": "Qwen/Qwen3-32B",
    "endpoint_url": "http://vllm:24002/v1",
    "args": { "temperature": 0.7, "max_tokens": 8192 }
  }
}

The endpoint hostname must contain vllm. Credential resolution returns VLLM_API_KEY only when the endpoint URL contains that substring, and otherwise falls back to OPENAI_API_KEY. The job bind-mounts an /etc/hosts mapping 127.0.0.1 vllm localhost precisely so this holds.Never set OPENAI_ENDPOINT_URL. It is a global fallback applied to every OpenAI-family model, so setting it as a deployment secret would silently re-route production traffic with no failing health check.

How the API token is obtained. POST /api/keys requires a browser session, which a throwaway database cannot provide. The job generates a fresh haico_pat_… per run and passes it in as ADMIN_DEFAULT_API_TOKEN, which migration 0017 installs for the seeded admin. You do not set this anywhere — not in an arm file, not in .env, not on the cluster. If the pinned image predates that migration the job falls back to logging in as the admin and calling POST /api/keys, so it works against any digest. See ADR-0006.

Results

make fetch EXP=<run-id> pulls runs/<run-id>/:

File	What
`ledger.json`	one row per conversation, the attrition table, and `stopped_early` naming why a run ended before its prompt set (signal, wall-clock ceiling, provider abort), or `null` — schema below
`transcripts/<id>.json`	per-turn outcomes and the captured workspace — schema below
`transcripts/superseded/<id>.attempt-N.json`	earlier, non-`ok` attempts of conversations that `make resume` re-ran; never delete these to “clean” a run
`run-config.env`	what each attempt actually ran with: job id, node, GPU, image digest, every knob, model hashes — appended per attempt
`haico-db.sql.gz`	`pg_dump` of the last attempt’s database, refreshed every `DUMP_INTERVAL` seconds and at teardown
`haico-db.before-job-<id>-<time>.sql.gz`	the dump that was in the run directory when job `<id>` started at `<time>`, i.e. the previous attempt’s last dump. Named after the job that displaced it, not the one that wrote it; kept so no attempt overwrites another’s
`gate-report.json`, `gate-vllm.log`	written by `make gate` for the same `RUN_ID`: per-prompt pass/fail with reasons, pass rate and threshold, plus the gate’s vLLM log; a second gate run overwrites them
`phoenix.db.gz`, `phoenix.log`	Phoenix trace database and service log, when `PHOENIX_ENABLED=1`
`vllm.log`, `backend.log`, `pg.log`	service logs

make fetch prints the attrition table, n_with_capture_errors and stopped_early; check the first two before calling a run clean. Each transcript captures the final state of every workspace panel. document ({thread_id, title, content, updated_at} or null), objective ({thread_id, text, updated_at} or null), todos, preferences and artifacts (lists of rows; artifacts carry turn_index, artifact_type and payload) are the panels as they stood when the conversation ended. messages is the chat history rebuilt from the checkpointer ({role: user|assistant|tool, content, tool_calls?, tool_name?}), which is where the agent’s replies and tool calls live. graph is the conversation DAG with one node per turn (user_message cut to 140 characters, assistant_summary to 200, reasoning_steps, artifact_count). snapshots is the per-turn index only: {id, turn_index, user_message, captured_at}, newest first, at most 50. The per-turn document, objective, todo and preference bodies, and the model_id and provider recorded for each turn, are not in any transcript; they are rows of workspace_snapshots in the database dump, so keep the dump if you need per-turn history. A conversation can be ok and still have lost a panel. capture_errors on the transcript and on its ledger row names each panel that could not be read (messages: HTTP 500 (after 3 attempts)), and the panel itself holds the reason as a string (HTTP <code>, thread not in this job's database (HTTP 404), or an exception name such as ReadTimeout) instead of data — any string in a panel slot is that marker. n_with_capture_errors in the ledger counts such conversations; treat a non-zero count as an incomplete capture. Within the same job the driver retries with a 120/240/480 s ladder. make resume re-fetches only the missing panels, and only for threads that exist in that job’s fresh database, so a panel lost in an earlier attempt comes back as thread not in this job's database; the only copy of that workspace is that attempt’s kept haico-db.before-job-*.sql.gz.

`ledger.json`

{
  "run_id": "qwen3-32b-pilot",
  "n_attempted": 8,
  "n_completed": 7,
  "outcomes": {"client_timeout": 1, "ok": 7},
  "n_with_capture_errors": 0,
  "stopped_early": null,
  "conversations": [
    {"id": "pilot-01-report", "thread_id": "th_…", "outcome": "ok",
     "turns_completed": 4, "turns_attempted": 4, "elapsed_s": 412.3,
     "capture_errors": []}
  ]
}

n_attempted counts every row, including not_attempted, interrupted and campaign_timeout rows, so it equals the size of the prompt set. n_completed is the number of ok rows. outcomes is the attrition table: one count per outcome name. stopped_early is null or one of received SIGTERM / received SIGINT (a cancel, or the pre-wall-clock signal), campaign ceiling of <n>s reached, or provider abort: <n> conversations in a row rejected immediately ('<message>'). Each row carries turns_attempted (turns the driver sent) and turns_completed (turns that ended ok); a failed conversation has turns_attempted = turns_completed + 1, and thread_id is null when turn 0 failed before the server assigned one. The ledger row and transcripts/<id>.json describe the last attempt of a conversation; earlier tries are in superseded/.

`transcripts/<id>.json`

Field	Meaning
`id`, `run_id`, `model`	the spec id, the `RUN_ID`, and the `MODEL` string passed to vLLM
`thread_id`	the server-side conversation; `null` if turn 0 failed before one was assigned
`outcome`	`ok`, or the outcome of the first turn that failed (later turns are never sent)
`turns[]`	one entry per turn sent: `index`, `query`, `outcome`, `elapsed_s`, `step_count` (SSE `step` frames), `artifact_count`, `error` (first 500 characters of the server’s message, or `null`)
`workspace`	the eight panels described above, captured after every conversation finished
`capture_errors[]`	one string per panel that could not be read
`started_at`, `elapsed_s`	wall-clock start (epoch seconds) and duration of the turns, excluding capture

A turn record does not contain the agent’s reply, its tool calls or its steps: the reply and tool calls are in workspace.messages, and the individual ReAct steps are recorded only when PHOENIX_ENABLED=1. Token counts are not recorded per turn anywhere — vllm.log carries only aggregate throughput lines — so a study that needs them must enable Phoenix, whose LLM spans carry the provider’s usage fields.

Reading the database dump

haico-db.sql.gz is a plain-SQL pg_dump taken by Postgres 15, so restore it with psql into a server of version 15 or newer (the local stack’s postgres:15-alpine will do):

docker run -d --name haico-run -e POSTGRES_USER=haico -e POSTGRES_PASSWORD=x -e POSTGRES_DB=haico postgres:15-alpine
gunzip -c runs/<run-id>/haico-db.sql.gz | docker exec -i haico-run psql -q -U haico -d haico

Tables worth knowing: conversation_users (one row per thread_id, with the final turn_index), workspace_snapshots (one row per (thread_id, turn_index): the document, objective, todos and preferences as the agent saw them at the start of that turn, plus model_id and provider — the per-turn provenance nothing else records), documents, objectives, todos, preferences, artifacts (final state, the same as the transcript panels), and the LangGraph tables checkpoints, checkpoint_blobs, checkpoint_writes (the full message history the messages panel is rebuilt from). api_keys holds only the hash and prefix of the run’s token. Every attempt starts a fresh database, so haico-db.sql.gz holds only the last attempt’s threads. Conversations completed by an earlier attempt of a resumed run, and every transcript in transcripts/superseded/, reference threads that exist only in the matching haico-db.before-job-<id>-<time>.sql.gz; restore each into its own database. A conversation that was cut mid-turn by a cancel has no transcript, but its partial thread is in the dump that attempt wrote.

Reading the Phoenix traces

phoenix.db.gz is the SQLite database of the Phoenix instance that ran inside the job (arizephoenix/phoenix:13.23.0, the same image the deployed stacks run). Unzip it and serve it with that image or a newer one, pointing it at the file:

gunzip -k runs/<run-id>/phoenix.db.gz
docker run --rm -p 6006:6006 -e PHOENIX_SQL_DATABASE_URL=sqlite:////data/phoenix.db \
  -v "$PWD/runs/<run-id>:/data" arizephoenix/phoenix:13.23.0

The traces are in project haico_slurm, one trace per turn, with the ReAct steps, tool calls and timings that neither the transcript nor the database record; sessions are keyed by thread_id, which is how to find the trace for a transcript.

Attrition is a result, not a cleaning step. Conversations fail most often in exactly the arms where the model is weakest, so they are not missing at random and summarising only the survivors overstates whatever the campaign measures. ledger.json reports n_attempted, n_completed, and a mutually-exclusive taxonomy: tool_arg_missing, recursion_exhausted, truncated, context_overflow, turn_deadline, client_timeout, stream_idle, transport, server_error, plus the harness outcomes interrupted, campaign_timeout, driver_error and not_attempted. Only the first indicts the model’s ability to drive the agent; recursion_exhausted is a step-ceiling hit with no tool-schema error visible in the turn’s steps. client_timeout is a harness artifact — the driver gave up after READ_TIMEOUT seconds of silence — and means you should raise READ_TIMEOUT or lower CONCURRENCY, not that the model failed.

Two design points worth knowing when reading results:

Resume granularity is the conversation, never the turn. The server increments the turn index and writes a snapshot before the agent runs, so replaying a failed turn would duplicate state. A conversation that fails at any point is abandoned whole. make resume keeps only conversations whose transcript ended ok; every other one (any failure outcome, interrupted, campaign_timeout, not_attempted, or missing) is run again from a fresh thread_id, and its earlier transcript is moved to transcripts/superseded/<id>.attempt-N.json so the number of tries stays on record. make submit refuses a RUN_ID that already has a ledger; make resume is the way to continue one. A resume of a finished run does nothing and leaves every transcript byte-identical.
A control arm is your responsibility, and this stack cannot run it. make submit always serves a model with vLLM, and the driver sends "API": "openai" with an endpoint_url on every request, so it cannot target the deployed default (gemini-2.5-flash) or any non-OpenAI provider. For a hosted comparison, bring up a local stack (deployment/local) with the provider key, mint a personal access token, and run the driver by hand with the same prompt set and sampling settings: HAICO_API_TOKEN=… python driver/run_batch.py --api http://localhost:8000 --endpoint-url https://api.openai.com/v1 --model <model> --prompts driver/prompts/<set>.jsonl --out runs/<control-id> --run-id <control-id> --temperature 0.7 --max-tokens 8192 --concurrency 2. That works for OpenAI-compatible endpoints only; a Gemini or Anthropic control arm needs a driver change to send a different API value. Record the control model and the date, since a hosted model is not pinnable.

Exit codes

Exit	Meaning
0	every conversation ran; ledger written
2	driver stopped by TERM/INT (`make cancel`, or Slurm’s signal five minutes before the wall clock); ledger written, unfinished rows `interrupted`
3	provider abort after three immediate rejections; ledger written, remaining rows `not_attempted`
4	campaign ceiling (ten minutes before the wall clock) reached; ledger written, cut rows `campaign_timeout`
1	the driver rejected its arguments (not reachable through `submit.sh`, which validates the prompts file first)
12	a port in the job’s window was in use before anything started
13	vLLM did not accept requests within `HAICO_READY_TIMEOUT`, or exited during start-up
14	the backend never became healthy
15	no usable API token (`could not log in as admin`, `POST /api/keys did not return a usable token`, or `the API token does not authenticate` — all three point at `backend.log`)
16	Phoenix did not become healthy
143 / 130	the job itself was signalled (TERM / INT) and ran its teardown; `sacct` shows `FAILED` with `15:0` / `2:0` on purpose, and the results are complete up to the signal

A gate job exits 1 when the pass rate is below the threshold. Codes 2, 3, 4 and 143 all mean the ledger was written and make resume continues the run; only a job that ends with no ledger.json in $WORK/haico-runs/<RUN_ID> failed in the sense that matters.

Secrets

One long-lived secret exists, and it lives on the cluster — never in the repository, never on your laptop.

$WORK/haico-cluster/secrets/ghcr.env      mode 600, in a 700 directory

make secrets writes it there. Nothing else does, and nothing writes a secret locally: the token is piped over the SSH channel to a cat on the far side, so it never appears in your shell history, in a local file, or in the remote process list.

Why that path

Not $HOME. On many clusters — including the one this was built against — $HOME is not mounted on compute nodes. A credential there is unreadable at exactly the moment the job needs it, and the failure looks like a broken token rather than a missing mount. Not the repository. deployment/slurm/secrets/ is in .gitignore, and secrets/ is excluded from the rsync in both submit.sh and the Makefile. That exclusion is doing two jobs: it stops a local secret being uploaded, and — because rsync --delete does not remove excluded paths on the receiver — it stops every make submit from wiping the token you stored.

What is in it

Variable	Needed for
`GHCR_USER`, `GHCR_PAT`	pulling the private backend image. Required.
`HF_TOKEN`	gated models only (Llama). Qwen and Mistral need none. Add with `bash scripts/bootstrap_secrets.sh --hf`.

Inspect what is stored, without printing any secret value:

bash scripts/bootstrap_secrets.sh --show

Everything else is generated per job and dies with it

Nothing below is ever stored, by you or by the tooling. Each sbatch mints its own from /dev/urandom, and the values vanish when the allocation ends.

Generated per job	Purpose
`JWT_SECRET`	signs the backend’s session tokens
`ADMIN_DEFAULT_PASSWORD`	seeds the admin account in the throwaway database
`ADMIN_DEFAULT_API_TOKEN`	the `haico_pat_…` the driver authenticates with
vLLM API key	authenticates the backend to the model server
Postgres password	`scram-sha-256`, because loopback is shared with co-tenants

This is why there is no .env to fill in for a cluster run, and why a leaked job log costs you nothing.

How the job handles the one real secret

Three properties, each deliberate:

Sourced inside the job, never exported before sbatch. Slurm records the submission environment, so a secret placed there is more exposed than one read at runtime.
unset immediately after the image pull, so it is not in the environment while the batch runs.
APPTAINER_DOCKER_USERNAME/PASSWORD are scoped to the private pull only. They apply to every registry, so exporting them globally breaks the Docker Hub pull of postgres with invalid username/password — which reads like a broken credential rather than a misapplied one.

Use a dedicated classic token with only read:packages. make secrets warns if you paste a gho_ token — that is the gh CLI credential, which typically carries repo, workflow and admin:org. It would work, and that is the problem: a token that can delete repositories should not sit on multi-user storage for the length of a campaign, whatever the file mode.

A second researcher

Each account has its own. $WORK is per-user and 700, so nothing is shared and nothing is inherited — a colleague runs make secrets once with their own token. If they lack read access to the private package, make secrets refuses the token immediately rather than letting them discover it twenty minutes into a job. To revoke: delete the file (rm $WORK/haico-cluster/secrets/ghcr.env) and revoke the token on GitHub. Runs already in flight are unaffected — they pulled their images at startup.

What differs from haico.gr

Recorded deliberately in deployment/slurm/config/backend.yaml, and worth stating in any write-up:

Setting	Cluster	Deployed	Why
`observability.provider`	`none` by default	`phoenix`	Set `PHOENIX_ENABLED=1` in the arm to run Phoenix in the job. Off suits a throughput run — per-turn provider and `model_id` are already in `workspace_snapshots`. On shows what happened inside a turn. The trace DB is copied to the run directory at teardown.
`study.enabled`	`false`	`true`	The in-app study is for human participants.
`api_keys.rate_limit_per_minute`	`0`	`120`	The limiter is in-process and would throttle the driver, not an abuser.
`MAX_TOKENS` (per request)	`8192` in the shipped arms	`32768`	Sent in every turn’s per-request `config.args`, which overrides `llm.max_tokens`; `config/backend.yaml` keeps 32768 as a fallback only. A self-hosted 32k model must leave room for a prompt carrying the whole document.
uvicorn workers	1	1	The rate limiter is in-process state; more than one worker multiplies the ceiling.

Troubleshooting

Every symptom below that starts with FATAL: is in the job’s .err log; make logs shows both files.

Symptom	Cause and fix
`unable to retrieve auth token: invalid username/password` on the postgres pull	`APPTAINER_DOCKER_` applies to every* registry, so GHCR credentials leaked into the Docker Hub pull. The job scopes them to the private pull only; if you script your own pull, do the same.
Gate reports `no tool_calls emitted` for every prompt	vLLM started without both `--enable-auto-tool-choice` and `--tool-call-parser`.
Gate reports `action_and_reasoning missing`	The model cannot satisfy the tool schema. Try another `TOOL_PARSER`, then another model.
Turns end in `tool_arg_missing`	Same cause, now bounded by `agent.recursion_limit` rather than looping.
Turns end in `recursion_exhausted`	The agent hit `agent.recursion_limit` without a visible tool-schema error; inspect the transcript’s steps before blaming the parser.
Gate reports `arguments are not valid JSON` or `arguments are not an object`	The model emits tool calls in a format the chosen `TOOL_PARSER` does not decode. This is the parser, not the schema: try the family-specific parser (`qwen3_xml`, `llama3_json`, `mistral`, …) before changing the model.
Gate prints `N/25 calls never reached the model … THIS IS NOT A MODEL RESULT`	vLLM passed readiness and then died or stalled. Read `gate-vllm.log` in the run directory; do not record this as a capability result.
Three `server_error` rows in under a second each, every later row `not_attempted`, `stopped_early` begins `provider abort:`, job exit 3	The first turn of three conversations in a row was rejected immediately with an authentication/permission-class error, so the driver stopped instead of burning the prompt set; the ledger is complete. The quoted message (also the first turn’s `error` in each transcript) names the cause: in this stack it means the backend or the loopback vLLM rejected the request — check `backend.log`. For a hand-run hosted control arm it is that provider’s credit or key. Fix it, then `make resume`.
Rows are `interrupted` or `campaign_timeout`	Cancelled, or the campaign ceiling fired ten minutes before `TIME` (expected, so the ledger and final dump are written with Postgres alive). Those conversations have no transcript; finished ones and the dumps are in `$WORK/haico-runs/<RUN_ID>`, and `make resume` re-runs the rest.
`sacct` shows the job `FAILED` with exit code 2, 3, 4 or `15:0`	The driver’s own codes, or a graceful stop (143 rendered as `15:0`); see Exit codes. The ledger was written; `make fetch` and `make resume` work the same as after a clean run.
Turns end in `truncated` or `context_overflow`	`MAX_TOKENS` too close to `MAX_MODEL_LEN`; the prompt carries the whole document.
Turns end in `client_timeout`, often with `steps=0` and no `thread_id`	The driver saw total silence before the first frame. Heartbeats do not cover that window, so a bigger `READ_TIMEOUT` may not help. Measured cause: the backend serialises requests — at `CONCURRENCY=4`, vLLM reported `Running: 1 reqs` in 122 of 150 samples while three conversations queued invisibly. Lower `CONCURRENCY` first; it buys little throughput and costs conversations.
Turns end in `turn_deadline`	One turn ran longer than the 1800 s hard ceiling (`TURN_DEADLINE_SECONDS` in `driver/run_batch.py`). Not an arm key: lower `CONCURRENCY` first; raise the constant only if a slow model writing long documents legitimately needs it, and record the change with the run.
Turns end in `stream_idle`	The backend kept sending keep-alives for 900 s (`STREAM_IDLE_SECONDS`) with no data frame: the agent’s LLM call is wedged while the connection stays alive. Server side, unlike `client_timeout` — look at `vllm.log` around the turn’s timestamp.
Turns end in `transport`, or rows are `driver_error`	`transport`: the connection to the backend dropped mid-stream; check whether `backend.log` ends with a crash. `driver_error`: an exception inside the driver, printed as `[FAIL] <id> driver_error <Exception>` in the `.err` log; the ledger keeps the row and `make resume` retries it.
Ledger shows `n_with_capture_errors > 0`, or a transcript panel is a string such as `HTTP 500`, `ReadTimeout` or `thread not in this job's database`	The turns succeeded but the panel could not be read back. See the capture paragraph under Results: the same job retries, a later job cannot, and the panel is in that attempt’s kept dump.
`FATAL: port … already in use on <node> by another tenant` (exit 12)	The pre-flight check found a socket in the job’s five-port window before anything was started; any TCP socket counts, including a co-tenant’s outbound connection. Resubmit: the window derives from the job id.
`FATAL: vLLM exited during startup` or `FATAL: vLLM did not accept requests within 2400s` (exit 13)	The last 40 lines of `vllm.log` are printed under the message; read them first. Still logging compile / CUDA-graph capture: raise `HAICO_READY_TIMEOUT`. A KV-cache or out-of-memory error: lower `MAX_MODEL_LEN` or `MAX_NUM_SEQS`, raise `GPU_MEMORY_UTILIZATION`, or ask for more GPU memory. `port N was taken by a co-tenant job between the check and the bind`: resubmit. `401 Client Error` / `GatedRepoError`: the model is gated and the job had no `HF_TOKEN` — accept the licence on huggingface.co, then `bash scripts/bootstrap_secrets.sh --hf`. At `TENSOR_PARALLEL_SIZE>1`, rank 0 logs `vLLM is using nccl` and rank 1 never does: an NCCL hang; prefer `TENSOR_PARALLEL_SIZE=1` whenever the model fits one card. `Could not find nvcc`: something in `EXTRA_VLLM_ARGS` reaches a JIT path the compute nodes cannot serve.
`FATAL: backend not healthy` (exit 14)	The backend did not answer `/health` within 240 s; the last 40 lines of `backend.log` follow. `PostgreSQL not reachable`: the entrypoint waits only 30 s for Postgres, so look at `pg.log`. An Alembic traceback: the pinned digest’s migration chain failed against a fresh database — try a newer digest.
`FATAL: Phoenix did not become healthy` (exit 16)	Only with `PHOENIX_ENABLED=1`: the image pull failed or the UI port did not answer `/healthz` in 120 s; the last 30 lines of `phoenix.log` follow. Tracing is optional: `PHOENIX_ENABLED=0` and resubmit.
`FATAL: could not log in as admin to mint an API key` (exit 15)	The image predates migration `0017` and the admin seed failed. The job falls back to logging in as `admin` and calling `POST /api/keys`, so it works against any digest; check `grep -i admin backend.log`. The other two exit-15 messages (`POST /api/keys did not return a usable token`, `the API token does not authenticate`) point at `backend.log` too.
Boxed `WARNING: this backend image predates the step-limit guard` after the image pull	The pinned `BACKEND_DIGEST` was built before the backend detected a step-limit hit, so a turn that exhausts `agent.recursion_limit` is recorded as `ok` and `n_completed` overstates the model. The job continues on purpose (reproducing an old result needs the old image); for a campaign that measures tool-schema adherence, re-pin a newer digest.
Job log shows `[mirror] … db dump FAILED` or `(pg_dump failed — any previous dump left intact)`	`pg_dump` could not connect. At teardown this means Postgres already received a cgroup-wide TERM — a plain `scancel`, `make cancel FORCE=1`, or the wall clock arriving before the driver stopped — and entered smart shutdown. Cancel with `make cancel` (no `FORCE`). The last periodic dump is what you have otherwise; lower `DUMP_INTERVAL` for short or risky runs. Failing every interval from the start is a Postgres problem: read `pg.log`.
Batch job stops in its first second with `TMPDIR is not set` (the gate for the same arm succeeded)	This site’s scheduler does not export a per-job `$TMPDIR`; see assumption 4 under Adapting to another cluster.
`apptainer pull` of `ghcr.io/petrosrapto/haico-backend@sha256:…` fails with a manifest / not-found error	The digest is not one the package has: `submit.sh` only checks the `sha256:` prefix. Resolve it again with `make digest TAG=<tag>` and paste the output verbatim. An `unauthorized` error instead means the stored token lost package access; re-run `make secrets`.
Job requeued from the start with no warning	`PARTITION=yolo` is preemptible with `GraceTime=0`. Use `gpu` for long batches.
`sbatch` appears to succeed but nothing queues	SSH connection multiplexing has been observed to swallow submissions on long-lived sessions. The tooling passes `-o ControlPath=none` throughout; re-check with `make status`.

Get started

How it works

Data & recording

Developer guide

Operations

Reference

SLURM experiments

Prerequisites

From zero to a first result

`make env`

`make secrets`

Run your first experiment

Stopping a run

Changing the experiment

Arm-file reference

Prompt sets

Adapting to another cluster

Choosing a model

How it works

Results

`ledger.json`

`transcripts/<id>.json`

Reading the database dump

Reading the Phoenix traces

Exit codes

Secrets

Why that path

What is in it

Everything else is generated per job and dies with it

How the job handles the one real secret

A second researcher

What differs from haico.gr

Troubleshooting

​Prerequisites

​From zero to a first result

​make env

​make secrets

​Run your first experiment

​Stopping a run

​Changing the experiment

​Arm-file reference

​Prompt sets

​Adapting to another cluster

​Choosing a model

​How it works

​Results

​ledger.json

​transcripts/<id>.json

​Reading the database dump

​Reading the Phoenix traces

​Exit codes

​Secrets

​Why that path

​What is in it

​Everything else is generated per job and dies with it

​How the job handles the one real secret

​A second researcher

​What differs from haico.gr

​Troubleshooting

Prerequisites

From zero to a first result

`make env`

`make secrets`

Run your first experiment

Stopping a run

Changing the experiment

Arm-file reference

Prompt sets

Adapting to another cluster

Choosing a model

How it works

Results

`ledger.json`

`transcripts/<id>.json`

Reading the database dump

Reading the Phoenix traces

Exit codes

Secrets

Why that path

What is in it

Everything else is generated per job and dies with it

How the job handles the one real secret

A second researcher

What differs from haico.gr

Troubleshooting