# Albedo (SN97)

> King-of-the-hill trajectory-distillation subnet on Bittensor (netuid 97, finney).
> Miners fine-tune **Qwen3.6-35B-A3B** models (MoE — 256 experts, 8 active per token;
> multimodal Qwen3VL architecture) and commit them on-chain; a backend pipeline ingests
> commits, validates artifacts, runs a stability pre-eval, then duels each challenger
> against the reigning king on coding trajectories from four datasets — mini-coder (400k),
> mini-coder-rs, open-swe-traces and swe-hero — pooled one source trajectory per instance and stratified by
> phase x bug family. Between turns, each model's shell command is answered by the
> **repo-context service**, which checks out the real repository at the sampled commit and *executes*
> the command against it — so `git`, `grep`, `find`, `sed` and friends return real output, and the
> model's own edits persist through a write overlay. Only what cannot be executed is simulated by an
> LLM (`deepseek/deepseek-v4-flash-0731`, provider-rotating ladder with the evaluator model as the last
> rung), always in whatever observation format that trajectory natively uses. For each task an
> evaluator (GLM 5.2) rolls **three** GLM 5.2 reference trajectories, reduces them to the task's
> **milestones** and writes a ladder of yes/no questions per milestone — generalized to the task, not
> to one run's route. King and challenger each roll every task out **twice**; a judge (GLM 5.2) answers
> every question 1/0 three times per trajectory (majority wins), and each side's score is its mean
> yes-rate over all its trajectories. A trajectory that has collapsed into a command loop is scored
> **0 outright, without being judged**.
> A challenger must beat the king by a **2.5% margin** to win, and must win **two independent
> evals** before it is crowned. Winners are crowned into a
> rolling 5-king chain and earn emissions.
>
> This codebase is the **production backend + miner CLI** — a set of independent,
> PM2-managed services backed by a single Postgres state machine, plus an `albedo`
> miner CLI. (The single-process validator design lives in the separate `albedo-refactor`
> repo; this repo splits that flow across ingest → validate → pre-eval → eval →
> reign → weight services so each stage scales, retries, and recovers independently.)

**Subnet:** netuid `97` (finney mainnet) · **Model class:** Qwen3.6-35B-A3B only · **Reveal format:** `v7|<repo>|<digest>`

---

## Repository layout

```
albedo/
│
├── pyproject.toml              All console entrypoints (see "Services" below) + deps (bittensor, fastapi, asyncpg, opensearch, vllm-via-remote)
├── chain.toml                  Subnet constants for the config_validation commit-validator (seed digest, arch-lock keys, file allowlist). The `[files]` allowlist now matches the live one; the LIVE authority is albedo_config ModelValidationSettings + architecture_spec.json — see below.
├── schema.sql                  Canonical Postgres schema — the single source of truth for pipeline state
├── docker-compose.yml          Local Postgres (albedo-postgres) on a localhost-only port
├── .env.example                Full backend/GPU-host env reference
├── .env.example_miners         Miner-only env (wallet + HF/Hippius creds + namespace)
│
├── miner/                      ← the `albedo` miner CLI (entry point miner.cli:main)
│   ├── cli.py                      arg parsing + dispatch for all subcommands
│   ├── validate.py                 local file-manifest + arch + safetensors-index + 16-bit-dtype checks (same code validators run)
│   ├── upload.py                   push model dir to HF (default) or Hippius, return repo@pin
│   ├── publish.py                  full pipeline: validate→upload→check→registered?→commit
│   ├── commit.py                   write v7 reveal on-chain (set_reveal_commitment)
│   ├── register.py                 burned_register a hotkey on netuid 97
│   ├── check_commits.py            scan chain for v7 commits
│   ├── env.py                      load .env (repo root + cwd) → defaults
│   └── tui.py                      full-screen prompt_toolkit console (`albedo on`)
│
├── src/                        ← backend services (src-layout, installed as packages)
│   │
│   ├── albedo_config/              ALL SETTINGS for every service — one BaseSettings class per role, each with its own
│   │   │                           env prefix (ALBEDO_EVAL_ / ALBEDO_JUDGE_ / ALBEDO_REMOTE_ / SANITY_DISPATCH_ /
│   │   │                           SANITY_REMOTE_ / ALBEDO_REPO_CONTEXT_ / CHAIN_ / ALBEDO_WEIGHT_ / …).
│   │   │                           `.env` carries only secrets + topology; every tunable has a code default here.
│   │   ├── config.py                   the settings classes (incl. the live file allowlist on ModelValidationSettings)
│   │   ├── chain_spec.py               chain.toml → constants (seed digest, arch-lock keys, SIM_THRESHOLD)
│   │   ├── models.py                   the model roster: JUDGE_MODELS, EVALUATOR_MODEL, SOTA_MODELS, SIMULATION_MODEL, ENGY_MODELS
│   │   └── db.py                       shared DSN helpers
│   │
│   ├── chain_reader/               INGEST: poll chain → write chain_commits + model_submissions(SUBMITTED)
│   │   ├── reader.py                   async poll loop (get_current_block every CHAIN_POLL_INTERVAL_S)
│   │   ├── chain.py                    decode v7 payloads, resolve hotkey→uid via metagraph
│   │   └── db.py                       upsert chain_commits / miners / model_submissions / events
│   │
│   ├── config_validation/          VALIDATION LIBRARY for the standalone commit-validator; also gives the miner CLI its Hippius download/list utils + ModelRef (the miner's actual checks come from model_validation, below)
│   │   ├── pipeline.py                 ordered checks: revision → files → architecture
│   │   ├── checks/revision.py          repo@digest resolves on Hippius (fast fail)
│   │   ├── checks/files.py             strict allowlist from chain.toml [files]
│   │   ├── checks/architecture.py      config.json matches seed on lock keys; no auto_map/quantization_config
│   │   ├── checks/genesis_metadata.py  the metadata-hash pin against the genesis repo
│   │   ├── repo_pattern.py             challenger repo-name rule from chain.toml
│   │   ├── storage/                    the download layer shared with the miner CLI: dispatch.py picks the backend from the
│   │   │                               pin (_hf.py / _hippius.py), _fastdl.py is the parallel fetcher, _supervise.py the
│   │   │                               killable-subprocess + stall watchdog, s3.py the artifact store
│   │   └── models/reveal.py            parse/build v7 reveal
│   │
│   ├── chain_guard/                HOTKEY-REUSE GUARD (imported by chain_reader): a `used_hotkeys` ledger seeded from every hotkey committed before CHAIN_START_BLOCK (all reveal versions); a hotkey is burned into the ledger after its submission finishes eval, so a hotkey can't re-submit to game the duel
│   │   ├── scan.py                     scan_all_raw(): iterate every RevealedCommitment on the netuid
│   │   ├── swap.py                     detect `swap_hotkey` on the metagraph (find_swaps/confirm_swaps against a pinned snapshot)
│   │   ├── db.py                       seed ledger from DB + record a hotkey after eval; ledger swaps + registration blocks
│   │   └── uploads.py                  publish the guard ledger for the dashboard
│   │
│   ├── model_validation/         VALIDATE WORKER: claim submissions → download → validate → dedup → persist (the live file/arch/index/dtype authority for miners + validators)
│   │   ├── validate_worker.py          end-to-end per-model: files→hashes→dtype-preflight→download→index→arch→sketch fingerprint→dedup gate
│   │   ├── validate/repo.py            the LIVE strict file allowlist — REQUIRED_FILES / ALLOWED_FILES / globs on
│   │   │                               `albedo_config` ModelValidationSettings (there is no model_validation/config.py any more)
│   │   ├── validate/genesis_files.py   sha256 of every metadata file pinned to the genesis repo revision (metadata_hash)
│   │   ├── validate/chat_template.py   chat_template.jinja + the copy embedded in tokenizer_config.json (chat_template_hash)
│   │   ├── validate/architecture.py    spec-driven arch lock (validate/architecture_spec.json — no hardcoded family)
│   │   ├── validate/dtype.py           16-bit weight check: every safetensors shard must be F16/BF16 (rejects quantized / F32 / F64)
│   │   ├── validate/safetensors_index.py  shard/index consistency: model.safetensors.index.json weight_map vs on-disk shards + tensors
│   │   ├── storage/preflight.py        header-only dtype preflight via HTTP Range (reject non-16-bit before the full download)
│   │   ├── storage/download.py         backend-agnostic fetch (HF or Hippius, chosen by the pin's shape) with a stall watchdog
│   │   ├── dedup/                      the similarity gate (see Hippius validation): sketch.py (secret-seeded per-tensor sketch ψ·W·ω + weights at
│   │   │                               secret sample positions), signals.py (distances / spectra / merge fits, all computed on sketches), verdict.py
│   │   │                               (COPY, OWN-COPY exact; LINEAR-COMBO, NOISE-COPY, NOISED-COPY, SPARSE-EDIT, TRIVIAL-EDIT heuristic), gate.py
│   │   │                               (run + enforcement), bank.py (OpenSearch kNN bank of accepted fingerprints), canon.py / layout.py / secret.py
│   │   ├── uploads/artifacts.py        publish fault.json to S3 (best-effort)
│   │   └── db.py                        state machine + lease/heartbeat (SUBMITTED→HIPPIUS_VALIDATED / TERMINAL_INVALID)
│   │
│   ├── sanity_service/             PRE-EVAL DISPATCHER: stability gate before spending GPU eval hours
│   │   ├── dispatcher.py               claim PRE_EVAL_QUEUED → dispatch to GPU worker → heuristics → judge → cache verdict
│   │   ├── checks.py                   per-response heuristics (empty / truncated / unclosed <think> / length / repetition /
│   │   │                               encoding / vocab) + across-response ones (collapsed, uniform length, code present)
│   │   ├── tail_check.py               tail + loop heuristic on the generated trajectory (duplicate-command ratio,
│   │   │                               longest consecutive run) — the pre-eval counterpart of shared/loop_check.py
│   │   ├── llm_check.py                injection probe + viability probe, judge quorum (≥2 resolved)
│   │   ├── judge_panel.py              concurrent OpenRouter judge calls
│   │   ├── rubric.py                   probe rubric wiring
│   │   ├── rubricisity.py              **gitignored on purpose** (`.gitignore`) — the probe/viability system prompts.
│   │   │                               `rubric.py` and `tail_check.py` import it, so a fresh clone cannot start the
│   │   │                               pre-eval dispatcher until this file is supplied out of band.
│   │   ├── dataset.py                  deterministic prompt sampling from the same shard manifest the duel uses
│   │   ├── uploads.py                  fault.json for terminal miner-fault rejections
│   │   └── db.py                       PRE_EVAL_* state machine + sanity_results cache
│   │
│   ├── sanity_remote/              PRE-EVAL GPU WORKER (stateless; no DB/dataset/keys)
│   │   ├── api.py                      FastAPI :<port> — POST /sanity-runs, GET status/events (bearer auth)
│   │   └── worker.py                   warm vLLM, generate on sampled prompts, run heuristics, return result
│   │
│   ├── albedo_eval_service/        THE DUEL: backend coordinator + remote GPU eval + judge + score bridge
│   │   ├── judge_api.py                judge/scoring API :<port> (/category-prep questions, /simulate-observation, /score-batch) — calls OpenRouter
│   │   ├── judge_core.py               judge prompts/schemas/parsers, per-response scoring, CHALLENGER_WIN_MARGIN, margin aggregation
│   │   ├── judge_llm_client.py         OpenRouter client: per-model semaphore, retry+backoff, provider rotation, forced json_schema (fp8)
│   │   ├── control/                    BACKEND SIDE
│   │   │   ├── dispatcher.py               claim EVAL_QUEUED (advisory lock) → pick EVAL host → stream verdict
│   │   │   ├── api.py                      backend status API :<port> (/health /ready /submissions/{id})
│   │   │   ├── repository.py               the eval state machine in SQL (host claim, win-both re-queue, verdict latch)
│   │   │   ├── requeuer.py                 PRE_EVAL_PASSED→EVAL_QUEUED + retryable eval re-queue
│   │   │   ├── remote_client.py            HTTP client for the GPU host's control plane
│   │   │   ├── artifacts.py                artifact rows + S3/R2 URI handling
│   │   │   └── notifications.py            verdict/event payload shapes
│   │   ├── remote/                     GPU-HOST SIDE
│   │   │   ├── api.py                      control plane :<port> (/eval-runs, /model-prefetch, /capacity, WS /score-bridge)
│   │   │   ├── worker.py                   run eval: load samples → x ROLLOUTS_PER_SAMPLE → one thread per trajectory against vLLM king+chal → score over bridge → verdict (win decided here)
│   │   │   ├── generation.py               one `vllm serve` per model (CUDA_VISIBLE_DEVICES per side); requests are independent, so no trajectory waits for another
│   │   │   ├── dataset.py                  load manifest samples, render transcripts (incl. tool_calls) into prompt text
│   │   │   ├── artifacts.py                upload generated-samples/scoring-results JSONL + verdict.json to S3/R2
│   │   │   ├── prompt_remote.py            candidate-side system prompt
│   │   │   └── state.py                    in-process run state machine (accepted→generating→scoring→succeeded)
│   │   ├── scoring/                    score_bridge.py (hub) + score_bridge_client.py (backend) + scoring_client.py (batch builder)
│   │   ├── evaluator/                  QUESTION GENERATION
│   │   │   ├── reference/                  prompt_milestones.py (vector of change + validate_vector), prompt_ladder.py
│   │   │   │                               (questions per milestone), leak filter + normalise_span
│   │   │   └── shared/                     label enforcement, sample_phase, HORIZON_STRATA
│   │   ├── judge/prompt_judge.py       the judge-side prompt assembled by judge_core
│   │   ├── simulator/prompt_simulator.py   simulator system prompt, per-format OUTPUT FORMAT block, transcribe prompt
│   │   ├── modelstore/                 canonical_model_config.py (pin genesis configs over the challenger's — anti-tamper) + resolver.py (cache layout)
│   │   └── shared/                     cross-role helpers
│   │       ├── sampling.py                 deterministic sampling: unique instance_ids pooled across sources, one source trajectory each, phase x bug-family strata
│   │       ├── observation_format.py       format detect/validate/repair + command output contracts (must_print / may_be_silent / not_derivable)
│   │       ├── loop_check.py               looped-trajectory detector — a looped side is scored 0 without calling the judge
│   │       ├── dataset_manifest.py         load + sha256-verify the dataset shard manifest
│   │       ├── faults.py                   fault taxonomy (MINER/INFRA/REMOTE_EVAL/PROVIDER/UNKNOWN)
│   │       └── models.py / json_extract.py shared request/response models + tolerant JSON parsing
│   │
│   ├── repo_context_service/       GROUNDING: executes the candidate's command against a snapshot of the real repo at the sampled commit
│   │   ├── api.py                      :<port> POST /repo-context (grounding block + exact_output/exact_returncode), /prefetch, /health
│   │   ├── core.py                     snapshot fetch/cache, command routing, COMMAND OUTPUT / GIT SEMANTICS / CHAIN EVIDENCE blocks
│   │   ├── command_search.py           parser+executor for find/grep/ls/sed/cat and friends (BRE→Python, -prune/-o, POSIX classes)
│   │   ├── overlay.py                  in-memory write overlay so a candidate's own edits (incl. sed -i) persist across turns
│   │   └── git_sim/                    git subset executed against the snapshot: parse, diffs, patches, session, render, chain
│   │
│   ├── set_reign_worker/           CORONATION: EVAL_WIN → promote into 5-slot king chain → create weight_epochs
│   │   └── service.py                  reign/reign_members/king_versions writes, weight_bps split
│   │
│   └── weight_setter/              WEIGHTS: consume weight_epochs → subtensor.set_weights → PERIODIC_REFRESH
│       └── service.py                  rate-limited by ALBEDO_WEIGHT_SET_RATE_BLOCKS, burn UID on no-king
│
├── pm2/                        One ecosystem.*.config.js per long-running / cron process (incl. ecosystem.monitor.config.js, ecosystem.eval-cache-cleanup.config.js, ecosystem.model-gc.config.js)
│
├── scripts/
│   ├── create_genesis_king.py      seed genesis king_version + reign + reign_members (UID 0)
│   ├── generate_arch_spec.py       regenerate architecture_spec.json from the genesis config (handles nested multimodal MoE via text_config)
│   ├── dedup_seed_bank.py          seed the dedup bank from a manifest of accepted models (dedup_bank_manifest.example.json)
│   ├── eval_cache_cleanup.py       king-aware model-cache GC (keep active kings + seed + in-flight, drop the rest) — PM2 loop
│   ├── cleanup_models.sh           hourly disk reclaim: delete model snapshots older than 4h from ALBEDO_MODEL_CACHE_DIR — PM2 cron
│   ├── full_flow_test.py           end-to-end: real chain commits → full validation pipeline
│   ├── setup_opensearch.sh         single-node OpenSearch container (:<port>, security off)
│   └── install_deps.sh             install opensearch-py + config_validation into a venv
│
├── website/                    Static dashboard (reads data/dashboard.json + data/state.json); this llms.txt lives here
│   ├── index.html / detail.html    reign · live queue (3-stage pipeline) · chart · fails; per-eval verdict/artifacts
│   ├── js/                          config.js, data.js (normalize), fetch.js, model.js (model names), render/* (reign,chart,history,pipeline)
│   ├── monitor.py                   PM2 service: DB → data/dashboard.json + data/state.json → upload to Hippius (on-change)
│   └── push_to_hippius.py           one-shot upload of the static site + data/*.json to Hippius S3 (no-cache, public-read)
│
├── docs/
│   ├── MINING.md                   miner-facing guide for the `albedo` CLI
│   ├── SCORING.md                  checklist construction, loop short-circuit, win margin, anti-gaming
│   ├── DATASETS.md                 the four corpora, observation formats, grounding + simulator ladder, sampling, manifest pin
│   ├── eval-service-status.md      eval-stack runbook (services, ports, PM2, smoke mode)
│   └── reign-and-weight-pm2.md     set-reign + weight-setter dev notes
│
└── tests/                      pytest suite (+ tests/integration needs ALBEDO_TEST_DATABASE_URL)
```

---

## The pipeline (one submission, end to end)

Each model submission flows through a Postgres state machine (`model_submissions.state`). Every
stage is an independent service that **claims** work, holds a **lease**, and either advances the
state or marks it retryable/terminal. This is the backbone of the whole repo.

```
on-chain v7 commit
  │
  ▼  chain_reader            poll chain → chain_commits + model_submissions(SUBMITTED)
SUBMITTED
  │
  ▼  model_validation      download → file manifest → arch lock → sketch fingerprint → dedup gate
HIPPIUS_RUNNING → HIPPIUS_VALIDATED          (fail → TERMINAL_INVALID / HIPPIUS_RETRYABLE)
  │
  ▼  requeuer                HIPPIUS_VALIDATED → PRE_EVAL_QUEUED
PRE_EVAL_QUEUED
  │
  ▼  sanity_service (+sanity_remote GPU)   generate on N prompts → heuristics + injection/viability judges
PRE_EVAL_RUNNING → PRE_EVAL_PASSED           (fail → TERMINAL_INVALID / PRE_EVAL_RETRYABLE)
  │
  ▼  requeuer                PRE_EVAL_PASSED → EVAL_QUEUED
EVAL_QUEUED
  │
  ▼  albedo_eval_service     dispatcher claims (advisory lock) → remote GPU duel → judge → verdict
EVAL_RUNNING → EVAL_WIN  or  COMPLETE_LOSS    (fail → EVAL_RETRYABLE)
  │            ↑
  │            └─ win-both: the FIRST win does not crown. `control/repository.py` sends the submission
  │               back to EVAL_QUEUED with priority=0, and only a SECOND independent eval win
  │               (a second SUCCEEDED eval_run with challenger_won) becomes EVAL_WIN for real.
  │ (second win only)
  ▼  set_reign_worker        promote challenger into 5-slot king chain → write weight_epoch(CORONATION)
SET_REIGN_RUNNING → REIGN_SET
  │
  ▼  weight_setter           consume weight_epochs → subtensor.set_weights → COMPLETE_CORONATED
WEIGHT_SET_RUNNING → COMPLETE_CORONATED
```

**Terminal states:** `COMPLETE_CORONATED` (won + crowned), `COMPLETE_LOSS` (eval'd, didn't dethrone),
`TERMINAL_INVALID` (miner fault — bad artifact/arch/dup/injection), `TERMINAL_INFRA_FAILED` (gave up after retries).

**Recovery pattern (every stage):** a *sweeper* marks expired leases `*_RETRYABLE`; a *requeuer* moves
retryable rows back to the queued state; the *dispatcher* only ever claims the queued state. Crashes
mid-stage are recovered by lease expiry, not by in-memory state.

---

## chain.toml — subnet constants

Mirrors the subnet rules for the standalone `config_validation` commit-validator. Arch-lock keys, seed
digest and the `[files]` allowlist are all current, and the `[files]` block now agrees
with what miners and the backend actually enforce (the one difference: the live allowlist also tolerates
`LICENSE`). The live file-manifest authority is `ModelValidationSettings` in `albedo_config/config.py`,
read through `model_validation/validate/repo.py`; the arch authority is `validate/architecture_spec.json`
(see *Validation internals* below).

```toml
[chain]
name         = "Albedo"
seed_repo    = "teutonic/qwen3.6-35b-a3b-genesis"
repo_pattern = "^[^/]+/albedo-qwen3\\.6-35b-.+$"     # challenger naming, any namespace

[arch]                                                # capacity keys, must match genesis exactly
extra_lock_keys = [
  "max_position_embeddings", "tie_word_embeddings", "rope_theta",
  "hidden_size", "num_hidden_layers", "num_attention_heads",
  "num_key_value_heads", "intermediate_size", "head_dim",
  "moe_intermediate_size", "shared_expert_intermediate_size",   # MoE capacity
  "num_experts", "num_experts_per_tok",
]
# vocab_size + model_type are always locked (config_validation COMPAT_KEYS)

[seed]
seed_digest = "sha256:efd5b8d0a1c1f472be56ff919419cdd0561bdecd9013d5c2a96dd0e23e89c165"

[files]                                               # strict allowlist
required            = ["config.json", "generation_config.json", "tokenizer_config.json",
                       "tokenizer.json", "chat_template.jinja", "preprocessor_config.json",
                       "video_preprocessor_config.json"]
require_safetensors = true
allowed             = ["model.safetensors.index.json", ".gitattributes", "README.md"]
allowed_globs       = ["model-*-of-*.safetensors", "model.safetensors"]
forbidden_globs     = ["*.py"]                        # no custom modeling code
```

> **Live manifest (what's actually enforced)** — `ModelValidationSettings.REQUIRED_FILES` in
> `albedo_config/config.py` **requires**
> `config.json`, `generation_config.json`, `tokenizer_config.json`, `tokenizer.json`,
> `chat_template.jinja`, `preprocessor_config.json`, `video_preprocessor_config.json`;
> **allows** only `model.safetensors.index.json`, `.gitattributes`, `LICENSE`, `README.md`.
> Anything else — including `merges.txt`, `vocab.json`, `configuration.json`,
> `special_tokens_map.json` — is an unexpected extra → `file_manifest` rejection. The arch lock
> compares against `architecture_spec.json` (regenerated from the genesis by `generate_arch_spec.py`),
> which pins `architectures = ["Qwen3_5MoeForConditionalGeneration"]`, `model_type = "qwen3_5_moe"`,
> `vocab_size = 248320`, the capacity keys above, and the MoE keys (`num_experts = 256`,
> `num_experts_per_tok = 8`, `moe_intermediate_size = 512`, `shared_expert_intermediate_size = 512`).

---

## Mining (the `albedo` CLI)

Albedo is king-of-the-hill for Qwen3.6-35B-A3B. Fine-tune → upload to HuggingFace (default) or Hippius
(`ALBEDO_MODEL_BACKEND=hippius`) → commit a v7 reveal on-chain. A model that passes validation **and**
beats the king by the **2.5% win margin** in **two separate evals** earns emissions. Full guide: [docs/MINING.md](../docs/MINING.md);
how the scores themselves are built: [docs/SCORING.md](../docs/SCORING.md); what they are scored on: [docs/DATASETS.md](../docs/DATASETS.md).

### Install

```bash
cd ~/albedo
python3 -m venv .venv && source .venv/bin/activate
pip install -e .                 # installs the `albedo` console script
pip install -e '.[train]'        # optional: trl + accelerate + deepspeed for SFT/RL
```

### Configure (`cp .env.example_miners .env`)

| Key | Purpose |
|---|---|
| `ALBEDO_COLDKEY` / `ALBEDO_HOTKEY` | wallet identity (skip `--coldkey/--hotkey`) |
| `ALBEDO_WALLET_PATH` | only if wallets aren't in `~/.bittensor/wallets` |
| `CHAIN_NETUID` / `CHAIN_NETWORK` | default `97` / `finney` (use `test` for testnet) |
| `HIPPIUS_HUB_TOKEN` | Hippius auth (or `HIPPIUS_HUB_USERNAME`/`_PASSWORD`) |
| `ALBEDO_NAMESPACE` | your Hippius namespace (skip `--namespace`) |
| `ALBEDO_REPO_PREFIX` | leave as `albedo-qwen3.6-35b` |

### Commands

```bash
albedo register                                   # one-time: burned_register on netuid 97
albedo check-model --path /path/to/model          # local validate (free) — must say VALID
albedo publish --path /path/to/model --name v1    # validate→upload→check→registered?→commit
albedo check-commit --hotkey 5F...                # confirm your v7 commit landed
albedo on                                         # interactive TUI
```

`publish` runs all five steps and prompts before writing on-chain (`--yes` to skip, `--skip-commit`
to stop after upload). Individual steps also exist: `upload`, `commit`, `check-commit`.

### What gets a model rejected

These are the **live** checks (`model_validation`), run by both the local CLI and the validator
except where noted. Each maps to a `fault_code`.

1. Repo name doesn't match `^[^/]+/albedo-qwen3\.6-35b-.+$`
2. File set violates the allowlist — **missing** a required file (`config.json`, `generation_config.json`,
   `tokenizer_config.json`, `tokenizer.json`, `chat_template.jinja`, `preprocessor_config.json`,
   `video_preprocessor_config.json`), any `*.py`, or an unexpected extra such as `merges.txt`,
   `vocab.json`, `configuration.json` (`file_manifest`)
3. No `*.safetensors` (`file_manifest`)
4. **Metadata not byte-identical to genesis** — sha256 of every metadata file is pinned to the
   genesis repo revision (`dendriteholdings/albedo-qwen3.6-35b-king-genesis@d7934c55…`):
   `config.json`, `generation_config.json`, `preprocessor_config.json`, `tokenizer_config.json`,
   `tokenizer.json`, `video_preprocessor_config.json` (`metadata_hash`); `chat_template.jinja` —
   and the chat_template string embedded in `tokenizer_config.json` — has its own pin
   (`chat_template_hash`). Copy these files from genesis verbatim; a single changed byte rejects.
   Only `model.safetensors.index.json` may differ (re-sharding), checked structurally instead
5. **Weights aren't 16-bit** — every safetensors shard must be F16/BF16; quantized / F32 / F64 is
   rejected (`weight_dtype`)
6. **Safetensors index inconsistent** — a sharded checkpoint's `model.safetensors.index.json`
   weight_map must match the shards + tensors actually on disk (`safetensors_index`)
7. `config.json` doesn't match the genesis arch spec on the lock keys (incl. the MoE keys), or
   contains `auto_map` (remote code) / `quantization_config` (quantized) (`architecture`)
8. **Duplicate** — weights identical to an accepted model, byte for byte or up to float noise, fault
   as `duplicate` (another miner's model — permanently blocks the hotkey) or `duplicate_own`; a copy
   whose only change from an accepted model is noise faults as `duplicate_heuristic` (`NOISE-COPY` /
   `NOISED-COPY`). *Not* checked locally — the sketch bank lives in `model_validation`. Make your
   model genuinely trained, not perturbed

> **Note:** local `check-model --path` runs the file manifest + arch + safetensors index +
> 16-bit dtype checks, but **not** the metadata-hash checks (#4) and **not** dedup (#8) — both run
> only in `model_validation`; the `--repo/--digest` remote check runs only the file manifest + arch
> (it lists the repo files and fetches `config.json`). Passing locally does not guarantee
> acceptance — in particular, verify your metadata files are byte-identical to genesis yourself
> (`sha256sum` them against the genesis repo).

---

## Services & ports

Console entrypoints (from `pyproject.toml`) — each has a matching `pm2/ecosystem.*.config.js`.

| Entrypoint | Role | Host | Port / cron |
|---|---|---|---|
| `chain-reader` | poll chain → DB (honors `CHAIN_START_BLOCK` — commits below it are skipped) | backend | — |
| `model-validation` | validate worker | PRE_EVAL GPU box (co-located with OpenSearch + sanity-remote; shares its model cache with the sanity worker via `ALBEDO_MODEL_CACHE_DIR` = `CV_MODEL_CACHE_DIR`) | — |
| `sanity-dispatcher` | pre-eval coordinator (PM2 `albedo-sanity-dispatcher`, plus `--reconcile-running` / `--sweep-abandoned` crons) | backend | — |
| `sanity-remote` | pre-eval GPU worker | PRE_EVAL GPU box | :<port> |
| `albedo-eval-api` | backend status API (PM2 app is named `albedo-eval-backend-api`) | backend | :<port> |
| `albedo-judge-api` | judge / scoring / observation simulation | backend | :<port> |
| `albedo-repo-context-api` | grounding: executes commands against the repo snapshot at the sampled commit | EVAL GPU box | :<port> |
| `albedo-score-bridge` | WS bridge → judge | backend | — |
| `albedo-eval-dispatcher` | claim + run evals | backend | — |
| `albedo-eval-dispatcher --reconcile-running` | replay active runs | backend | cron 1m |
| `albedo-eval-dispatcher --sweep-abandoned` | expire stale leases | backend | cron 1m |
| `albedo-eval-requeuer` | retryable → queued (+ the win-both re-queue) | backend | cron 1m |
| `albedo-remote-eval-api` | GPU eval control plane | EVAL GPU box | :<port> |
| `set-reign-worker` | coronation | backend | — |
| `weight-setter` | set_weights on-chain | backend | — |
| `website/monitor.py` (PM2 `albedo-dashboard-monitor`) | publish dashboard.json + state.json → Hippius | backend | on-change poll (~2s) |
| `scripts/eval_cache_cleanup.py` (PM2 `albedo-eval-cache-cleanup`) | king-aware model-cache GC | backend | ~60s loop |
| `scripts/cleanup_models.sh` (PM2 `albedo-model-gc`) | disk reclaim (delete >4h-old model snapshots) | model-cache box | cron hourly |
| `scripts/dataset_creator/pipeline.py --watch` (PM2 `albedo-dataset-creator`) | harvest finished eval trajectories into a HF dataset | backend | watch loop |
| `website/registration_history.py` (PM2 `albedo-registration-history`) | registration-history series for the dashboard | backend | poll loop |

The dashboard monitor and the two cleanup jobs are the PM2 processes that are **not** `pyproject`
console entrypoints — they run standalone scripts (`website/monitor.py`, `scripts/eval_cache_cleanup.py`,
`scripts/cleanup_models.sh`; the monitor is like `push_to_hippius.py`). See *Dashboard publishing* below.

Full runbook (env, one-time DB setup, smoke mode, health checks): [docs/eval-service-status.md](../docs/eval-service-status.md).

---

## The duel (eval service internals)

```
albedo-eval-dispatcher (loop, every dispatch_poll_seconds)
  pg_try_advisory_xact_lock('full_eval')        ← serialize: one full eval at a time
  claim one model_submissions.state = EVAL_QUEUED
  pick remote_gpu_hosts WHERE role='EVAL' AND state='READY' AND free_gpu_count >= 8
                        ORDER BY free_gpu_count DESC, last_heartbeat_at DESC  (FOR UPDATE SKIP LOCKED)
  build EvalRequest:
    king + challenger model refs
    dataset_sample_ids  ← multi_source_manifest_sample_ids(manifest, block_hash=…, sample_count=…)
    dataset_manifest_hash, judge_config_hash
  POST /eval-runs on the GPU host  → remote_run_id
  follow GET /eval-runs/{id}/events until a "verdict" event
    ~1/min: peek next EVAL_QUEUED → POST /model-prefetch (GPU host pre-downloads the next challenger)
  record verdict → EVAL_WIN | COMPLETE_LOSS | EVAL_RETRYABLE (faults.classify_failure_verdict)

albedo-remote-eval-api  (GPU host)
  POST /eval-runs → spawn RemoteEvalWorker (background)
    state: accepted → generating → scoring → succeeded|failed
    question-prep (early): POST /category-prep over the bridge → GLM 5.2 rolls ALBEDO_JUDGE_REFERENCE_RUNS (3) reference trajectories, extracts the task's milestones and writes the question ladder (async; see Question prep below)
    GPU split: king = ALBEDO_REMOTE_PREVIOUS_KING_GPU_IDS (0,1,2,3)
               challenger = ALBEDO_REMOTE_CHALLENGER_GPU_IDS (4,5,6,7)  (no overlap, 4 each)
    one `vllm serve` per side (tensor_parallel_size = #gpus; ports ALBEDO_REMOTE_PREVIOUS_KING_VLLM_PORT / _CHALLENGER_VLLM_PORT)
    every sample is rolled out ALBEDO_REMOTE_ROLLOUTS_PER_SAMPLE (2) times per side — ids `<sample>#r1`, `#r2`; the judge is sent the
      dataset id, so all rollouts of a sample are judged against its one checklist
    one thread per trajectory: generate a turn → fetch its observation → next turn, with no lock-step barrier; the engine batches
      whatever is in flight and keeps decoding while other trajectories wait on their observations
    multiturn trajectories: the horizon is stratified per sample — HORIZON_STRATA (12, 16)
      assistant turns, assigned round-robin within each phase bucket (evaluator/shared/questions.py);
      ALBEDO_REMOTE_TRAJECTORY_ASSISTANT_TURNS (8) is only the fallback when no horizon is assigned.
      Between turns the observation comes back over the score bridge (POST /simulate-observation
      → judge API), which answers in this order:
        1. contract short-circuit — a command whose tool is absent in this environment gets its
           canonical refusal output, no model call at all
        2. GROUNDED — the repo-context service executes the command against a real checkout of the
           repository at the sampled commit; if it yields exact output + returncode, that IS the
           observation (no LLM involved). git/grep/find/ls/sed/cat are covered, the candidate's own
           writes persist in an overlay, and multi-stage `&&` chains report per-stage evidence
        3. TRANSCRIBE — if grounding produced a COMMAND OUTPUT block but not an exact result, the
           simulator is handed just `$ <command>` and told to transcribe that block
        4. SIMULATE — otherwise the LLM ladder: ALBEDO_JUDGE_SIMULATION_MODEL over one rung per
           provider in ALBEDO_JUDGE_SIMULATION_PROVIDERS (rotated, later rungs forced through
           OpenRouter), then the evaluator model as the last rung. Candidates are ranked against the
           command's output contract (must_print / may_be_silent / not_derivable) and the best-ranked
           one is kept; a must-print command that came back empty is re-asked
      The observation is always emitted in the format that trajectory natively uses
      (<returncode> / OBSERVATION: / OpenHands trailer), detected per sample — see docs/DATASETS.md
    score: build batches → send up to ALBEDO_REMOTE_SCORING_BATCH_CONCURRENCY (128) batches concurrently over the WebSocket /score-bridge (multiplexed by request_id)
    emit "verdict" event (scores, win_margin, challenger_won, vllm/judge error counts, artifact URIs)
    upload request.json, progress/generated-samples/scoring-results JSONL, remote-logs.txt + verdict.json to S3

score bridge (backend-initiated WebSocket, GPU→backend)
  remote sends {type:"score_request", request_id, payload}
  albedo-score-bridge forwards payload → albedo-judge-api POST /score-batch (or /category-prep)
  replies {type:"score_response", request_id, body}
```

**Question prep (per sample)** (`/category-prep`, evaluator = OpenRouter `z-ai/glm-5.2`, fp8): before
judging, `ALBEDO_JUDGE_REFERENCE_RUNS` (3) reference trajectories (GLM 5.2) are rolled concurrently
through the same observation loop the candidates face. They share one world — the observation memo
keys on (sample_id, format, repo state, command) — so they diverge only where the model chose
differently. Below two usable runs the sample is dropped; there is no task-only fallback checklist.
The evaluator then builds the checklist in two stages, each read several times and merged, because
one reading of the same runs comes back with a different subset of the facts:

- **vector of change** (`evaluator/reference/prompt_milestones.py`, `vector_merge.py`) —
  `ALBEDO_JUDGE_MILESTONE_READINGS` (4) independent extractor calls each read all runs at once and
  reduce them to an ordered list of task-level **milestones**, each of exactly one category
  (`claims` / `explore` / `action` / `verification`), justified for necessity by counterfactual (never
  by majority vote) and backed by a verbatim span per consenting run tagged with where that span came
  from. `validate_vector` enforces this in code on every reading: a milestone survives only if it is
  necessary, not already given by the TASK block, and evidenced by a span that really occurs in the
  kind of block its `source` claims. The readings are then aligned by fact (exact duplicates first, one
  aligner call for the rest) and **unioned**: a fact any reading found is kept once, under its
  best-evidenced wording, with the evidence pooled.
- **question ladder** (`prompt_ladder.py`) — `ALBEDO_JUDGE_QUESTION_READINGS` (3) writer calls each
  turn the whole vector into a SET of questions per milestone at a spread of depths — `RUNGS_MAX` (6)
  per milestone, `QUESTIONS_MAX` (60) overall. Questions are aligned by what they test and kept by
  **majority** (asked by at least 2 of the 3 readings, earliest wording), thin milestones topped up from
  the spares; nothing is re-asked. Where the runs reached a milestone through different code, or only
  one run reached it, the question may name no file, function or expression at all. Every question
  weighs the same (`TAG_WEIGHTS` is uniform); there is no separate behaviour regime any more.

The result is then filtered: reference leaks dropped (`filter_reference_leaks`), inaction-earning
"avoids X" checks dropped (`enforce_question_labels`), and finally **pruned against every reference
run** — each run is judged on the finished checklist and a question not one of them earns is dropped.
A sample is rejected outright if fewer than `QUESTION_FLOOR` (6) questions survive. Every discard is
recorded in the scoring artifact. Questions are generated once per task and reused for both models
and all their rollouts.

**Judge** (`albedo-judge-api`, OpenRouter): the judge model (`z-ai/glm-5.2`) answers every question
**1/0** (forced `json_schema`, fp8) `ALBEDO_JUDGE_JUDGE_REPEATS` (3) times per trajectory, seeing only
the questions + one side's trajectory — never the other side's; a question's answer is the majority of
the repeats (the record carries `repeats`, `repeats_held`, `disputed`). Each side rolls every sample out
`ALBEDO_REMOTE_ROLLOUTS_PER_SAMPLE` (2) times and every rollout is judged against the sample's one
checklist. King and challenger are scored **independently**: the mean yes-rate over all of a side's
trajectories → an absolute score in `[0,1]` per side (`score_king` and `score_challenger` are
independent — they do NOT sum to 1). The challenger takes the eval only if
`score_challenger − score_king ≥ CHALLENGER_WIN_MARGIN` (`0.025`, margin-only, decided on the GPU box
in `remote/worker.py`); otherwise `COMPLETE_LOSS`. Winning once is not enough — see the win-both step
in the pipeline above. Per-model concurrency is capped by a semaphore shared across concurrent
score batches (`ALBEDO_JUDGE_MAX_CONCURRENCY_PER_MODEL`) with retry + backoff; an eval needs
`min_valid_fraction` (0.8) of samples scored on both sides or it's a provider fault.

**Degenerate trajectories are not judged.** Before a side is sent to the judge, `shared/loop_check.py`
scans the `CANDIDATE OUTPUT` blocks of the document it would see. If the duplicate-command ratio is
≥ 0.5 or the same command repeats ≥ 4 times consecutively, that side is short-circuited: every question
is answered `0`, the explanation names the looping commands with their repeat counts, and no judge call
is made (`parse_ok` stays true, and the record carries `looped`, `loop_reasons`, `loop_commands`). The
same rule already applies at pre-eval via `sanity_service/tail_check.py`. Truncated output is
short-circuited the same way.

**Deterministic sampling:** `shared/sampling.py` (`multi_source_manifest_sample_ids`) draws `sample_count`
(100) **unique `instance_id`s** pooled across all four sources, one **random source trajectory** per instance,
stratified by `STEP_TRIM` phase (`cold` 65 / `pre_edit` 15 / `at_edit` 20, anchored on the
instance's `first_edit`) x `FAMILY_MIX` bug family (`pr` 50 / `lm` 15 / `combine` 10 /
`mechanical` 25), with `REPO_CAP` 2 per repo, 30% non-`python`, and prefixes over
`MAX_PREFIX_CHARS` (54k) skipped — seeded by the commit `block_hash`, so the same block always
yields the same eval set. The prefix ends on a user turn and the model generates the next assistant
turn; IDs are `shard:row:turn`. Full detail: [docs/DATASETS.md](../docs/DATASETS.md).

---

## Validation internals (model_validation)

```
claim model_submissions oldest-first  (SUBMITTED | HIPPIUS_RETRYABLE)  → HIPPIUS_RUNNING
  1. file manifest    albedo_config ModelValidationSettings allowlist (required / allowed / forbidden globs)
  2. chat template    sha256 of chat_template.jinja — AND of the chat_template embedded in
                      tokenizer_config.json — must match the genesis pins (chat_template_hash)
  2.5 metadata hashes every other metadata file byte-identical to the genesis repo
                      (validate/genesis_files.py — sha256 pinned to
                      dendriteholdings/albedo-qwen3.6-35b-king-genesis@d7934c55: config.json,
                      generation_config.json, preprocessor_config.json, tokenizer_config.json,
                      tokenizer.json, video_preprocessor_config.json → metadata_hash;
                      model.safetensors.index.json is exempt — re-sharding is legit, checked
                      structurally at step 4)
  3. dtype preflight  header-only HTTP Range read of each shard — must be F16/BF16, else weight_dtype
                      (rejects quantized / F32 / F64 BEFORE the full download)
  3.5 download        full repo from the committed hub (HF or Hippius) → ALBEDO_MODEL_CACHE_DIR
  4. safetensors index  model.safetensors.index.json weight_map vs on-disk shards + tensors
  5. architecture     config.json vs validate/architecture_spec.json (architectures + expected + forbidden_keys)
  6. fingerprint      per-tensor secret-seeded sketch ψ·W·ω (K=64) + the weights at secret sample positions,
                      computed on GPU ALBEDO_DEDUP_GPU (model_validation/dedup/sketch.py)
  7. dedup gate       nearest ALBEDO_DEDUP_NEAREST_K (10) bank entries by OpenSearch kNN → signals on the sketches
                      (distances, spectra, merge fits) → verdict:
                        COPY / OWN-COPY   identical bytes, or relative distance < ALBEDO_DEDUP_COPY_REL (1e-5) → always fault
                        NOISE-COPY / NOISED-COPY / LINEAR-COMBO / SPARSE-EDIT / TRIVIAL-EDIT   heuristic — recorded; fault
                                          only when listed in ALBEDO_DEDUP_ENFORCE_REASONS (gated by ALBEDO_DEDUP_ENFORCE)
  success → HIPPIUS_VALIDATED ; fingerprint added to the bank
  miner fault → TERMINAL_INVALID + fault.json
                (file_manifest | chat_template_hash | metadata_hash | weight_dtype | safetensors_index
                 | architecture | duplicate | duplicate_own | duplicate_heuristic | hotkey_already_validated)
  infra fault → HIPPIUS_RETRYABLE (≤ 5 attempts) → TERMINAL_INFRA_FAILED
```

- **One validated model per hotkey:** a later commit from an already-validated hotkey fails with
  `hotkey_already_validated`. Hotkey *reuse across submissions* is separately blocked by `chain_guard`
  (its ledger burns a hotkey after eval — see *Repository layout*).
- **Spec-driven arch lock:** `architecture.py` reads a JSON spec, so changing the locked model family
  needs no code change — regenerate with `scripts/generate_arch_spec.py`.
- **Where the checks live:** the live file / arch / safetensors-index / dtype checks the miner CLI and
  this worker both run come from `model_validation.validate`. `config_validation` backs the separate
  commit-validator and supplies the miner's Hippius download/list utilities + `ModelRef`. The local
  `check-model` skips only the dedup gate.

---

## Pre-eval / sanity gate

A cheap stability gate that runs **before** the GPU duel so broken or adversarial models never reach it.

```
sanity-dispatcher: claim PRE_EVAL_QUEUED → PRE_EVAL_RUNNING
  sample N deterministic prompts from the SAME shard manifest the duel uses
    (SANITY_DISPATCH_SAMPLE_COUNT, default 3, seeded by block_hash)
  POST to sanity-remote GPU worker → warm vLLM → generate a trajectory
    (SANITY_DISPATCH_TRAJECTORY_ASSISTANT_TURNS, default 32 turns)
  per response:
    text heuristics    (empty / truncated / unclosed <think> / too-short / repetition /
                        encoding / vocab ratio)
    across responses   (collapsed to one answer / suspiciously uniform length / no code at all)
    tail + loop check   (tail_check.py: duplicate-command ratio, longest consecutive run)
    injection probe     (judge: did the model try to jailbreak / inject a verdict?)
    viability probe     (judge: coherent + on-task?)
  aggregate (injection > infra > viability-fail > pass), quorum ≥ 2 resolved judges
  PRE_EVAL_PASSED  → cache in sanity_results
  fail → TERMINAL_INVALID (injection / viability) | PRE_EVAL_RETRYABLE (infra)
```

`sanity-remote` is **stateless** (no DB, no dataset, no keys) — it just loads the model on a GPU,
generates, runs heuristics, and returns. All judgment lives on the backend.

**Failure reports → Hippius:** on a *terminal* miner-fault rejection (injection / not-viable), the
dispatcher (`sanity_service/uploads.py`) uploads `sanity/{submission_id}/{digest}/fault.json` —
the reason, `fault_code`, the full per-judge injection/viability evidence, prompts, and the model's
responses — public-read, and records a `SANITY_RESULT` artifact row so the dashboard links it. Passes
and retryable/infra faults upload nothing. Env-gated on `ALBEDO_S3_*` (no-op when unset). The judge
system prompts themselves are never included.

---

## Coronation & weights

```
set-reign-worker: claim EVAL_WIN (| SET_REIGN_RETRYABLE)
  load active reign (state=ACTIVE) + current 5 kings (slots 1–5)
  insert challenger into the chain → shift the others down → king falling out of top-5 is RETIRED
  write king_versions + reign_members (slot, uid, hotkey, model_hash, weight_bps)
  insert weight_epoch(reason=CORONATION, state=PENDING, uids[], weights[], weight_hash)
  submission → REIGN_SET

weight-setter: claim weight_epochs (PENDING | FAILED_RETRYABLE)  [advisory lock]
  respect rate limit: no write within ALBEDO_WEIGHT_SET_RATE_BLOCKS (101) blocks of last success
  if nothing pending → create a PERIODIC_REFRESH epoch (keeps weights live)
  subtensor.set_weights(wallet, netuid=97, uids, weights)
  success → record block_number → submission COMPLETE_CORONATED
  genesis-only state → weights are [uid 0] → [1.0] (burn UID)
```

The king "chain" is a rolling 5-slot ring: emissions are split across the current king and the
previous four (`weight_bps` per `reign_members` row), so dethroning is gradual rather than winner-take-all.

---

## Dashboard publishing (the monitor)

`website/monitor.py` is a standalone PM2 service (`albedo-dashboard-monitor`) — it watches the eval DB
and, **only when something changes**, regenerates and uploads the two files the static site reads.

```
loop every ALBEDO_MONITOR_INTERVAL_S (default 2s):
  signature = max(model_submissions.updated_at), count(*),
              max(eval_runs.finished_at), max(reigns.version)
  if signature unchanged → do nothing
  else:
    build dashboard.json   reign · eval_runs(history) · current_eval · queue · fails · stats
                           (score_breakdown read from stage_attempts.result_summary;
                            artifact s3:// URIs rewritten to https public URLs)
    build state.json       live pipeline: hippius_validate / pre_eval / eval, each running + queued
    write website/data/{dashboard,state}.json  AND  put_object → s3://$ALBEDO_S3_BUCKET/data/*.json
                                               (public-read, no-cache; upload skipped if ALBEDO_S3_* unset)
```

- **state.json stage buckets** (handoff states count as the *next* stage's queue, since that's what the
  next dispatcher claims):
  `hippius_validate` → queued `SUBMITTED`/`HIPPIUS_RETRYABLE`, running `HIPPIUS_RUNNING`;
  `pre_eval` → queued `HIPPIUS_VALIDATED`/`PRE_EVAL_QUEUED`/`PRE_EVAL_RETRYABLE`, running `PRE_EVAL_RUNNING`;
  `eval` → queued `PRE_EVAL_PASSED`/`EVAL_QUEUED`/`EVAL_RETRYABLE`, running `EVAL_RUNNING`.
- The website renders **state.json as the live queue** (3 stage cards, running + queued) and
  **dashboard.json** as reign / history / chart / fails. Models display as **`ALBEDO-<first-6-of-hotkey>`**,
  upgraded to **`ALBEDO-<Roman>`** (e.g. ALBEDO-II) once crowned; the real repo is the hover tooltip + hub link.
- **Model filter:** history + chart only include `eval_runs` whose `model_uri` matches
  `ALBEDO_DASHBOARD_MODEL_FILTER` (SQL `LIKE` substring, default `qwen3.6-35b` — the 35B genesis plus
  any `albedo-qwen3.6-…` challenger), so a model migration doesn't mix old and new runs.
- The **per-eval detail page** surfaces each sample's yes/no questions and per-side judge scores
  (from scoring-results.jsonl) and the run's `scoring_mode` (`binary`).
- Reactive, not asyncio: a synchronous on-change poll loop, each tick wrapped in try/except (a DB/S3 blip
  just retries next tick). Env: `ALBEDO_MONITOR_INTERVAL_S`, `ALBEDO_DASHBOARD_NETUID` (97),
  `ALBEDO_DASHBOARD_ARTIFACT_BASE_URL` (https://s3.hippius.com), `ALBEDO_DASHBOARD_MODEL_FILTER`
  (`qwen3.6-35b`), plus `ALBEDO_EVAL_DATABASE_URL` + `ALBEDO_S3_*`.

---

## Postgres schema (schema.sql)

The whole system is a durable state machine in one database. Key tables:

| Table | Holds |
|---|---|
| `chain_commits` | raw on-chain v7 commits (netuid, block, hotkey, model_uri, payload_hash) |
| `miners` | hotkey → coldkey / uid / netuid |
| `model_submissions` | **the spine** — one row per submission + its `state` + `fault_class/code` |
| `stage_attempts` | per-stage claim/lease/heartbeat (`HIPPIUS`/`PRE_EVAL`/`EVAL`/`SET_REIGN`/`WEIGHT_SET`) |
| `remote_gpu_hosts` | GPU fleet registry (role `PRE_EVAL`/`EVAL`, free_gpu_count, heartbeat) |
| `eval_runs` | one duel: king/challenger hashes, scores, win_margin, sample/turn/error counts |
| `king_versions` / `reigns` / `reign_members` | the 5-slot king chain + reign history |
| `weight_epochs` / `weight_transactions` | weight intents + their on-chain submissions |
| `artifacts` | S3 / Hippius / local-cache pointers (eval samples/scoring JSONL, verdict.json, fingerprints, `SANITY_RESULT` fault reports) |
| `events` | append-only audit log per submission / stage_attempt |
| `sanity_results` | pre-eval verdict cache keyed by `digest` |

Concurrency guards baked into the schema: a partial unique index allows **one active EVAL run**
globally, **one active attempt per stage** per submission, and **one ACTIVE reign**. The dispatcher
adds a `pg_try_advisory_xact_lock('full_eval')` on top.

---

## Environment variables (highlights)

Full reference in [.env.example](../.env.example) (backend + GPU host) and [.env.example_miners](../.env.example_miners) (miner).

### Postgres / backend core
| Variable | Purpose |
|---|---|
| `ALBEDO_EVAL_DATABASE_URL` | DSN for the whole eval stack |
| `ALBEDO_POSTGRES_*` | docker-compose local Postgres (a localhost-only port) |
| `ALBEDO_EVAL_REMOTE_AUTH_TOKEN` | shared bearer for backend↔remote eval API |
| `ALBEDO_EVAL_DATASET_MANIFEST_HASH` | sha256 pin of the four-source shard manifest (mini-coder, mini-coder-rs, open-swe-traces, swe-hero) |
| `ALBEDO_EVAL_SAMPLE_COUNT` | duel size — unique instances sampled per eval (default 100) |
| `ALBEDO_EVAL_PREFETCH_NEXT_CHALLENGER` | pre-download the next queued challenger during the running eval (default true) |

### GPU eval host
| Variable | Purpose |
|---|---|
| `ALBEDO_REMOTE_HOST_ROLE` | `EVAL` or `PRE_EVAL` |
| `ALBEDO_REMOTE_PREVIOUS_KING_GPU_IDS` / `_CHALLENGER_GPU_IDS` | `0,1,2,3` / `4,5,6,7` |
| `ALBEDO_REMOTE_ROLLOUTS_PER_SAMPLE` | trajectories per sample per side (default 2); the score is the mean over all of them |
| `ALBEDO_REMOTE_PREVIOUS_KING_VLLM_PORT` / `_CHALLENGER_VLLM_PORT` | loopback ports of the two `vllm serve` processes (9201 / 9202) |
| `ALBEDO_REMOTE_VLLM_MAX_NUM_SEQS` | in-flight sequences per engine (default 256; 2 rollouts x 100 samples) |
| `ALBEDO_REMOTE_GENERATION_BACKEND` | `vllm` |
| `ALBEDO_REMOTE_SCORING_BACKEND` | `websocket` (score bridge) or `http` |
| `ALBEDO_REMOTE_SCORING_BATCH_CONCURRENCY` | concurrent /score-batch requests per eval (default 128) |
| `ALBEDO_REMOTE_TRAJECTORY_ASSISTANT_TURNS` | fallback turn count only — the live horizon is stratified per sample via HORIZON_STRATA (12, 16) |
| `ALBEDO_REMOTE_COMPILE_CACHE_DIR` | shared vLLM torch.compile cache dir (e.g. /root/vllm_shared_compile) — same-arch models reuse backbone graphs; unset → per-model dirs pile up in ~/.cache/vllm |
| `ALBEDO_REMOTE_MOCK_AUTO_VERDICT` | smoke mode — verdict without GPUs |
| `ALBEDO_REMOTE_REPO_CONTEXT_URL` | the GPU box's own route to the grounding service |
| `ALBEDO_REMOTE_S3_*` | artifact upload credentials |

### Judge
| Variable | Purpose |
|---|---|
| `ALBEDO_JUDGE_OPENROUTER_API_KEY` | OpenRouter key for evaluator/judge/simulator/reference calls |
| `ALBEDO_JUDGE_MAX_CONCURRENCY_PER_MODEL` | per-model semaphore, shared across concurrent batches |
| `ALBEDO_JUDGE_MIN_VALID_FRACTION` | min fraction of samples scored on both sides (default 0.8) |
| `ALBEDO_JUDGE_EVALUATOR_MODEL` | evaluator + judge model (default `z-ai/glm-5.2`) |
| `ALBEDO_JUDGE_SOTA_MODELS` | reference-trajectory model (prod `z-ai/glm-5.2`) |
| `ALBEDO_JUDGE_SOTA_TRAJECTORY_TURNS` | reference-trajectory length (default 8) |
| `ALBEDO_JUDGE_REFERENCE_RUNS` | reference trajectories generated per sample (default 3); below two usable runs the sample is dropped |
| `ALBEDO_JUDGE_REFERENCE_PRUNE` | judge every reference run against the checklist and drop what none of them earns (default true); costs one judge call per run per sample at prep |
| `ALBEDO_JUDGE_SIMULATION_MODEL` | observation-simulator model (default `deepseek/deepseek-v4-flash-0731`; empty → evaluator model) |
| `ALBEDO_JUDGE_SIMULATION_PROVIDERS` | provider list for the simulator, tried as one rung each in rotation (default `deepseek,cloudflare`; empty → no pin) |
| `ALBEDO_JUDGE_MILESTONE_READINGS` | independent extractor readings, merged by union into the milestone vector (default 4) |
| `ALBEDO_JUDGE_QUESTION_READINGS` | independent ladder writers, merged by majority (default 3) |
| `ALBEDO_JUDGE_JUDGE_REPEATS` | judgings per trajectory; a question's answer is the majority (default 3) |
| `ALBEDO_JUDGE_NUM_QUESTIONS` | recorded in `question_source` for provenance only; the checklist size is the surviving milestones x `RUNGS_MAX` (6), capped at `QUESTIONS_MAX` (60) |
| `ALBEDO_JUDGE_ENGY_MODELS` | model pool for grounded/engy calls (default `z-ai/glm-5.2,deepseek/deepseek-v4-flash-0731`) |

### Repo-context grounding
| Variable | Purpose |
|---|---|
| `ALBEDO_REPO_CONTEXT_API_HOST` / `_API_PORT` | where the grounding service listens (default `127.0.0.1` / a localhost-only port) |
| `ALBEDO_REPO_CONTEXT_CACHE_DIR` | snapshot cache root (bounded by `_MAX_CACHE_GB`, default 60) |
| `ALBEDO_REPO_CONTEXT_DATASET_ROOT` / `_DATASET_MANIFEST_PATH` / `_DATASET_MANIFEST_HASH` | resolve a sample id → repo + commit |
| `ALBEDO_REPO_CONTEXT_GITHUB_TOKEN` | fetch the repository snapshot at the sampled commit |
| `ALBEDO_REPO_CONTEXT_MAX_FILES` / `_MAX_PATHS` / `_MAX_FILE_CHARS` / `_MAX_CONTEXT_CHARS` / `_MAX_SNAPSHOT_MB` | grounding block budgets (16 / 2000 / 30k / 120k / 500) |
| `ALBEDO_JUDGE_REPO_CONTEXT_URL` | judge-api → grounding service; unset disables grounding entirely (the simulator then answers every command) |

### Hippius validation
| Variable | Purpose |
|---|---|
| `ALBEDO_OPENSEARCH_URL` / `_USER` / `_PASSWORD` / `_INDEX` | dedup bank (kNN index of accepted fingerprints) |
| `ALBEDO_DEDUP_SECRET` / `_SECRET_FILE`, `_GPU`, `_NEAREST_K`, `_COPY_REL`, `_ENFORCE`, `_ENFORCE_REASONS` | sketch seed, GPU, neighbours checked, copy tolerance, enforcement switch and which verdicts fault (default `COPY,OWN-COPY`) |
| `ALBEDO_MODEL_CACHE_DIR` | downloaded model cache |
| `ALBEDO_S3_*`, `HIPPIUS_HUB_TOKEN` | artifact + model store auth |

### Weights
| Variable | Purpose |
|---|---|
| `ALBEDO_WEIGHT_COLDKEY` / `_HOTKEY` / `_WALLET_PATH` | validator wallet |
| `ALBEDO_WEIGHT_NETWORK` / `_NETUID` | `finney` / `97` |
| `ALBEDO_WEIGHT_SET_RATE_BLOCKS` | min blocks between weight writes (101) |
| `ALBEDO_WEIGHT_BURN_UID` | UID weights burn to when no registered king (0) |

### Dashboard monitor
| Variable | Purpose |
|---|---|
| `ALBEDO_MONITOR_INTERVAL_S` | change-detection poll interval (default 2) |
| `ALBEDO_DASHBOARD_NETUID` | netuid stamped into dashboard.json (default 97) |
| `ALBEDO_DASHBOARD_ARTIFACT_BASE_URL` | base for rewriting artifact `s3://` URIs → https (code default `https://s3.hippius.com`; prod overrides it to `https://albedo.tech` since artifacts moved to R2) |
| `ALBEDO_DASHBOARD_MODEL_FILTER` | `model_uri` LIKE-substring for history/chart (default `qwen3.6-35b`) |
| `ALBEDO_S3_*` | bucket/endpoint/keys for uploading dashboard.json + state.json (shared with model_validation) |

---

## Running locally

```bash
cp .env.example .env
set -a; source .env; set +a
uv sync
docker compose up -d albedo-postgres
docker compose exec -T albedo-postgres psql -U "$ALBEDO_POSTGRES_USER" -d "$ALBEDO_POSTGRES_DB" < schema.sql
```

Smoke-test the eval stack without GPUs by setting `ALBEDO_REMOTE_MOCK_AUTO_VERDICT=true` on the
remote API, then start the PM2 ecosystem files (see the runbook). Seed a genesis king first:

```bash
python scripts/create_genesis_king.py     # genesis reign + king_version + reign_members (UID 0)
```

---

## Key contracts

**Reveal (on-chain):** `v7|<repo>|<digest>` — e.g. `v7|alice/albedo-qwen3.6-35b-v1|sha256:...`

**EvalRequest (backend → POST /eval-runs):** king + challenger model refs, `dataset_sample_ids`,
`dataset_manifest_hash`, `judge_config_hash`, `dataset_sample_seed` (= commit block hash).

**Verdict event (GPU → dispatcher, SSE/events):** `state`, `score_challenger`, `score_king`,
`win_margin`, `challenger_won`, `valid_turns`/`total_turns`, `king_vllm_errors`/`chal_vllm_errors`/`judge_errors`,
plus artifact URIs.

**Question prep (GPU → backend over WS /score-bridge, before scoring):** forwarded to judge API
`/category-prep` → GLM 5.2 rolls three reference trajectories, extracts the task's milestones and writes
the question ladder; the score batch then judges every rollout of both sides against it.

**Score request (GPU → backend over WS /score-bridge):** `{type:"score_request", request_id, payload}`
→ forwarded to judge API `/score-batch` → `{type:"score_response", request_id, body}`. Scoring records
carry `scoring_mode` (`binary`), the `questions`, per-side `judge_results`, and `question_source`
(question regime + reference trajectory provenance + every question discarded during prep). A side that
was short-circuited also carries `looped`, `loop_reasons` and `loop_commands`.

**Observation request (GPU → backend over WS /score-bridge, once per turn):** forwarded to judge API
`/simulate-observation` with the sample id, the model's assistant output and the messages so far →
`{observation}` in the trajectory's own format. The judge API resolves it via the grounding →
transcribe → simulate ladder described in *The duel* above.

**Grounding request (judge API → repo-context service):** `POST /repo-context` with `{sample_id,
assistant_output, messages}` → `{sample_id, context (the grounding block), kind, exact_output,
exact_returncode}`. `exact_output`/`exact_returncode` are present only when the command was actually
executed against the snapshot; both are optional, so a judge API and a grounding service on different
versions degrade to plain simulation instead of failing.

---

## Testing

```bash
uv run pytest -q                                                  # unit tests
ALBEDO_TEST_DATABASE_URL=postgresql://user:pass@localhost:<port>/db \
  uv run pytest -q tests/integration                              # needs Postgres + schema.sql
uv run ruff check src/ && uv run ruff format src/                 # lint
```

---

## Links

- **Dashboard:** static site in `website/`; reads `data/dashboard.json` + `data/state.json`, published to
  Hippius S3 by `website/monitor.py` (live, on-change) or `website/push_to_hippius.py` (one-shot)
- **Datasets** (four sources, pooled one-source-trajectory-per-instance and stratified by phase x bug family — see docs/DATASETS.md):
  - mini-coder (400k, python): https://huggingface.co/datasets/ricdomolm/mini-coder-trajs-400k
  - mini-coder-rs (rust, SWE-smith-rs): https://huggingface.co/datasets/AlienKevin/SWE-smith-rs-gpt-5-mini-trajectories
  - open-swe-traces (4 arms): https://huggingface.co/datasets/nvidia/Open-SWE-Traces
  - swe-hero (OpenHands): https://huggingface.co/datasets/nvidia/SWE-Hero-openhands-trajectories
  - shard manifest (per-source weights + sha256s): https://albedo.tech/datasets/manifest.json
- **Mining guide:** [docs/MINING.md](../docs/MINING.md)
- **Scoring reference:** [docs/SCORING.md](../docs/SCORING.md)
- **Datasets & observation formats:** [docs/DATASETS.md](../docs/DATASETS.md)
- **Eval runbook:** [docs/eval-service-status.md](../docs/eval-service-status.md)
- **Reign/weight notes:** [docs/reign-and-weight-pm2.md](../docs/reign-and-weight-pm2.md)
- **Sibling repo (single-process design):** `albedo-refactor` — same subnet, monolithic validator