@rhei-agent/local-learning

Local-only trace validation, deterministic task splits, dataset manifests, and asynchronous SDFT job state for RHEI.

This package does not train in-process and never promotes a model. The coding agent materializes accepted or verifier-approved traces and launches the separate python/rhei-learning worker. Candidate review, promotion, rollback, and vLLM reload remain explicit operator actions.

SDPO is deliberately unsupported until RHEI can collect fresh verifier-backed rollouts with the required token/log-probability information.

Evidence that the loop works

These measurements came from real RHEI /update jobs using accepted GPT-5.6 Luna traces, SDFT, and LoRA adapters. “Before” is greedy exact-match accuracy from the base checkpoint; “after” is the same evaluation with the candidate adapter loaded. Candidates remained manual-review artifacts and were not automatically promoted.

| Student / task | Evaluation split | Before | After | Change | |---|---|---:|---:|---:| | Qwen3.5-0.8B / GSM8K | 100 fresh problems | 37/100 (37%) | 45/100 (45%) | +8 pp | | Qwen3-0.6B / modulo-7 protocol | 100 fresh probes | 0/100 (0%) | 6/100 (6%) | +6 pp | | Qwen3-1.7B / modulo-7 protocol | 23 held-out traces | 6/23 (26.1%) | 8/23 (34.8%) | +8.7 pp |

The table demonstrates the intended contract: traces are materialized, the worker trains a non-empty adapter, and the adapter changes held-out behavior. It is not a promise that every update improves every task; evaluation and promotion are deliberately separate safety gates.

Configuration

The coding agent discovers project configuration first at .rhei/local-learning.json, then global configuration at ~/.rhei/agent/local-learning.json:

{
  "enabled": true,
  "providerId": "vllm-local",
  "modelId": "Qwen3-Coder",
  "endpoint": "http://127.0.0.1:8000/v1",
  "checkpointPath": "/models/Qwen3-Coder",
  "artifactRoot": "/var/lib/rhei/learning",
  "sourceFilters": [
    { "providerId": "openai-codex" },
    { "providerId": "anthropic" },
    { "providerId": "vllm-local", "modelId": "rhei-qwen3-0.8" }
  ],
  "algorithm": "sdft",
  "externalReporting": false
}

Bare providerId/modelId identify the local checkpoint that receives the update. In the normal self-distillation loop, approved traces from the previous version of that same model are the teaching context for its next candidate. By default, /update reads every accepted or verified trace stored by RHEI, regardless of provider, model, or authentication method. Use sourceFilters only when you want to restrict the data to particular provider/model pairs; filters use exact matching. RHEI reads only its own session files, so Codex, Claude, or other-provider traces are eligible when those providers were used through RHEI. Importing a conversation from another application is not implemented. Both filesystem paths must be absolute, the checkpoint must already exist, and the endpoint must resolve to localhost or a private address. Run /update --since 7d to queue training, /update status to inspect the latest job, and /update cancel <job-id> to stop one. Label an assistant entry accepted or verified before collection; unlabeled and rejected entries are excluded.

What users need to install

The /update coordinator launches the worker with:

uv run --project python/rhei-learning --extra training rhei-learning train ...

Therefore users do not need to pre-install TRL, Transformers, PEFT, or vLLM into their global Python environment. uv creates or reuses the project environment and installs the pinned training extra on the first update. The user still owns the system prerequisites: uv, a working Python 3.11–3.13 runtime, NVIDIA drivers/CUDA that PyTorch can use, and a local checkpoint whose weights and tokenizer fit the available GPU memory. useVllm: false avoids the vLLM generation path for incompatible checkpoints, but the current training extra still includes trl[vllm] because the default path supports colocated vLLM generation.

checkpointPath is deliberately required and must already exist. /update does not download a model, convert quantization, start a serving endpoint, or reload a live server. The endpoint in the config is validated as a local/private serving identity; the worker loads checkpointPath directly for the candidate training run.

What counts as a training trace

RHEI stores sessions as JSONL. The collector reconstructs a completed assistant turn together with its preceding user/tool context. A trace is eligible only when:

• it contains at least one user message and one assistant message;
• the assistant turn completed successfully; and
• the target assistant entry has the exact label accepted or verified.

rejected, unlabeled, incomplete, aborted, error, and secret-containing traces are excluded. In the interactive UI, open /tree, select the assistant answer, press Shift+L, and enter exactly accepted or verified. In an integration, attach the label after a verifier passes, for example sessionManager.appendLabelChange(assistantEntryId, "accepted").

The worker receives a compact JSONL row, not the raw session file:

{
  "prompt": [{"role": "user", "content": "the original task"}],
  "privileged_context": "the accepted assistant/tool trajectory and verifier feedback"
}

The raw session files remain the provenance source; /update writes a hashed manifest, deterministic train/evaluation splits, and a temporary LoRA candidate under artifactRoot. It stops at awaiting_promotion, so an operator or a future promotion policy must evaluate the candidate before making it active.

Capture is automatic; eligibility is explicit

Normal RHEI sessions are persisted automatically as local JSONL under the session directory (by default under ~/.rhei/agent/sessions/<encoded-cwd>/). There is no separate “start trace collection” command. /update scans those files when it starts. This is local trace capture, not telemetry: prompts, responses, and tool content are not uploaded by the learning collector.

The separate install/update telemetry described in the coding-agent settings is an anonymous product/version ping and is unrelated to training traces. It does not provide data to /update and can be disabled with the normal offline or telemetry settings.

What is not automatic is deciding whether an answer is safe and useful to learn. A human, verifier, or integration must label the completed assistant entry accepted or verified; otherwise the session remains available for normal history but is ignored by /update.

Automatic labeling with a verifier

Task-specific integrations can label turns from an extension hook. The verifier should be deterministic whenever possible (tests, a compiler, a schema check, or an exact expected answer), and should reject on uncertainty:

import type { ExtensionAPI } from "@rhei-agent/coding-agent";

export default function (rhei: ExtensionAPI) {
	rhei.on("turn_end", async (event, ctx) => {
		if (event.message.role !== "assistant" || event.message.stopReason !== "stop") return;
		const entry = ctx.sessionManager.getLeafEntry();
		if (!entry || entry.type !== "message" || entry.message.role !== "assistant") return;
		const answer = event.message.content
			.filter((block): block is { type: "text"; text: string } => block.type === "text")
			.map((block) => block.text)
			.join("\n");
		const passed = await verifyAgainstTheTask(answer, ctx);
		ctx.setLabel(entry.id, passed ? "verified" : "rejected");
	});
}

verifyAgainstTheTask is intentionally task-specific; RHEI cannot infer correctness for arbitrary conversations. An LLM judge can be used for triage, but deterministic checks or human review should remain the final acceptance boundary for high-value updates.