tt - Tuned Tensor CLI

tt is the local agent and CLI for Tuned Tensor. Develop and orchestrate one training workflow locally, using a local NVIDIA GPU or an existing GPU instance in your AWS account. Inspect the resulting artifacts and metrics locally, and optionally publish run evidence to the web dashboard.

Local training, evaluation, inspection, and serving need no TT access token. Agent inference is a separate choice:

• Managed: sign in once with tt auth login. TT reuses your access token through the Tuned Tensor model proxy. The server selects the model; no OpenRouter key or model selection is needed.
• Bring your own: use your own OpenRouter or other provider credentials and choose the model. Your requests go directly to that provider. TT does not apply its managed model policy to BYO requests.

A saved BYO selection stays selected after TT login. Switch back with tt agent configure --provider tunedtensor or /model tunedtensor/managed. Inference choice is independent of where training uses a GPU. Your AWS GPU needs AWS credentials and SSH access; TT account operations under tt cloud require a TT access token.

New to cloud GPU training? Follow your first AWS run for project creation, AWS quota and SSH setup, a preview, training, result verification, and shutdown. You still run tt pipeline run on your laptop; tt cloud is for TT account records, not user-owned GPU execution.

Unified commands

Local workflows remain the default: tt pipeline run, tt runs list, tt models list, and tt serve. GPU processes can use an existing instance in your AWS account through gpu configuration. tt cloud runs list and tt cloud models list access existing account records. tt usage reports managed agent allowance and usage; tt balance reports TT credits for hosted labeling and historical training charges, which do not pay for your AWS instance. tt publish uploads local run evidence to the dashboard. Hosted training submission (tt cloud runs start and estimate) is retired.

TT includes the former TT Local runtime. tt local … and tt run remain hidden compatibility aliases; new scripts should use root commands and tt pipeline run --spec tunedtensor.json. Existing token-free local projects and saved BYO provider settings continue to work. Cloud operations previously hidden in the local-only release are available under tt cloud.

Install

On Linux or macOS:

curl -fsSL https://tunedtensor.com/install.sh | sh
tt --version

The installer bootstraps Node.js 22.19+ if needed, installs @tuned-tensor/cli with npm --ignore-scripts, and falls back to ~/.local when the global npm prefix is not writable. Inspect it first with curl -fsSL https://tunedtensor.com/install.sh | less. Pin a dist-tag or version on sh, not curl:

curl -fsSL https://tunedtensor.com/install.sh | TT_VERSION=beta sh
curl -fsSL https://tunedtensor.com/install.sh | TT_VERSION=0.13.0 sh

Or install the package directly:

npm install -g --ignore-scripts @tuned-tensor/cli
tt --version

Upgrade later with tt upgrade. Uninstall with npm uninstall -g @tuned-tensor/cli (add --prefix ~/.local if the curl installer used that prefix).

Node.js 22.19.0 or newer is required. Training additionally needs uv locally and a supported NVIDIA CUDA host, either local or in your AWS account. The laptop does not need a GPU when gpu is configured. The locked Python runner ships with the npm package and is prepared on first use.

Run from source:

git clone https://github.com/tunedtensor/tuned-tensor-cli.git
cd tuned-tensor-cli
npm install
npm run build
npm link

Behavior spec workflow

tunedtensor.json is the source of truth for behavior and training settings. In the shell, /spec shows a readable overview grouped into behavior, examples, training, evaluation and runtime sections; /spec <section> (for example /spec examples) shows one section in full and /spec show prints the highlighted JSON. /spec diff, /spec validate, and /spec history complete the review, all without a model call. Outside the shell, tt spec view and tt spec <section> print the same readable view. Ask the agent for changes, inspect its proposed diff, then press y when tt asks (or /approve later) to save a validated edit with revision history. Pipeline runs use the saved spec. See the spec workflow.

Conversational terminal

The interactive agent harness and conversation state run on your laptop. Inference uses the TT managed model or a BYO provider you select. A BYO provider may be a local endpoint or a remote service.

Run tt to open the primary conversational workflow. The agent can inspect the project, derive the canonical pipeline from tunedtensor.json, present the resolved plan, and prepare a sealed dry-run for /approve. Workflow commands such as runs list and doctor work immediately without any credentials.

For managed inference, save your TT access token with hidden input:

tt auth login
tt

Get a token from the dashboard's Settings → API Keys page. You can also set TUNED_TENSOR_API_KEY in CI. When there is no saved BYO selection, TT selects tunedtensor/managed automatically. In the shell, /login tunedtensor saves the same account token and switches the saved agent selection to tunedtensor/managed, including after restart. Managed inference requires no OpenRouter key or additional login. The server controls the model and request limits. tt usage shows the weekly limit remaining, the daily allowance, and reported usage.

For BYO OpenRouter inference:

› /login openrouter
› /model openrouter/<model-id>

/login asks which provider, then prompts for the TT access token or BYO provider key with hidden input. /login <provider> skips the provider prompt. /model openrouter lists the bundled catalog, but you can select any OpenRouter model ID, including one newer than that catalog. OpenRouter validates the selected ID. New IDs use conservative context/output defaults; custom metadata can be supplied in ~/.tuned-tensor/agent/models.json. For a model without reasoning support, configure it with tt agent configure --provider openrouter --model <model-id> --thinking off. BYO inference is billed by your provider.

Provider keys are stored under TT's local agent config, separately from the account token. The managed provider never stores your TT token in provider files and does not accept endpoint/model overrides in models.json; use a separate BYO provider for custom endpoints. tt does not read ~/.pi/agent/. /model and /login feature Tuned Tensor, OpenAI, and OpenRouter. Other catalog providers remain available by ID. Unauthenticated entries are marked auth required. Local endpoints such as Ollama use a placeholder apiKey in models.json and need no TT account.

tt checks npm for a newer stable release at most every 12 hours and remembers the answer in ~/.tuned-tensor/update-check.json. The notice appears when the shell opens and after commands run in a terminal; explicit commands never wait on the registry. Upgrade with:

tt upgrade            # latest stable, into the same npm prefix tt runs from
tt upgrade --check    # only report
tt upgrade beta       # or a specific version

Offline or unavailable registry checks are ignored. CI, TT_NO_UPDATE_CHECK=1 or NO_UPDATE_NOTIFIER=1 disable the check.

After a model is selected, the banner shows it and ordinary sentences go to the agent:

tt v0.15.0
agent tunedtensor/managed · workflow model base
ctrl+c stop/clear · ctrl+d exit · /help commands · tab complete

Ask TT anything. Known commands run directly.
› What happened in my latest training run?
› runs list

Ordinary sentences go through the locally orchestrated model session. The model has no shell or general filesystem tool. It can read and propose reviewed edits to an existing spec, or prepare one new folder directly beneath the shell's current working directory with a validated tunedtensor.json, or prepare a local pipeline dry-run sealed to the reviewed workspace and spec contents. Writes and pipeline previews wait for your approval: after the answer, tt shows the proposal and asks inline, where y approves, n rejects and Esc leaves it pending for a later /approve or /reject. Any other key is ignored. The tools refuse path traversal, symlinked workspace roots, overwriting a new-project target, unsupported spec fields, cloud pipeline targets, and spec changes after review. Known CLI commands such as runs list, doctor, and models list still execute directly. Prefix a command with : when you want to make that intent explicit.

The agent can describe the built-in adapter and foundation pipeline stages, including optional foundation RL. When asked to train or dry-run, it derives or validates the pipeline, shows the exact plan, and prepares a deterministic dry-run. /approve starts that preview outside the model, and Ctrl-C stops the child process group. Real training is intentionally not model-mediated: run the displayed tt pipeline run command directly when you choose to execute. It can also search public Hugging Face model or dataset metadata for foundation and fine-tuning discovery. A Hub search sends only the search query to huggingface.co; it uses no Hugging Face token, downloads nothing, and omits model-card and dataset-card text. Search results are discovery metadata, not a guarantee that a model or dataset is compatible with a TT workflow. The agent can inspect a public dataset's exact JSONL/Parquet filenames, sizes, revision, and declared columns/splits before proposing a download. Declared schema comes from dataset-card metadata; it is not validation of the file contents.

To reuse a download, ask “Which datasets have we downloaded?” followed by “Use CodeAlpaca in this spec for fine-tuning.” The agent can list TT-recorded local datasets and inspect actual Parquet columns/row counts or sampled JSONL fields, plus two bounded sample rows. It proposes a dataset_prebuilt pointer with the cached revision, training file, separate validation/test file and column mapping. Approval saves the edit; listing and inspection need no download or approval. Existing examples and model settings stay in the spec, but prebuilt data supplies the training rows instead of automatically mixing in the inline examples. Samples do not establish full dataset quality, and a download with only a training split still needs distinct held-out data.

When a conversation approaches the model's context budget, TT automatically summarizes earlier turns using Pi and the selected model, and shows progress. The summary preserves goals, decisions and action outcomes and is saved for /resume; recent exchanges remain in context. Summarization uses an additional model request (or several for unusually long histories). The transcript stays in local history subject to the existing storage limits, and pending approvals remain intact. If the current request is still too large, summarization fails, or the model stops at its token limit, TT reports the problem instead of silently returning to the prompt. A failed summary does not replace the saved one.

For educational questions about how or why training works, the agent can read the exact Python implementation shipped with the running TT build. The source tool is limited to named foundation components (tokenizer, pretraining, checkpointing, model, data, SFT, RL, evaluation, and common helpers) and adapter components (training, data, model contract, and evaluation); it cannot read arbitrary paths. Answers separate behavior directly visible in the code from inferred rationale and include the inspected package-relative path and source hash.

Agent conversations are durable and local under ~/.tuned-tensor/agent/threads (or $TUNED_TENSOR_HOME/agent/threads), with user-only directory/file permissions. Existing XDG agent state remains in use until the new agent directory exists. Persistence normally retains up to 100 recent threads; unsettled safety records are never pruned even when that exceeds the cap. Each thread is limited to 1 MB, 200 messages, and 200 actions; one-way action claims are capped at 1,000. Use /new to start one, /threads to list recent conversations, and /resume <id> to continue one.

Local spec creation waits for /approve, requires at least two examples, uses exclusive private folder/file creation, and never overwrites an existing path. Secure filesystem mutation is currently supported on Linux, where writes are anchored to open workspace and destination-directory handles. The proposal is bound to the current workspace so changing directories before approval fails safely instead of writing somewhere else. Ambiguous write failures remain sealed as outcome_unknown for manual inspection. /reject never mutates. Pipeline dry-runs also wait for /approve. Preparation fingerprints the workspace directory identity, spec, and adjacent config, revalidates them, copies the resolved spec, config, and pipeline into private temporary inputs, and invokes tt pipeline run --dry-run deterministically. Persisted proposals that request real execution fail before the child command is dispatched. Real training remains available through the explicit direct tt pipeline run command. A changed workspace or spec requires a new review.

Useful shell controls include /help, /status, /context, /system, /model, /login, /cd, /clear, and /exit. When the shell opens, the TT logo mark (drawn in the terminal) greets you with the current spec, a live machine panel (GPU utilization, VRAM and temperature from nvidia-smi, CPU, RAM, and free space in the model cache) and whether this machine can fine-tune the spec's base model, plus a suggested next step. /system refreshes that panel and lists every certified base model's verdict. Fit verdicts come from the same certified memory profiles as tt hardware; a fresh tt hardware snapshot takes precedence over the quick live check. The shell banner and /context show the inference selection (agent provider/model) separately from the workflow model. The managed alias identifies inference through the server-selected model. Run /model with no arguments to list the managed option and suggested BYO models, /model <provider> to list that provider's models, or /model <query> to search. Switch with /model <provider>/<model>, including /model tunedtensor/managed to return to managed inference. For managed inference, tt agent configure --provider tunedtensor needs no model argument. For BYO inference, both provider and model are required. The equivalent non-interactive commands are tt agent models, tt agent configure, and tt agent status. For BYO models, tt agent configure --thinking accepts off, minimal, low, medium, high, xhigh, or max; managed inference uses the server's reasoning policy and leaves the CLI thinking option off. Selection metadata can be overridden per process with TUNED_TENSOR_AGENT_PROVIDER, TUNED_TENSOR_AGENT_MODEL, and TUNED_TENSOR_AGENT_THINKING. These values are not credentials. The shell keeps normal terminal scrollback and command history only for the current process.

Explicit commands remain non-interactive, including in CI. tt --help shows the command surface, tt status inspects local project context without a network or GPU probe, and tt shell opens the conversational terminal explicitly.

Composable pipelines (v1)

A pipeline is an ordered JSON recipe. The CLI orchestrates every plan locally; optional gpu configuration moves its GPU processes to your AWS instance. The portable document contract lives in @tuned-tensor/pipeline-contract. The CLI adds execution planning (step selection) on top of it.

Three local engines share that document version:

• Adapter (default): LoRA SFT on a certified Hugging Face checkpoint. Runs through tt pipeline run. Uses train / evaluate / compare with evaluate.with.evaluator: "behavior". Certified models (Qwen/Qwen3.5-2B, Nemotron 3.5 Lightning, Muse Glimmer) have a reviewed revision, architecture, and memory profile. Any other Hugging Face text model can be trained as uncertified by pinning its commit; it loads as a causal LM with safetensors weights, no remote code, and generic LoRA defaults (override them in hyperparameters):

  {
    "base_model": "mistralai/Mistral-7B-Instruct-v0.3",
    "hyperparameters": { "base_model_revision": "<40-char commit SHA>" }
  }

  Pick an instruct/chat variant (the trainer needs a chat template) and check hardware fit with tt doctor, since TT has no memory profile for it.

• Foundation: from-scratch tokenizer + GPT. Also runs through tt pipeline run. Uses tokenize / pretrain / finetune / optional rl plus bpb, chat, and inference evaluators. Requires a foundation tunedtensor.json (engine: "foundation", at least two examples, no base_model).

• Decision: fine-tunes a small typed decision model (convaiinnovations/laya, 421M) that answers one choice, yes/no or score question per input with a probability per label, instead of generating text. Uses the adapter vocabulary (train / evaluate / compare) and runs on CPU, Apple MPS or CUDA. Requires engine: "decision" and a decision question; example outputs are labels. See Decision models.

All engines are spec-driven. system_prompt, guidelines, and constraints are compiled into one canonical system instruction, and examples supply the demonstrated behavior. Adapter training, foundation tokenizer/pretraining, chat SFT, optional RL, and evaluation all receive that same compiled instruction. Engine-specific fields choose how TT learns the behavior; they do not redefine the behavior itself.

The foundation engine is a readable, single-GPU local trainer inspired by Andrej Karpathy's nanochat, not a port of its distributed training stack. With foundation.corpus_path, it streams .txt or JSONL ({"text":"..."}) shards into a deterministic on-disk token cache; without that field it retains the bounded prompt/example corpus for smoke runs. foundation.validation_path supplies a distinct held-out BPB corpus. Foundation plans reject compare before creating run artifacts; comparison remains an adapter-engine capability. Optional RL samples one seeded on-policy completion per step with a sparse last-number exact reward and therefore requires a numeric expected output for every example. Treat those metrics as an end-to-end execution and overfit check—not held-out capability or multi-GPU evidence when no validation corpus is configured.

tunedtensor.json contains behavior, training settings, optional evaluation, runtime, and advanced pipeline sections. Pipelines use the embedded recipe or derive it from the saved spec. Only explicit --file loads a separate recipe; it cannot override a conflicting embedded recipe. Foundation recipes must match the spec's training settings.

tt pipeline validate --spec tunedtensor.json
tt --json pipeline plan --spec tunedtensor.json
tt pipeline run --spec tunedtensor.json --dry-run
tt pipeline run --spec tunedtensor.json
# Optional: embed the current derived recipe for customization
tt pipeline init --spec tunedtensor.json
# Existing projects: consolidate sidecars, retaining .bak copies
tt pipeline migrate --spec tunedtensor.json

Relative runtime paths resolve beside the spec. New project state uses .tuned-tensor/store and .tuned-tensor/artifacts. Legacy local-runner.json remains readable with a warning; migrate it before adding inline runtime or evaluation settings. See single-file examples and the spec workflow.

--only and --skip preserve dependency safety: a selected step cannot refer to an omitted predecessor. Keep step targets local and configure runtime.gpu in tunedtensor.json to execute GPU work on your AWS instance. Adapter and foundation workflows use the same local pipeline. The TT agent may describe, validate, and prepare a sealed pipeline action; only deterministic /approve handling can execute it.

Long pretraining uses BF16, warmup plus cosine decay, gradient clipping, periodic metrics, and atomic rolling checkpoints. --resume <run-directory> starts or resumes one stable run identity; completed stages are verified and skipped, while an interrupted pretrain step restores model, optimizer, scheduler, RNG, counters, and token cursor. See Long foundation runs before an unattended job.

Cloud runs and account reporting

Use the same TT token for cloud operations and managed inference. A BYO model selection continues to work while you operate cloud runs with that token.

tt auth login
tt cloud push --file tunedtensor.json
tt cloud runs watch <run-id>
tt cloud runs report <run-id>
tt cloud runs list
tt cloud models list
tt usage
tt balance

Teacher-label raw data

Label unlabeled inputs with a stronger teacher model, review them, and train an adapter on the result. Labeling uses TT credits. Add a few high-quality examples to the spec first: the teacher follows them as the output format.

In the agent, say what you want, for example "label emails.jsonl with the teacher". It checks the spec and proposes the job. Approving it pushes the spec, uploads the file and starts labeling in the cloud. Ask for the status at any time. When labeling finishes, the agent asks whether to use only rows you reviewed or also the teacher's drafts, proposes the download, and then proposes the dataset_prebuilt edit.

The same steps as commands:

tt cloud push --file tunedtensor.json                         # prints the cloud spec ID
tt cloud label upload emails.jsonl --spec <spec-id> --watch   # {"input": "..."} per line
tt cloud label status <job-id>
tt cloud label rows <job-id> --status labeled
tt cloud label accept <job-id> --rows 0,3,12                  # or --all after spot checks
tt cloud label edit <job-id> --row 5 --output "..."
tt cloud label download <job-id>                              # writes datasets/<job-name>/

download keeps accepted and edited rows (add --include-unreviewed for teacher drafts), drops rows blocked for secrets, removes duplicate inputs, drops inputs that received conflicting labels, and holds out a deterministic validation split (--validation-ratio, default 0.1). It prints the dataset_prebuilt paths to add to tunedtensor.json.

tt cloud specs, tt cloud datasets, tt cloud label, and tt cloud models expose the corresponding account operations; each command's --help documents its arguments. Hosted training submission is retired. Use your AWS GPU with tt pipeline run; historical cloud reports remain accessible.

With a TT token, the local agent can inspect cloud resources, run reports, balances, transactions, and managed usage. It can prepare cloud spec edits for /approve. Those proposals show the API origin and remain bound to the token and origin used during preparation. After changing either, return to the original account and origin or prepare a new proposal. Cancelling older hosted runs remains an explicit CLI command. Without a TT token these account tools are unavailable, while local workflow commands and BYO inference continue to work.

tt usage reports how much of the weekly managed inference limit remains, when it resets (Monday 00:00 UTC), and the server-configured daily request allowance. When the weekly limit is used up, managed requests are refused until the reset; BYO provider keys are unaffected. Check tt usage before a long session to avoid hitting the limit mid-task. Each provider completion, including a tool follow-up, consumes one request; failed and cancelled requests also count. Weekly cost includes reservations for running requests and estimates for interrupted requests or missing final costs. Token/cost coverage shows which requests have provider usage reports, so incomplete reporting is not mistaken for zero consumption. The displayed provider cost is informational, not a charge to TT credits. BYO provider usage is separate. tt balance shows TT credits for hosted labeling and historical training charges, plus recent transactions; tt topup opens checkout.

Publish a local report for review in the dashboard:

tt publish <local-run-id> --dry-run
tt publish <local-run-id>
tt cloud runs archive <published-run-id>

Publishing shows the upload before confirmation and includes the spec, metrics, prompts, and example model outputs. Archiving is supported for inactive local reports published to the dashboard. Local model weights stay on your machine. Use --json with reporting commands for scripts. TUNED_TENSOR_URL or --base-url selects an alternate Tuned Tensor API deployment.

Quick start

Create a project on your development machine:

mkdir support-adapter && cd support-adapter
tt init \
  --name "Support Adapter" \
  --model Qwen/Qwen3.5-2B

Edit the generated tunedtensor.json, replacing the placeholder system prompt and both placeholder examples. For a local GPU, tt hardware reports this machine's training and serving capacity. For an AWS GPU, add the following runtime section to your tunedtensor.json, replacing the instance, region, profile and SSH settings with your own:

{
  "runtime": {
    "gpu": {
      "provider": "aws",
      "profile": "research",
      "region": "eu-west-1",
      "instanceId": "i-0123456789abcdef0",
      "user": "ubuntu",
      "identityFile": "~/.ssh/research-gpu.pem"
    }
  }
}

The instance must already be running and reachable over SSH. See AWS GPU setup for prerequisites, authentication, transfer behavior and recovery. tt hardware always inspects the machine running the CLI; tt doctor uses gpu to check the remote runtime. Then validate, prefetch the base model locally, preview, and run:

tt validate tunedtensor.json
tt models prefetch tunedtensor.json
tt doctor tunedtensor.json
tt pipeline run --spec tunedtensor.json --dry-run
tt pipeline run --spec tunedtensor.json

tt models prefetch shows a progress bar with transfer rate and ETA. Large downloads survive flaky networks: TT pins the requested revision to one commit, retries interrupted transfers with backoff (TT_HF_DOWNLOAD_MAX_ATTEMPTS, default 10), and keeps completed files in the Hugging Face cache, so running the command again resumes instead of starting over.

These commands read runtime and evaluation settings from the core spec. The preview needs no GPU or AWS connection. Results return to the local run and model stores:

tt runs report <run-id>
tt models verify local-<run-id>

Train on a Hugging Face dataset

Point dataset_prebuilt at files in a Hugging Face dataset repo instead of local paths. It works like base_model: name the repo, pin a commit, and TT downloads it into the same Hugging Face cache.

{
  "dataset_prebuilt": {
    "huggingface": {
      "repo": "SetFit/tweet_sentiment_extraction",
      "revision": "0d5359cfbb2470332f2b82e62269cf755e0ac5c3",
      "columns": { "input": "text", "output": "label_text" }
    },
    "training": "train.jsonl",
    "test": "test.jsonl"
  }
}

tt datasets prefetch tunedtensor.json   # download with progress; rerun to resume
tt datasets verify tunedtensor.json     # offline checksum check of the cached files
tt pipeline run --spec tunedtensor.json # downloads automatically if still missing

In tt agent, ask it to download a dataset or model. It proposes the same tt datasets prefetch or tt models prefetch command and runs it after you approve. The source does not have to be in the spec yet: download first, look at it, then ask the agent to adopt it, and it proposes a spec edit pinned to the commit the download resolved. A download alone needs a training filename, not a validation split or column mapping; those are required when configuring training. Approval checks the workspace, spec and adjacent legacy runner configuration, then runs a private snapshot of the reviewed settings. Later edits cannot redirect the running download. After a failure or cancellation, ask for a new download proposal to resume from the cache.

Downloads and the lock file

Any Hugging Face dataset files or model can be downloaded without editing a spec:

tt datasets prefetch --repo HuggingFaceH4/ultrachat_200k \
  --training data/train_sft-00000-of-00003.parquet --test data/test_sft-00000-of-00001.parquet
tt models prefetch --repo Qwen/Qwen3-4B-Instruct-2507   # add --revision <sha> to pin

Direct downloads also work before a spec exists or while it is invalid; those workspaces use the default cache. An approved agent download reports the commit from that command's result for a later spec edit.

Files stay in the Hugging Face cache (HF_HOME, or runtime.paths.modelCache). Each download also writes a manifest with the repo, commit, file digests and location under ~/.tuned-tensor/assets/, so downloads are easy to find later:

tt assets list                 # everything downloaded, newest first
tt assets path org/name        # where it is on disk (org/name@<sha> for one revision)
tt assets verify [org/name]    # re-hash the files against their manifests

tunedtensor.lock.json beside the spec pins what a project trains on, the way .terraform.lock.hcl pins providers: the dataset and base-model repo, commit, and each file's Hugging Face content digest (sha256: for LFS files, git-sha1: for small git files). tt datasets prefetch and tt models prefetch for the spec write it, and the first real run records any entry it is missing. After that, tt pipeline run:

• uses the locked commit when the spec's dataset does not pin revision;
• re-hashes the cached bytes and stops if the spec names another repo or revision, a cached file's digest differs, or a locked model file is missing. If the change is intended, rerun the matching prefetch command for the spec to update the lock.

A first local run with an empty model cache downloads and locks the model before training starts. Dry runs never download assets or write the lock.

An unpinned dataset stays on its locked commit, including for tt datasets prefetch. To move it to a newer commit, pin that revision in the spec and prefetch again.

Commit the lock file with the spec. Runs that load the base model from runtime.paths.baseModel or on an AWS GPU host check only the dataset.

• Split paths are files inside the repo: one .jsonl or .parquet file per split (pick one shard of a sharded split). CSV is not supported yet.
• Parquet supports Snappy, gzip, zstd, Brotli and LZ4 compression, as well as uncompressed files.
• Without columns, files must already be TT chat JSONL whose system message matches the spec. With columns, each record's input and output fields become chat rows that use the spec's instruction. Parquet files require columns. For chat datasets such as HuggingFaceH4/ultrachat_200k, use "columns": { "messages": "messages" }: each conversation's first user turn and the assistant reply that follows become one example; later turns are not used. Records missing a mapped field are skipped. Evaluation splits keep the first of any duplicate input, and training drops inputs that also appear in an evaluation split. The run report lists every skipped or dropped count.
• Without revision, TT uses the cached main snapshot and records the commit it resolved. tt validate and tt datasets prefetch flag the spec as unpinned. Pin the commit for reproducible runs.
• Runs never contact the Hub after the download. Each run checks every file against its content-addressed blob and records the repo, commit, file SHA-256s and conversion counts in run_metadata.dataset_source.
• Gated datasets use HF_TOKEN, as model downloads do.

Run records and audit

tt pipeline run prints each stage live with the time since the run started. --verbose adds training and evaluation process output, and --quiet hides both. Full logs stay in the run directory either way.

Every run report has a provenance block recording what produced it:

• TT, Node and platform versions, and the digest of the locked Python runtime;
• where training ran (local, or the AWS instance and region);
• the spec file and snapshot digests;
• the base model id, revision and content fingerprint;
• every dataset file's SHA-256, the compiled training data digest, and the evaluation split and seed;
• the hyperparameters and the training exit code.

The run's event log is hash-chained: each event records the previous event's hash. When a run completes, TT records the report's SHA-256.

tt runs audit <run-id>

tt runs audit checks the event chain and re-hashes the report, then prints the stage timeline with durations and the provenance. Its verdict is verified, incomplete (older TT versions or unfinished runs) or tampered. tampered exits with code 2 and names the edited event or file; removing the completion event or emptying the log also counts as tampering.

tt publish runs the same audit and uploads the summary with the report. It refuses tampered evidence. It also stops on unverifiable evidence, such as runs from older TT versions or logs with unreadable lines, unless you pass --allow-unverified. The chain makes edits evident; it is not a signature, so anyone with write access to the store can rewrite the whole log.

Serving still runs on the machine executing tt serve and requires a local Linux/CUDA GPU, even when training used AWS:

tt serve local-<run-id>

Local currently certifies text SFT with Qwen/Qwen3.5-2B (pinned to snapshot 15852e8c16360a2fea060d615a32b45270f8a8fc), nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, and meta-models/Muse-Glimmer-30B, using LoRA/PEFT and CUDA. The 30B/3B-active Nemotron path is intended for DGX Spark-class unified memory, uses activation checkpointing, and adapts bounded shared attention/Mamba projections rather than every routed expert matrix. Muse Glimmer is a vision-language checkpoint whose text tower is fine-tuned through the image-text-to-text loader with the vision tower frozen and unused. Evaluation may use CPU; packaged local serving requires Linux and NVIDIA CUDA. AWS execution transfers the required datasets and model files to your instance and returns outputs locally; normal cleanup removes the remote staging directory.

Activation is optional and requires the run to pass a configured generalRegression gate. Without that suite, tt models activate fails closed. tt serve active also fails closed if nothing is activated — it does not silently serve the protected base model. Once activated, use tt serve active.

tt serve base does not automatically inject the adjacent project spec. Pass --spec tunedtensor.json when the server should enforce its instructions. Serving uses a separate locked vLLM runtime on Linux/CUDA. A tuned launch exposes both the adapter and its base model, sharing weights, with upstream streaming and native tool-call parsing. tt serve <target> --print-client-config pi exports isolated Pi configuration; tools no longer need to be disabled. See Local serving and coding harnesses for Pi/OpenCode setup, resource budgets, hardware limits, and the verified integration scope.

Serve a foundation checkpoint

Serve a completed pretrain, fine-tune, or RL model directory with its training tokenizer:

tt serve foundation --checkpoint /path/to/run/pretrain/output --tokenizer /path/to/run/tokenize/output/tokenizer.json

Use the model artifact directory containing config.json and model.safetensors, not a resumable optimizer checkpoint. The API model name is foundation. --spec tunedtensor.json applies a foundation spec's system prompt. Existing port, authentication, sampling, streaming, and client configuration options apply. The context limit defaults to the checkpoint's sequence_length; a smaller --context-length is allowed. Foundation serving supports text completions and chat with system/user/assistant messages, without tool-call parsing.

TT exports the weights and tokenizer to a temporary GPT-2 snapshot and uses the existing pinned vLLM server on Linux/CUDA. Training artifacts are unchanged; the temporary export is removed when the server exits. Export requires temporary disk space for a copy of the model weights.

Export parity tests run on CPU as part of npm test. To run them directly:

uv run --frozen --project training/foundation --group test python -m unittest discover -s training/foundation/tests -p test_foundation_export.py

Useful discovery commands:

tt runs list
tt models list
tt models active
tt status

For the full command reference, including long-example policies, eval token budgets, evaluation caps, and local model serving, see the CLI docs.

Development

npm install
npm run build
npm run dev
npm run typecheck
npm test
npm run test:workflows  # Focused product workflow regressions; no model key or GPU
npm run check           # Typecheck, all tests, and build (the release gate)

See testing and workflow contracts for what the suite guarantees, how to evaluate a real agent with npm run eval:agent, and the remaining steps toward the full conversational product promise.

License

Apache-2.0