JARVIS - Local-First Desktop Assistant

Install

npm install -g @ashutosh0x/jarvis
jarvis

Or without installing anything permanently:

npx @ashutosh0x/jarvis

No API key is required to start. Web search works out of the box. Every key you add unlocks a feature, and the app degrades honestly without one rather than erroring — run jarvis doctor to see exactly what is and is not available on your machine:

  Jarvis 0.1.0

  Runtime
    ✓ Node                   v22.14.0
    ✓ Platform               win32 x64
    ✓ Electron               installed
    ✓ Interface built
    ✓ jarvis command         %LOCALAPPDATA%\Jarvis\bin\jarvis.cmd

  Optional services
    · GEMINI_API_KEY         unset — conversational answers disabled
    ✓ Ollama                 http://127.0.0.1:11434
    · SearXNG                unset — using public search providers

Everything marked · is optional.

The jarvis command

Whichever way you install it, jarvis becomes a command you can type in any terminal. Installing an app normally gives you a Start-menu entry, not a command, so setup closes that gap itself: it writes a small launcher to a per-user directory and puts that directory on your PATH.

| | | |---|---| | Windows | %LOCALAPPDATA%\Jarvis\bin\jarvis.cmd, registered in HKCU\Environment | | macOS / Linux | ~/.local/bin/jarvis, with a guarded block in your shell rc if that directory is not already on PATH |

jarvis link      # do it now, if setup could not
jarvis unlink    # undo it completely
jarvis doctor    # shows where the command resolves from

What it will not do:

• No setx. It truncates PATH at 1024 characters and rewrites REG_EXPAND_SZ as REG_SZ, so %VAR% entries stop expanding. The registry is written directly, preserving the original value kind. (The PATH on the machine this was built on is 2,152 characters — setx would have destroyed it.)
• No elevation, no machine-wide change. HKCU only.
• No second jarvis. If one is already on PATH — npm i -g puts one there — it is left alone and reported rather than shadowed.
• No claimed success. The previous PATH is saved to disk first, the new value is read back after writing, and the change is verified against the stored PATH rather than this process's environment.

A shell inherits its environment when it starts, so the terminal you ran it from cannot see the change — open a new one. Every message says so rather than implying otherwise.

The package bundles Electron, so the first install downloads a platform binary (~100 MB). That is the price of npm i -g producing a working app instead of a list of instructions. Prefer a native installer? See releases for signed .exe, .dmg, .AppImage, .deb and .rpm builds, each with a SHA-256 checksum.

Use the search engine as a library

The search engine has no Electron dependency and no DOM — it is plain Node, independently tested, and installable on its own terms:

import { search } from '@ashutosh0x/jarvis';

const { results, answer, providers } = await search('rust async runtime comparison');
console.log(answer);      // extracted answer, or null
console.log(providers);   // ['crates', 'github', 'hn', …]

import {
  rrfFuse,          // Reciprocal Rank Fusion, k=60
  bm25Search,       // BM25, k1=1.2 b=0.75
  editDistance,     // Damerau-Levenshtein
  isTimeSensitive,  // does this answer change by the hour?
  gatherAll,        // parallel provider fan-out, returns on quorum
  SearchCache,
  hedgedRace,       // race N RPC endpoints, take the first good one
} from '@ashutosh0x/jarvis';

import { stats, rollup } from '@ashutosh0x/jarvis/metrics';

Search — search, buildProviders, detectIntents, isTimeSensitive, gatherAll, rrfFuse, bm25Search, rankResults, dedupeResults, extractAnswer, verifyAnswer, providerWeights, editDistance, shouldApplyCorrection, htmlToText, SearchCache

Metrics — stats, windowed, rollup, rollupByDay, pruneRaw, makeSample

Networking — hedgedRace, createStickyOrder, backoffDelay, createDedup, createBlockTracker, prioritizeAlerts

Market analytics — dailyReturns, correlation, beta, peerIndex, realizedVol, trailingReturn, drawdown, PEER_GROUPS

The full architecture, feature reference and configuration guide follow below.

JARVIS is a desktop assistant whose intelligence runs entirely on your own machine. Speech recognition, language understanding, retrieval, and vision all execute locally. No model API keys, no network calls to a model provider, and no conversation data leaving the device.

Some features ask the outside world for facts it alone has — a share price, a headline, the state of a blockchain. Those calls send a ticker or an address and nothing else: no transcript, no memory, no conversation. Everything else works with the network unplugged.

It presents as a frameless, transparent 3D visualizer that floats above your desktop, listens continuously, and answers by voice. A companion Android app extends the same interface and control surface to a paired phone over Wi-Fi, and a spoken "mirror my phone" puts that phone's live screen on the desktop with touch and keyboard control.

Contents

• Install
• What makes this different
• Architecture
• Feature reference
  • Listening to a video
  • Connection queries
• Calendar and meetings
• On-chain intelligence
• Web search
• Retrieval engine
• Evaluation
• Android companion
• Screen mirror
• Globe
• Mineral intelligence
• Routing
• Music
• Installation
• Running
• Configuration
• Network ports
• Troubleshooting
• Known limits

What makes this different

Most assistants send your microphone to a datacenter. This one does not.

| Capability | Typical assistant | JARVIS | | --- | --- | --- | | Speech to text | Cloud ASR | Whisper via transformers.js, local | | Language model | Hosted API | Gemma 3 via Ollama, local | | Embeddings | Hosted API | nomic-embed-text, local | | Vision / screen reading | Cloud vision | Gemma 3 multimodal, local | | Conversation storage | Provider servers | Local disk only | | Works without internet | No | Yes, except live data lookups | | Per-query cost | Metered | Zero |

Outbound traffic is limited to fact lookups that cannot be answered locally: keyless web search when a query is search-shaped, quote and news endpoints, and blockchain RPC. Each sends only the subject of the question — a ticker, a search string, an address. Inference never leaves the machine.

Architecture

System overview

Diagram: System overview →

| Colour | Layer | | --- | --- | | Purple | Renderer. Visualizer, voice loop, retrieval, intent routing | | Cyan | Electron main. Service supervision, IPC, LAN listeners | | Green | Local inference. Everything bound to loopback | | Light green | Android companion, reached over Wi-Fi | | Red | External network. The single outbound path | | Amber | Local persistence |

The Electron main process (electron.js) supervises every local service and restarts them on failure. The renderer owns the visualizer, voice loop, and retrieval.

Voice pipeline

Diagram: Voice pipeline →

Two feedback loops are load-bearing. The dashed edges back into the VAD and echo guard are what stop JARVIS transcribing its own voice, and both are required: the ttsActive gate leaks because synthesised audio bypasses Chromium echo cancellation, so the text-level guard catches what the gate misses.

Speech to text is JavaScript, not Python. server/stt-server.mjs runs Whisper (onnx-community/whisper-base.en, q8) through transformers.js on ONNX Runtime, spawned on Electron's own Node with ELECTRON_RUN_AS_NODE=1. The previous implementation shelled out to uv run --python 3.12 --with faster-whisper, which meant every machine needed uv, a Python toolchain and a package download before voice input worked at all — and silently had no voice input if any of that was missing. The wire protocol is unchanged, so anything already speaking to :8770 needs no edit:

| Client sends | Server does | | --- | --- | | binary frames | raw 16 kHz mono PCM16, appended to a buffer | | {"type":"end"} | transcribe what has accumulated, reply, clear | | {"type":"reset"} | discard without transcribing | | | replies {"type":"final","text":"…","ms":123} |

Three gates sit in front of the model, because a language model asked to describe silence will invent words rather than return nothing:

1. Under 250 ms of audio — dropped without transcribing. Note that this path sends no reply at all, matching the old Python server's behaviour exactly. A client that waits for one final per end will wait forever on a blip, so drive it on arrival rather than on a strict request/response pairing.
2. RMS below 0.0015 — the window is effectively silence. Returns "" without invoking the model.
3. The artefact list — a whole-utterance match against "you", "thank you", "thanks for watching", "bye", "." and similar is blanked after the fact. Matching is on the entire utterance, so a real sentence that merely contains "thank you" is untouched.

Requests are serialised. ONNX Runtime is not reentrant for a single session, so two overlapping transcriptions would interleave into each other's output; they queue instead, which preserves the ordering everything downstream depends on.

Measured on this machine, 20 Aug 2026, whisper-base.en at q8, after the model is cached:

| Input | Transcribe | Result | | --- | --- | --- | | 3.38s utterance | 844 ms | exact | | 5.00s window | 1060 ms | exact | | 5.00s window | 1407 ms | exact | | 3s digital silence | — | "", not transcribed | | 3s low room noise | — | "", not transcribed |

The first run downloads the weights into the transformers.js cache. Every run after that is fully offline.

Process supervision

Diagram: Process supervision →

Two invariants are encoded above. Only kill what you spawned: an Ollama the user started is reused and left running on quit. Preload is not optional: Ollama's default keep_alive is 5 minutes, so without the 60-minute preload the first question after any idle period pays a multi-second cold load.

On launch, electron.js starts and monitors:

| Service | Behaviour on failure | | --- | --- | | Ollama | Reuses an existing instance if one is running; otherwise spawns ollama serve, waits for readiness, preloads the model with keep_alive: 60m, and auto-respawns after 15s | | Whisper STT | Runs on Electron's own Node via ELECTRON_RUN_AS_NODE, no Python. Auto-respawns after 15s; port conflicts exit harmlessly and the 30s watchdog reclaims the port if its holder later dies | | Phone bridge | Token-authenticated HTTP listener | | Companion bridge | WebSocket server plus mDNS advertisement | | Downloads watcher | chokidar; new documents are OCR'd and ingested | | Clipboard monitor | Scans for leaked secrets, reports masked hints only | | Active window tracker | 10s cadence | | Finance service | 60s quote cadence |

Services that JARVIS spawns are terminated on quit. Services it merely reused, such as an Ollama you started yourself, are left running.

Feature reference

Voice

Open conversation mode. Every transcript is routed and answered; no wake word is required. Leading "Jarvis" and common mis-hearings are stripped.

• Always-on microphone with adaptive noise-floor VAD
• Deliberate microphone selection, excluding loopback devices such as Stereo Mix which would otherwise capture JARVIS listening to itself
• Streaming speech: each completed sentence is spoken during token generation, cutting time-to-first-word from roughly 5-10s to 1-2s
• Echo guard using word-overlap against recently spoken text, because the synthesis-active flag alone is known to leak
• Self-healing microphone recovery with a 5s watchdog for eventless device death

Listening to a video

Paste or say a YouTube link and JARVIS opens it, listens to it, and says what is being said.

https://www.youtube.com/watch?v=...     start listening
what did he say                         replay the last ~60 words, from the buffer
stop listening                          stop, and report how many passages were spoken

The link is checked before intent detection. A bare URL matches no intent, so it used to fall through to the model and come back as "I encountered an error processing your request".

Two properties the design exists to protect:

The words are the speaker's. No model rewrites the transcript on its way to the speakers. transcriptEngine.js is string comparison with fixed rules, so what JARVIS says is what Whisper heard, minus the repetition the window overlap creates. Summarising is a separate thing you have to ask for.

The audio comes from the player, not the room. Capture is scoped to the player frame through getDisplayMedia rather than system loopback. That is what lets JARVIS speak while still listening — system capture would hear its own voice and transcribe it back into the queue, which compounds into nonsense within a few turns.

| Constant | Value | Why | | --- | --- | --- | | Window | 5000 ms | The floor on lag: Whisper cannot start until it has one. Measured at 1.06–1.41s to transcribe, landing end-to-end around two to three seconds behind the source | | Overlap | 700 ms | A word spanning the cut survives whole in one of the two windows. The engine removes the duplication this creates | | Sample rate | 16 kHz mono | What Whisper wants; the AudioWorklet in youtube-pcm-processor.js does the conversion |

Windows are transcribed concurrently but spoken strictly in order — transcriptEngine.js holds a window until every lower sequence number has arrived, because two overlapping utterances are unintelligible and a re-ordered transcript is worse than a late one.

Connection queries

correlationEngine.js answers how things in the ontology relate. This is the surface that lets a sentence reach it.

how is apple connected to foxconn        paths between two entities
what do apple and foxconn have in common
what is connected to apple               everything adjacent to one entity
show me everything related to apple
what happened around the same time       events that co-occur
what are the bridges                     articulation points — what holds the graph together

Three outcomes are deliberately kept distinct all the way to the spoken sentence, because collapsing them is the tempting bug — "no connection found" and "I have never heard of either of these" sound identical to a listener and mean opposite things:

| Case | Answer | | --- | --- | | The graph has never heard of the entity | Cede — the web may know | | The graph holds both, and no path exists | A finding, said plainly | | The graph holds a path | Report it, with sources |

"Connect" is the most overloaded verb in this assistant: it already means pair my phone, link my calendar, and go online. Every one of those matchers sits further down detectIntent than this one, so an over-broad pattern here would not merely mis-answer a graph question — it would silently break phone pairing. Two guards prevent that. A connection question must name two things, so a bare pronoun or a possessive ("my", "our", "their") disqualifies it; and any utterance containing phone, calendar, wifi, bluetooth, wallet, spotify and the rest of that list is refused outright before parsing starts.

Visualizer

• Icosahedron with Perlin-noise vertex displacement driven by live FFT
• Bass, mid, and treble bands weighted separately
• Frequency-mapped colour, with time-based hue cycling when idle
• Transparent frameless window that floats over other applications
• F2 toggles between orb-only and full HUD

Feedback

Full write-up: docs/FEEDBACK.md

Confirmation on the channels the machine actually has. Nothing vibrates on a desktop — navigator.vibrate is callable in Electron and moves nothing, because there is no motor — so a press is carried by a short animation and a synthesized click, and the vibration channel reports itself unavailable rather than pretending.

• Every heard utterance gets an instant 8 ms acknowledgement, before the answer exists. Speech has no click of its own, so the gap before the first word back was otherwise indistinguishable from not being heard
• Every unprompted event — a phone notification, a download, a whale transfer, a price alert — carries the same rising marker
• Destructive actions get the one effect with a gap in it: pulse, pause, pulse, falling and low. Every other effect is a single gesture, so it cannot be mistaken for a confirmation. A warning falls; a success rises
• Audio is synthesized rather than sampled: no binary assets, nothing to fetch, and the sound is a table of numbers that can be reviewed and tested
• prefers-reduced-motion gates the animation only. It is a statement about motion, not about feedback, and silencing the audio would strip the non-visual confirmation from the user who just asked for less movement
• The paired phone has a real motor, and gets the same vocabulary mapped onto Android's own haptic API — probed by primitive, not by API level

System control

• Application launch through an allowlist
• Volume, brightness, media keys, power state
• Wi-Fi scan, connect to saved profiles, disconnect, and measured link diagnostics reporting real latency and packet loss
• File operations, clipboard read and write
• Windows Settings deep links
• Live CPU, RAM, uptime, and active window telemetry in the HUD

Files and folders

Create folders and files by voice, anywhere inside your own user folders.

"create a folder called notes on the desktop"
"make a file called todo.txt in documents"
"make a file called shopping list saying milk and eggs"

• Parsing is rule-based, never model-driven. These commands write to disk, and a model deciding what "create a file called that thing" means is a model deciding what to name a file on your Desktop
• Confined to Desktop, Documents, Downloads, Pictures, Videos and Music. Containment is checked at a path-separator boundary, so ~/Desktop-evil is not treated as ~/Desktop
• An existing file is never silently overwritten
• Executable types (.exe, .bat, .vbs) are refused. Source files are not — writing .js is the point, and a .js is only dangerous when something runs it, which JARVIS never does
• A name that does not survive sanitising is reported, not replaced with a fallback. A file called untitled appearing because a name was misheard is worse than being told the name was not understood

Writing code

"open vscode and write a binary search in java"
"write a quicksort in python on the desktop"

Gemma writes the file contents; the filename, directory and language are fixed by rule before the model is asked anything. Java and C# get PascalCase names because those languages resolve the type by filename. The file opens in VS Code when an editor was named, and VS Code is resolved to the real Code.exe rather than shelling out to the code shim.

If the model returns nothing, no file is created — an empty file reported as success would be a lie.

Alarms and timers

"set a timer for 40 minutes"
"set an alarm for tomorrow at 2:30 pm"
"set an alarm for an hour and a half"
"set a timer for twenty minutes to check the oven"
"cancel the timer"   ·   "what timers do I have"   ·   "stop"

• Scheduled to the exact instant rather than polled, so a 30-second timer works
• Persists across restarts, and a missed alarm is announced on return rather than silently dropped
• Fires with speech, a synthesised tone and a notification; each is independent, so a suspended AudioContext does not suppress the other two
• Repeats every five seconds until dismissed, stopping after two minutes
• An unresolvable time is refused. "Set a timer for the pasta" asks how long instead of choosing a duration

Waking up

"wake me up tomorrow morning"        ·   "wake me up in the morning"
"wake me up every day at 6"
"wake me up at 6 tomorrow morning"
"set an alarm on Monday at 9am"

Named parts of the day resolve to a time: morning 07:00, breakfast 08:00, noon 12:00, lunch 13:00, afternoon 15:00, evening 19:00, dinner 20:00, night 22:00.

This is the one place the parser uses a default, which sits against the no-guessing rule the rest of the file keeps. The rule survives in the only way that matters: an assumed time is always spoken back concretely — "Alarm set for 7:00 am tomorrow. I have taken morning as 7:00 AM — say a different time if you would rather" — so a wrong default costs one sentence rather than being discovered at 7am.

Two details that are easy to get backwards:

• A day-part beside a number settles the meridiem. "Wake me at 6 tomorrow morning" is 06:00. Without this it takes the roll-forward branch and lands on 18:00, the one reading the sentence rules out.
• "Wake me" implies AM between 4 and 11. Bare hours roll forward everywhere else, but waking is a morning act; "wake me at seven" said at 9am means 07:00 tomorrow, not 19:00 tonight.

Recurrence — every day, daily, every weekday, every Monday — is carried on the alarm and re-armed by the scheduler after each firing. A repeat advances by whole calendar days rather than by 86,400,000 milliseconds, because those differ across a DST change and a 7am alarm quietly becoming a 6am alarm twice a year is a bug discovered at 6am.

The daily routine

"wake me up at 7 every day"           ·   "move lunch to two"
"remind me every day at 8 to have breakfast"
"what's left today"                   ·   "what is my routine"
"no dinner reminder"                  ·   "set my morning walk for 7"

Named slots — wake, walk, breakfast, study, lunch, dinner — each firing daily on its own. Seeded with a starting template on first run, because a routine that starts empty needs six spoken commands before it does anything.

• The routine owns only its own alarms. Entries are tagged source: 'routine', so editing lunch never cancels the alarm you set for a meeting.
• "What's left today" reads the live schedule, not the template, so it stays true after you move a slot mid-morning.
• Slots are matched longest-alias-first, so "set my morning walk" edits the walk slot rather than the wake slot.

[!IMPORTANT] A wake-up alarm belongs on your phone, not here. The desktop scheduler runs in the renderer: if JARVIS is closed at 6am, nothing fires, and nobody leaves a desktop awake and unlocked overnight to be woken by it.

With the companion paired, every alarm is mirrored to Android and scheduled through AlarmManager.setAlarmClock(), which survives a closed app, a locked screen and Doze. Both copies ring; two alarms is a far better failure than none. See Android companion.

Diagram: Waking someone up →

Calendar and meetings

"connect my calendar"
"what are my meetings"        ·   "when is my next meeting"
"schedule a meeting"
"create a meet room"
"I know about the meeting"

Scheduling is a conversation. JARVIS asks what to call it, what it is about, when, how long, and who to invite, then reads the whole thing back before creating anything:

You    schedule a meeting
JARVIS Of course, Sir. What should I call this meeting?
You    project review
JARVIS "Project review". And what is this meeting about?
You    Q3 roadmap and hiring
JARVIS Noted. When should I schedule it, Sir?
You    tomorrow at 3 pm
JARVIS 3:00 PM tomorrow. How long should it run?
You    an hour
JARVIS Anyone to invite? Say their email addresses, or "no one".
You    no one
JARVIS "Project review", 3:00 PM tomorrow for 1 hour, ending 4:00 PM.
       About: Q3 roadmap and hiring. Shall I create it, Sir?
You    yes
JARVIS Creating it now, Sir.

• Background awareness escalates rather than repeats: one warning at 30 minutes, then 10, 5 and 1, each phrased differently. Eight identical five-minute reminders train you to ignore the one that matters
• The calendar is fetched every five minutes, but checked against local clocks every twenty seconds, so alert timing does not depend on when a network call landed
• A failed poll keeps the last known schedule. Announcing "no meetings today" because one fetch failed would be a fabrication
• Spoken email addresses work — "john at example dot com" — and an address that cannot be parsed is refused rather than quietly inviting nobody
• The model is asked for exactly one thing: a better title when you gave a generic one, which you then confirm. Nothing that lands in your calendar is model-decided

[!IMPORTANT] Creating a Google Meet link requires a paid Google Workspace account. On a personal Gmail the event is created without one — the API returns no link and no error — and JARVIS says so rather than implying a link exists.

Running in the background

"start with windows"     ·   "don't start with windows"
"hide yourself"          ·   click the tray icon to bring it back

Closing the window hides JARVIS to the tray rather than quitting, so alarms still fire and meetings are still watched. Only Quit Jarvis from the tray menu actually exits.

• Autostart is registered through the OS login-items API, and starts hidden — someone who wanted a window on every boot would not need autostart
• A single-instance lock is claimed before anything else. This is not a nicety: JARVIS spawns a speech server, a TTS server, a vision server and Ollama on fixed ports, and runs a microphone listener and an alarm scheduler. Autostart plus a manual launch is the ordinary case on day one, and a second instance would fight the first for all of it. Launching again surfaces the running window instead
• Autostart can only be registered from an installed build. In a development run process.execPath is electron.exe, and registering there would put a bare Electron runtime in your startup that launches and shows nothing. JARVIS refuses and says so, rather than leaving an entry that looks installed and does nothing

What JARVIS can reach

JARVIS runs as you, not as Administrator, and that is deliberate.

Unelevated already covers everything you do day to day: your files, launching programs, reading the process list and network state, the microphone and the screen. What it does not cover is writing to Program Files, the Windows directory, or another user's data — none of which JARVIS has a reason to touch.

Running it elevated would mean a misheard word, or anything that reached the renderer through a web result, inherits Administrator. JARVIS acts on speech recognition, which mishears; that is the whole reason the file commands are rule-parsed and the write path is allowlisted in the first place.

To widen its reach, name the directories:

JARVIS_EXTRA_ROOTS=D:\Projects;C:\Work

Absolute paths, semicolon-separated. A relative entry is dropped with a warning rather than resolved against the working directory, because a typo'd Work becoming <cwd>/Work would grant a directory nobody chose. Point it at a drive root if you genuinely want that — the difference that matters is that it is your decision, written down, rather than an implicit consequence of enabling autostart.

Screen and documents

• Screen reading through Gemma 3 vision, fully offline. The captured question is passed through, so "what error is showing" reaches the model intact
• Optional Unlimited-OCR server for dense text
• Downloads are watched, OCR'd, and ingested into memory automatically
• Your Android screen, live on the desktop with touch and keyboard control — see Screen mirror. "take a phone screenshot" grabs the current frame and describes it through the same local vision path

Knowledge

• Hybrid retrieval over local memory, detailed below
• Keyless web search with a three-provider failover chain — DuckDuckGo HTML, DuckDuckGo Instant Answer, then Wikipedia — injected as cited context for search-shaped queries. The chain exists because the HTML endpoint starts serving a captcha once an IP is flagged, which silently emptied every result
• Finance watchlist with crossing alerts. Read-only by design: no order placement code exists anywhere in the project

Markets and quantitative analysis

Every number here is computed by tested code from measured data. The language model is never asked to calculate a financial figure, because it cannot be trusted with one and a wrong figure stated confidently is worse than no answer.

• Live quotes with day change, resolved name to ticker
• Single security — src/js/services/quant.js: annualised return, volatility, Sharpe, Sortino, maximum drawdown, beta and alpha against a benchmark, correlation, R², historical VaR, expected shortfall, information ratio, tracking error, up/down capture, and Black-Scholes pricing with greeks
• A book of holdings — src/js/services/portfolio.js: covariance, risk contribution, risk parity, minimum variance, maximum Sharpe, diversification ratio, and portfolio-level VaR and expected shortfall
• A name against its peers — sectorMove.js: how much of a move the sector explains and how much belongs to the company
• SEC filings through a pinned fetch guard, edgarGuard.js
• Disclosure venues beyond EDGAR — see below
• Headlines from Google News with Bing failover, keyless

Three choices in here are load-bearing rather than incidental.

VaR is historical simulation, never parametric. The Gaussian form (mu - z*sigma) is one line shorter and wrong in exactly the situation the number exists for: return distributions have fat tails, so it understates the 99th percentile precisely when that matters. The implementation reads the quantile off the observed returns and refuses fewer than 30 observations rather than resting a 99% loss estimate on a single bad day.

R² gates the interpretation of beta and alpha. Measured against the S&P 500 over 250 sessions, Micron's R² is 0.283 — the benchmark explains 28% of its variance, so its beta of 3.29 and alpha of 170% are weakly determined and are reported as such. Without that gate the assistant would speak a 170% alpha as though it meant something.

Dollar weight is not risk weight. A 60/40 book is roughly a 94% equity-risk book, which is arithmetic rather than opinion: it is what riskContributions() returns for those weights, and it is checked against that case in the tests.

Peer and portfolio analysis

Two questions the single-security metrics cannot answer, each with its own module and voice intent.

SECTOR_QUERY — "decompose Micron's move", "break down the memory sector". The move is split into the part the peer group explains (beta times the sector's move) and the part that belongs to the company. Measured 29 July 2026: Micron fell 9.94% while its peer group fell 4.96%; with a beta of 0.91 that leaves -5.40% as its own, while Western Digital's flat day was +4.77% of relative strength. Group mode ranks every member by that residual, because the largest faller is often just the highest beta.

PORTFOLIO_QUERY — "how risky is my watchlist", "what would risk parity do", "minimum variance weights for MU, SNDK, WDC". Holdings are aligned by date, never by index, because two venues do not share a trading calendar. The covariance inverse refuses a singular matrix rather than returning the enormous offsetting weights that a collinear book produces, and a short position in the minimum-variance solution is surfaced rather than clipped — "hold none" and "sell short" are different instructions.

Both modules state their limits instead of implying them with a null. A holding that only listed three weeks ago truncates every other series to match; the analysis names it and says that dropping it would widen the window.

Disclosure venues beyond EDGAR

The memory industry is mostly not American, so an EDGAR-only assistant answers "no filings" for half of it and is wrong every time. edgarGuard.js carries a venue registry naming where each issuer's filings actually live, and every entry records the date it was checked rather than a permanent claim.

| Issuer | Venue | Reachable how | | --- | --- | --- | | Micron, Sandisk, Western Digital | SEC EDGAR | Atom feeds, keyless, declared User-Agent | | SK hynix | Both — SEC since 9 Jul 2026, and DART | 6-K and 424B4 on EDGAR; business reports on DART | | Samsung Electronics | Korea's DART | Open API, free key required | | CXMT | Shanghai STAR Market since 27 Jul 2026 | HTML announcements only | | YMTC | None — privately held | No public filings of any kind |

Probed live on 30 July 2026, because pasted endpoint lists have been wrong repeatedly in this project:

• HKEX publishes real RSS. Two feeds are wired and return 25 parsed items each. They are exchange-wide, not per-company.
• DART advertises no RSS anywhere. Its Open API is real and returns clean JSON, but rejects unregistered callers with {"status":"010"}. The key is free, so this follows the Alchemy and Helius pattern: dormant without one, and it says so rather than failing obscurely.
• SSE serves announcements as HTML only — no feed, no public API.
• SZSE does not complete a fetch from here at all.

The registry exists because of a specific failure. Its first version asserted that CXMT was "a private Chinese DRAM maker with no US listing." That was true when written and false three days later: CXMT listed on the STAR Market on 27 July 2026, rose 466% on debut, and now trades as 688825.SS. A registry that hardcodes a company's status will state a falsehood the moment that status changes, so entries carry a venue and a checked-on date instead.

On-chain intelligence

Read-only by construction. Only a hard allowlist of JSON-RPC read methods is ever sent; there is no signing code, no transaction construction, and no private key handling anywhere in the project.

The governing rule is the same one the quant engine follows: the chain is the source of truth, and anything the chain cannot prove is not claimed. An address with no ENS name stays an address. No exchange or entity is ever named from a guess.

Address and contract reads

• Native and ERC-20 balances, gas, nonce, across Ethereum, Arbitrum, Base, Optimism, Polygon and BNB Chain
• ENS forward and reverse resolution, implemented from a pure keccak-256 in src/js/services/keccak.js and verified against public vectors
• Transaction decode: status, native value, and every ERC-20/721 Transfer in the receipt, resolved to symbols and exact decimal amounts
• Contract classification through ERC-165 supportsInterface and an ERC-20 probe. Classification only; this is not a vulnerability auditor
• Cross-chain portfolio. With an Alchemy key this returns everything a wallet holds, priced; without one it falls back to scanning known tokens per chain
• Solana wallet assets and recent activity through Helius, including native SOL balance and USDC/USDT supply

Real-time whale stream

A websocket subscription to new block headers. Each confirmed block is scanned for large movements, and everything announced is a fact read out of that block.

Diagram: Real-time whale stream →

• Token flows, not just native. Most large value on Ethereum moves as stablecoins. Sampled over five live blocks: 0-2 native ETH whales versus 16 token movements
• Token decimals are verified on chain with a decimals() call before any amount is decoded. Reading a 6-decimal token as 18 turns $4M into $4
• One transaction is one movement. An arbitrage route through several pools emits the same tokens repeatedly; a live drill caught the same 14,050 WETH being announced three times. Transfers are now grouped per transaction, the source is the address that only sends, the destination the one that only receives, and the hop count and any round trip are stated
• Ranked across assets by measured USD. 100 ETH has more raw units than 4,000,000 USDC, so unit ordering picks the wrong headline
• Stablecoin issuance. A mint is a Transfer from the zero address and a burn is one to it, so supply changes need no label database. Live-verified against mainnet: a 5,414,317 USDC mint, and in one hour DAI net +6.9M against USDC net -6.0M
• Address context on both ends: ENS name, contract or wallet via eth_getCode, transactions sent, ETH held. The display carries full addresses and the transaction hash; speech carries the readable form
• Operational hardening: exponential backoff with jitter, 30s heartbeat and 90s silence detection, gap detection with in-order backfill through the same code path as live blocks, and bounded dedup so memory stays flat

What is deliberately not built

| Asked for | Why not | | --- | --- | | Exchange labels ("from Binance") | Not on-chain data. It requires a proprietary attribution database; naming a wallet on a guess is the one thing that would make these alerts untrustworthy. Arkham is supported with your own key, and its labels are spoken attributed | | Wallet classification by the model | A 4B model producing "institutional accumulator" is a confabulated verdict, not analysis | | Mempool alerts | Pending transactions get dropped and replaced. An alert about a transaction that never lands is misinformation | | Global Solana whale scanning | Measured: the Helius socket delivers over 200 token-program events in 15 seconds. Filtering that firehose is not something this machine does while also running voice | | Bitcoin monitoring | A different data source entirely, and none is connected |

Provider keys

All optional. JARVIS runs keyless and degrades honestly, saying which chains it can read and why one is missing.

| Key | Unlocks | Without it | | --- | --- | --- | | ALCHEMY_API_KEY | Full wallet holdings with prices, faster RPC, keyed websocket | Public endpoints, known-token scanning only | | HELIUS_API_KEY | Solana wallets, activity, stablecoin supply | No Solana | | DUNE_API_KEY | Aggregate analytics: top holders, USD-priced flows | Those queries state the key is needed | | ARKHAM_API_KEY | Entity labels, spoken with attribution | Addresses stay addresses | | GOOGLE_MAPS_API_KEY | Globe: country/state/street geocoding, place photos, weather, air quality | Cities only, from the bundled gazetteer; Nominatim, Wikipedia and USGS still work | | LUMA_API_KEY | Globe: events from one Luma calendar (needs Luma Plus) | No events layer; every other globe feature is unaffected | | AVIATIONSTACK_API_KEY | Globe: real flight routes with origin and destination | Live aircraft still shown from OpenSky, but a route query reports traffic over the corridor rather than named flights | | WINDY_WEBCAMS_API_KEY | Globe: ~70,000 opt-in public webcams worldwide | Camera layer still covers London and Singapore road cameras | | PARSE_API_KEY | Globe: refreshing the company ranking, and per-company revenue history. Metered — one credit per uncached call | The 11,222-company crawl in data/ still draws in full; it is a snapshot rather than today's prices |

Networks are discovered, not assumed: each candidate endpoint must return the chain ID it claims before it is used. On the free Alchemy tier this correctly rejects Optimism and Polygon, which answer 403, rather than failing later with a confusing error.

Measured provider limits that shape the design: Alchemy's free tier caps eth_getLogs at 10 blocks, 1rpc at 50, and drpc handles a few hundred but refuses under load. Wide-range log queries are therefore chunked at 50 blocks across the keyless pool, and any chunk that fails is reported rather than silently dropped — "nothing happened this hour" and "I could only read half the hour" are different answers.

Fund tracing

src/js/services/tracer.js implements the deterministic Approximate Personalized PageRank from the TRacer paper, with its tracing-tendency and weighted-pollution strategies, plus structural pattern detection (amount consistency, cycles, consistent chains). It reports pattern presence, never a verdict. The algorithm is tested and works; live tracing needs address history, which public RPC cannot enumerate, so it awaits an Etherscan-family key.

Web search

webSearch.js (main process) and src/js/services/webSearchIntent.js (renderer) answer questions from the live internet. Split along the process boundary, not by topic: the renderer cannot fetch these origins because CORS blocks it, and Rollup cannot take named imports from a CommonJS module.

Why it exists

There was no web search. search about elon musk was classified TYPE_TEXT — the dictation intent — so asking for a search typed the words into whatever window had focus. Anything that instead reached AI_COMMAND was answered by the local model, which has no network access. It did not decline; it invented:

"search about elon musk"       -> "...recognized as a trillionaire in US dollars ."
"list latest vulnerabilities"  -> "According to OpenCVE, Google released Chrome 151
                                   with patches for 382 vulnerabilities"
"latest cve number of chrome"  -> "According to Google's Chrome Releases,
                                   CVE-2026-15905 is the latest critical vulnerability"

Those citations are fabricated. A fabricated CVE number is worse than a refusal.

Pipeline

Diagram: Pipeline →

Providers are measured, not assumed

HTML scraping was tried first and does not work:

| Endpoint | Result | | --- | --- | | html.duckduckgo.com | HTTP 202 + challenge page, 0 results | | lite.duckduckgo.com | HTTP 202 + challenge page, 0 results | | mojeek.com | HTTP 200, body is an altcha CAPTCHA | | searx.be | HTTP 200, JSON output disabled |

The first DuckDuckGo query of a session usually succeeds, which makes this especially deceptive: it looks like it works until it is used twice.

Keyless general open-web search is not available in 2026. Google's Custom Search JSON API closed to new signups in 2025 and shuts down on 1 Jan 2027; Bing's Search APIs were retired on 11 Aug 2025; Brave withdrew its free tier. So the providers below are the keyless endpoints that are official, each measured before being added:

| Provider | Measured | Intent | | --- | --- | --- | | DuckDuckGo Instant Answer | 361 ms | general (sourced abstract) | | Wikipedia | 541 ms | general (encyclopedic) | | Google News RSS | 642 ms | news, anything current | | Hacker News (Algolia) | 831 ms | discuss | | crates.io | 1147 ms | code (Rust) | | Open Library | 1259 ms | book | | NVD | 1462 ms | security | | GitHub repos | 1523 ms | code | | Stack Overflow | 1555 ms | code, discuss | | arXiv | 1973 ms | academic | | npm | 2078 ms | code (JS) | | Brave | — | general, only with BRAVE_API_KEY |

Probed and rejected: GitHub code search (HTTP 401, needs auth), Semantic Scholar (HTTP 429), Reddit (HTTP 403 to datacentre traffic).

Gather, don't race

Providers here are complementary rather than interchangeable — a Rust question wants the crates.io entry and the GitHub repo and the Stack Overflow thread — so gatherAll collects everything that arrives inside the budget instead of resolving on the first success.

The early exit counts providers, not results. Counting results was tried first and silently destroyed the feature: Google News alone returns six, which satisfied a result quota instantly and ended the query before any other source replied — measured as answered 1: google-news on every single query, a first-wins race wearing a gather's clothes.

rrfFuse merges the ranked lists by position only (k=60), because GitHub stars, Stack Overflow votes and news recency cannot be normalised against each other. Provider weights are derived from the query, after plain RRF put npm's uniffi-bindgen-react-native first for "best rust crate for async runtime" — an off-target index's rank-1 beating a relevant index's rank-2.

Query understanding

Spelling and entity correction run concurrently with the search, so they cost nothing when nothing needs correcting, and the corrected query is only re-run when the original returned fewer than three results — and only kept if it did better. A bad suggestion cannot make results worse.

Jarvis auto-corrects your spelling. Mistype a word and it recognises what you meant, fixes it, and shows you what it searched for — so a typo never silently returns nothing.

There is no hardcoded dictionary. Building one from the local corpus was tried and measured useless: 721 feed items yield 2652 "entities" that are almost entirely filing boilerplate (Filer, Filed, AccNo), and the result knew none of the terms people actually mistype. Wikipedia's search API knows all of them, live, and stays current for free.

Two kinds of correction, handled differently:

| Kind | Example | Behaviour | | --- | --- | --- | | Spelling | situtational → situational | corrected silently, shown on screen | | Entity | a misspelt name → the right one | corrected and spoken aloud |

The difference matters. Reading "showing results for situational awareness" aloud after a one-letter typo is noise. But a misheard name resolves to a different person entirely, and answering about someone else without saying so is indistinguishable from being wrong — so entity corrections are always announced.

Suggestions are never applied blindly. The decision is made locally on Damerau-Levenshtein distance relative to word length, and a correction is only kept if it actually returned better results than the original. A bad suggestion cannot make things worse.

Latency

Search returns in 46–956 ms against 31–51 s for the old path, which ran retrieval plus local generation. Repeat queries are 0 ms (cached).

Connection warmth was measured rather than assumed. Node 22's default dispatcher holds pooled connections for at least 120 s idle — far longer than the ~4 s commonly quoted — so no custom undici dispatcher is needed and none is added:

cold (first ever fan-out)   7812 ms
after   0s idle              664 ms
after  60s idle              690 ms
after 120s idle              564 ms

Only the cold start is worth removing, so the three general origins are warmed once, 3 s after launch. The eight specialised providers are intent-gated and left cold. There is no repeating warmer: warmth already survives a session, and periodic warming would be unsolicited traffic to third parties.

Retrieval engine

src/js/services/ragService.js implements hybrid retrieval. Design choices are evidence-driven and each is traceable to a measurement or a paper.

Diagram: Retrieval engine →

Components

| Stage | Implementation | Rationale | | --- | --- | --- | | Sparse | BM25 over a persistent inverted index, incremental on ingest | Re-tokenising the corpus per query measured 104.8ms at 5k chunks on the render thread | | Dense | nomic-embed-text through Ollama, cosine similarity | Degrades to BM25-only when no embedder is present | | Fusion | Reciprocal Rank Fusion, k=60 | Derived from PubHealthBench, where hybrid beat both single-retriever modes. This did not reproduce locally — see Retrieval accuracy | | Expansion | PRF: top 4 chunks, top 6 non-query terms, fused as a separate list at weight 0.5 | Kept separate so a poor feedback pool can dilute but not corrupt the original ranking | | Entities | Normalised Levenshtein, threshold 0.25, after exact-match miss | Input is speech-to-text, so names arrive mangled | | Selection | Late sentence selection, IDF-weighted overlap with lead bias, budget 10 | LongEval's winning system paired plain passages with late sentence selection | | Reranking | Ambiguity-gated LLM rerank, opt-in | See below |

Retrieval accuracy

Speed was measured long before accuracy was, which is backwards: a fast ranker that ranks the wrong passage first is worse than a slow one that does not. eval/ now carries a labelled benchmark — 29 questions over 30 documents, driving the shipped module through ablation switches rather than a reimplementation of it. Full numbers and method in eval/RESULTS.md.

| Configuration | P@1 | P@3 | MRR | ms/query | | --- | ---: | ---: | ---: | ---: | | lexical only (BM25) | 69.0% | 79.3% | 0.737 | <1 | | dense only | 89.7% | 100% | 0.948 | 60 | | hybrid, as shipped | 72.4% | 93.1% | 0.825 | 61 | | hybrid + rerank | 72.4% | 93.1% | 0.825 | 3,243 |

Dense-only beats the shipped hybrid by 17 points at rank 1. The rationale for hybrid fusion came from the literature and did not reproduce on this corpus with this embedding model. Lexical retrieval is in the stack to catch rare proper nouns; dense matched it there (5/5) and beat it on every other question type, so its weight in the fusion is diluting a better ranking rather than protecting against a weakness.

The default has not been changed on that basis, for reasons stated in full in the results: the benchmark's author also wrote its questions, 29 questions makes anything under ~7 points a single labelling choice, and BM25 is the only thing that still works when the embedder is down. But the shipped weighting is currently unsupported by the only measurement that exists, and saying so is more useful than citing the paper it came from.

Reranking changed no answer on this set while costing 3.2 seconds per query.

Memory accuracy

The belief store's claims, measured by replaying 12 scripted observations over 6 simulated days (node eval/memory-eval.mjs):

| Claim | Result | | --- | --- | | A repeated genuine preference becomes durable | 3/3 held | | A one-off speech mangling never does | 0/2 admitted | | A changed fact replaces the old value | VS Code durable, Sublime archived | | Confidence bounded and reported | 83% after 3 observations | | Provenance retained | 3 records, sources voice and text |

This exercises the state machine — corroboration, decay, competition, revision. It does not measure how well a 4B model distils facts from real conversation, nor whether durable beliefs improve the final answer. Both need labelled real data, and neither is claimed here.

Measured performance

Inverted index against the previous implementation, top-10 rankings verified bit-identical at every size:

| Corpus | Before | After | Speedup | | --- | --- | --- | --- | | 100 chunks | 1.87 ms | 0.008 ms | 223x | | 500 chunks | 8.71 ms | 0.037 ms | 238x | | 2,000 chunks | 37.1 ms | 0.116 ms | 319x | | 5,000 chunks | 104.8 ms | 0.456 ms | 230x |

Late sentence selection, measured end to end on real document text:

| Metric | Result | | --- | --- | | Context size reduction | 81 percent, 11,396 to 2,192 characters | | Correct evidence position | ranks 1 to 3 | | Determinism across repeated calls | byte-identical |

On reranking

Ollama exposes no /api/rerank endpoint, so a conventional cross-encoder is not available. Gemma 3 can rerank correctly, scoring 3 of 3 top-1 on labelled passages, but a single call costs roughly 3 seconds.

Reranking is therefore gated on ambiguity and is opt-in rather than default:

| Path | Frequency | Latency | | --- | --- | --- | | Gate skips, top-1 clearly dominant | ~50 percent | ~90 ms | | Gate fires, candidates close | ~50 percent | ~4,800 ms |

Typed input opts in. Voice input does not, because roughly 5 seconds of added silence is unacceptable on the spoken path. Any timeout or malformed response falls back to lexical order, so reranking is an enhancement and never a dependency.

Why not agentic retrieval

A-RAG style agentic retrieval was evaluated and deliberately not adopted. Benchmarked on this hardware, Gemma 3 routes queries to the correct source with 92 percent accuracy, which is sufficient. The blocker is latency: a single planning call costs about 3 seconds, and the published agent loops use 5 to 20 steps. That is 15 to 60 seconds of silence before the first word, which does not work for a voice interface.

Evaluation

node eval/retrieval-eval.mjs   # ranking accuracy across configurations
node eval/memory-eval.mjs      # belief store: corroboration, revision, garble rejection

Both harnesses drive the shipped modules. Results, method, and the caveats that bound them are in eval/RESULTS.md; the headline numbers are in Retrieval accuracy and Memory accuracy above.

The benchmark corpus is synthetic and labelled. It supports comparison between configurations, since each sees identical data; it does not predict accuracy on a real user's memory, and it is not presented as doing so.

What is still unmeasured, and should be: whether retrieved context and durable beliefs improve the final answer, as opposed to the ranking. That needs answer-level labels and a judge. The rankings are now measured; the answers are not, and no claim is made about them.

Android companion

companion/ contains a Kotlin application that mirrors the visualizer to a phone and exposes device control back to the desktop.

The visualizer is copied, not reimplemented

| Asset | Origin | State | | --- | --- | --- | | visualizerModes.js | src/js/visualizerModes.js | byte-identical, SHA-256 verified | | three.module.js | [email protected] | byte-identical | | Vertex and fragment shaders | src/index.html | verbatim | | visualizer.js | src/js/scripts.js | renderer, uniforms, and FFT blend preserved |

visualizerModes.js still carries import * as THREE from 'three'. Rather than edit the copy, an import map in the host page resolves the bare specifier, so the file stays identical to the desktop original.

Assets are served through WebViewAssetLoader on a virtual https origin rather than file://. WebView blocks ES module scripts from file:// because the origin is opaque, which presents as a silent black screen.

Audio bridge

The desktop fills window.jarvisFrequencyData from a WebAudio AnalyserNode. A WebView cannot obtain microphone access that way, so AudioFft.kt reads AudioRecord, applies a Hann window, runs a radix-2 FFT, and writes the same 64 bins natively. Bins use WebAudio's decibel mapping, minus 100 to minus 30 dB onto 0 to 255. Linear magnitude was tried and leaves the orb nearly static at speaking volume.

Pairing

Diagram: Pairing →

The phone always dials outward, which avoids Doze restrictions and handset address churn. Pairing retries every 10 seconds while unpaired, because the window is usually opened after discovery has already resolved.

/pair and /apk return 403 once the window closes. That window is the only thing standing between a network neighbour and the bridge token, so it is short and user-initiated.

Capability negotiation

On connect the phone reports what it can actually do, probed rather than assumed:

{"open_app":true,"list_apps":true,"clipboard":true,"battery":true,
 "tts":true,"flashlight":true,"volume":true,"alarms":true,
 "ui_automation":false,"screenshot":false,"read_screen":false,
 "silent_install":false}

The desktop reasons about the device instead of firing commands blindly. A request needing accessibility explains how to enable it rather than failing opaquely. alarms reports whether exact alarms are currently permitted — it is user-revocable on Android 12 and 13, and an assistant that keeps promising wake-ups it cannot schedule is the failure this project refuses everywhere else.

Control tiers

| Tier | Requires | Commands | | --- | --- | --- | | 1 | Nothing beyond install | ping, device_info, battery, clipboard_get, clipboard_set, tts, list_apps, open_app_by_name, flashlight, volume, capabilities, schedule_alarm, cancel_alarm, list_alarms | | 2 | AccessibilityService enabled | get_layout, click, long_press, swipe, input_text, global, screenshot, read_screen | | 3 | Wireless Debugging enabled | Desktop-side ADB: brightness, volume, keyevents, package management, file transfer, screenrecord |

Tier 3 runs entirely on the desktop through adbService.js. The APK is not involved. All ADB invocations pass argument arrays, never concatenated strings, and raw shell passthrough is disabled at the IPC boundary.

Alarms that actually wake you

The desktop's scheduler cannot, and says so. The companion schedules through AlarmManager.setAlarmClock() — the only API the platform treats as a real alarm clock, exempt from every battery-saver relaxation and special-cased by OEM battery layers. setExactAndAllowWhileIdle is rate-limited to roughly one firing per app per nine minutes in Doze; a wake-up that is "usually within a few minutes" is not a wake-up.

• Parsing stays on the desktop, so both surfaces agree on what "tomorrow morning" means. Only the scheduling happens on the phone.
• Re-armed on boot. AlarmManager forgets everything across a restart, and an alarm app that skips this works perfectly until the first reboot and then fails silently overnight. Xiaomi's QUICKBOOT_POWERON is handled alongside the AOSP broadcast.
• The ring screen shows over the lock screen, says what the alarm was for out loud, and offers a nine-minute snooze that stays one-shot even on a repeating alarm — copying the repeat rule would walk a 7am wake-up into the afternoon over a week of snoozing.
• Rings on STREAM_ALARM, which survives silent mode and Do Not Disturb, and raises the alarm volume off zero rather than playing nothing.

The floating orb

OverlayService keeps JARVIS on screen after th