@segmentstream/agent

The SegmentStream extension for a workspace's agent on eve. It gives the agent the workspace's ontology as tools, acting as the member who is talking to it, lets the workspace's workflows hand it tasks, and puts it behind the workspace's Telegram bot without a bot token in the deployment: Telegram calls go through the workspace's connection, unknown senders pair with a member first, voice messages become text, and group chats are answered only when the agent is addressed or asked for something it does.

The package is built against one eve version, 0.85.0, and the workspace's agent/ pins the same one. eve and the package move together.

Install

npm install @segmentstream/agent [email protected]

Deploying workspace source

The agent deploys with its workspace. Main pushes deploy by default; to accumulate source updates for an explicit release, commit .segmentstream/deployment.json with { "schemaVersion": 1, "trigger": "manual" }. Cloud still stores and indexes pushes, and segmentstream deploy explicitly deploys current main with the existing authorization and idempotency checks. This changes no agent schedules or tool permissions and adds no scheduler. Removing the file or selecting "push" restores automatic deployment.

For an existing hourly release process, the optional productionWindow setting in the same file restricts explicit production admission to configured UTC minutes. Outside that window, create and retry return deployment_window_closed with the next opening time; no attempt starts. This adds no scheduler or preview restriction.

Set up

The workspace's agent/ directory is an eve app in eve's documented layout, a package of its own like ontology/: the package files at the root, the authored files under agent/agent/:

agent/
├── package.json            # eve and @segmentstream/agent, both pinned; "build": "eve build"
├── segmentstream.json      # what the extension mounts, and what the platform reads at deploy
└── agent/                  # eve's agent directory
    ├── agent.ts            # model, reasoning, limits
    ├── instructions.md     # who the company is, how the agent behaves
    ├── sandbox.ts          # re-exports @segmentstream/agent/sandbox
    └── extensions/
        └── segmentstream.ts  # mounts this package with segmentstream.json

agent/agent/extensions/segmentstream.ts mounts the extension. The file name is the mount's namespace, so every contribution is named segmentstream__…:

import { segmentstream } from "@segmentstream/agent/extension";

import config from "../../segmentstream.json";

export default segmentstream(config);

agent/agent/agent.ts chooses the model once per session:

import { defineAgent } from "eve";
import { segmentstream } from "@segmentstream/agent";

export default defineAgent({
  defaultTools: false,
  model: segmentstream.model(),
  reasoning: "medium",
  limits: { sessionUsage: ({ window }) => (window.costUsd ?? 0) >= 2 },
});

segmentstream.model() resolves on session.started from models.chat in agent/segmentstream.json ({ "default", "byRole", "preview" }), where the platform reads it too; an inline segmentstream.model({ default, byRole, preview }) works the same. On a preview deployment it returns preview when set. Otherwise it returns the model of the first byRole entry whose role the member holds, in the order written, and default when none matches. The roles come from the member's pairing, not from the message. The model is never switched mid-session, which keeps the provider's prompt cache. A group shares one session, so its model follows the member who started it; each member's rights are still their own, because the ontology connection acts as the member of the current turn.

Both entry points export the same handle: @segmentstream/agent and @segmentstream/agent/extension.

agent/agent/sandbox.ts gives the agent its isolated sandbox. eve takes a sandbox only from the agent itself, never from an extension, so the agent re-exports the package's:

export { default, environment } from "@segmentstream/agent/sandbox";

See Sandbox.

segmentstream.json

{
  "channels": {
    "telegram": { "connection": "telegram", "policy": "smart" }
  }
}

| Field | Default | Meaning | | ---------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | channels.telegram.connection | none | The Connect key whose credential is the bot token. Without channels.telegram, no Telegram. | | channels.telegram.policy | "smart" | Classifies unaddressed group messages and captionless uploads; mention observes unaddressed messages without decoding or inference | | channels.telegram.linkPreviews | external | Suppress own-workspace previews and preview the first external link; opt into all for native previews or none to disable them. Caller options take precedence. | | channels.telegram.additionalOwnOrigins | [] | Additional exact HTTP(S) workspace origins, including local ports; no paths or credentials. | | channels.telegram.interimMessages | false | true delivers the text the model writes before calling tools as its own message, like a reply; see Telegram answer delivery | | models.stt | "spacexai/grok-stt" | The AI Gateway transcription model for voice messages; false leaves them untranscribed | | models.imageDescription | google/gemini-2.5-flash-lite | Image evidence for group routing even with native vision; false disables descriptions | | models.decision | "typesafe-ai/jev-1.14.0" | The AI Gateway decision model of the inbound router | | ontology.connection | true | Whether the agent reaches the ontology over MCP | | sandbox | false | true adds bash, read_file and write_file in the agent's sandbox without network; see Sandbox |

An unknown field or an invalid value fails the mount with SegmentStreamAgentConfigError, naming each path. manifestOf(config) returns the normalized configuration, defaults applied, for the platform's deployment manifest.

Channel admission and first contact

The platform and SDK provide admission, identity resolution, onboarding execution, notice tracking and durable limits. Workspace apps own channel policy, member grants, onboarding actions, notice content and lower limit values. No channel defaults to open, and no notice or limit reply text ships as a default. Existing invited-channel pairing and refusal replies remain unchanged.

Admission belongs in the ontology's signIn.channels; each unlisted provider stays invited, independently of email signIn.policy. Opening a channel is an explicit workspace opt-in. For an intentionally open provider, declare safe named member.permissions. Wildcards and generated create/update/delete grants are rejected. Shared named reads need review; move staff reads into a named role. Actions handling a member's own records enforce ownership with principal.id. Groups never register users. Closing a channel stops new registrations; block or deactivate an existing member to stop their access.

Configure app-owned text and lower limits in agent/segmentstream.json. These placeholders represent text authored by the workspace, not SDK defaults:

{
  "firstContact": {
    "version": "1",
    "notice": {
      "en": "<workspace-authored first-contact notice>",
      "ru": "<workspace-authored translated notice>"
    }
  },
  "limits": {
    "perSender": { "turnsPerMinute": 4, "turnsPerDay": 40 },
    "newUsersPerHour": 20,
    "workspaceTurnsPerDay": 2000
  },
  "replies": {
    "perSenderLimited": { "en": "<workspace-authored sender limit reply>" },
    "workspaceLimited": { "en": "<workspace-authored workspace limit reply>" },
    "registrationsLimited": { "en": "<workspace-authored registration limit reply>" }
  }
}

All three sections are optional. If present, firstContact requires a nonblank version of at most 200 characters, without surrounding whitespace, and a nonempty locale-to-text notice map. Text values must be nonblank and at most 4000 characters; locale keys must be valid language tags. Unknown settings, including the superseded publicUsers field, fail validation. The sender's language code chooses the exact locale, then its base language, then en, then the first entry. The same selection applies to configured replies. replies.limited is an optional fallback for any missing scope-specific limit reply. Without either reply, limits silently stop processing. groupPairingRequired and refused may override existing pairing/refusal copy.

Omitted limit fields use platform ceilings: 6 turns/minute and 80/day per sender, 30 new users/hour, and 3000 workspace turns/day. Overrides must be positive integers at or below each ceiling. The trusted runtime submits these effective limits; the ontology clamps and persists them for other admission paths. Only self-registered users with exactly the member role consume sender/workspace turn quotas, keyed by user id across identities. Paired-only users and holders of named roles remain exempt. Every registration shares the hourly new-user cap. These settings grant no access.

Admission runs after cheap edit/service gates, before decoding, then before jev and chat. Private messages with content and directly addressed group messages resolve with turn: true. Other group messages first resolve with turn: false, including replies to the bot that address another recipient. Mention-policy and other-recipient messages observe without counting, downloading or decoding. A recipient-only reply to a recent pending question remains eligible as answer data. Empty events remain uncounted even with a pending question. Eligible senders obtain counted admission before transcription, attachment work, smart-routing inference or pending-answer dispatch. Smart decisions consume a turn even if they choose observation, and reuse it if they reply. Each message requests counted admission at most once. No-op edits never resolve; caption-only edits reuse decoded evidence without downloading the same attachments again. Deferred-card presses use GET lookup and act-as authorization, never resolve or quotas: their initiating turn was already counted. Limited private inbound events and replies retain the member's archive grant through an uncounted binding lookup before the first persistence.

Resolver statuses are existing, created, pairing_required, refused (inactive/blocked) and limited. A limit reply requires the server's notify flag and an addressed message or chosen reply. Empty events get no turn or pairing reply. A configured due notice is sent only on a private eligible turn, before processing; resolve carries its version and selected-text SHA-256, and acknowledgement records the same version/hash after delivery. Version or selected-text changes request delivery again. Failures retain diagnostics and retry on the next private turn. Without a configured notice, the SDK sends and acknowledges nothing. Ontology check warns about missing notice/limits when agent config is supplied; its CLI reports when that check is skipped.

An app may declare signIn.onNewUser as an idempotent member-runnable action. The ontology executes it before returning an accepted eligible resolve, retries pending failures, and never delegates it to the model. Handlers receive read-only principal.identities[{provider, subject, username?, verified}], createdVia and authenticatedVia. The SDK forwards verified provider provenance with the member's act-as header. No identity facts contain credentials.

Release order

Deploy backend-next's strict validator before publishing agent SDK 0.8.0. Upgrade the workspace ontology to the v2 resolver/notice contract before its agent. A resolve-route HTTP 404 falls back to the unchanged GET lookup and invited pairing; other failures stop admission. Older agents continue using GET lookup. Starter templates and pins remain unchanged in this stage.

What the extension contributes

| Name | Slot | What it does | | ------------------------------------------------------------------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | segmentstream__ontology | connections | Direct MCP tools through Eve search, calls and approvals, acting as the current member or task. | | segmentstream__telegram | channels | POST /eve/v1/telegram: Telegram through the connection, with pairing, routing and voice. | | segmentstream__tasks | channels | POST /eve/segmentstream/v1/tasks: a workflow's agentTask, run as the workflow's user; see Workflow tasks. | | segmentstream__attachments_list, _read, _promote, _discard | tools | The member's temporary attachments in this chat: list, read, keep through an action, drop. | | segmentstream__attachments_send | tools | Sends readable stored ontology files into the current Telegram chat and topic. | | segmentstream__bash, _read_file, _write_file | tools | With "sandbox": true: eve's file tools, only in the sandbox @segmentstream/agent/sandbox opened. | | platform rules | instructions | Changes go through actions; describe indexes or inspects refs; discover searches metadata; tools are already narrowed; when to answer [SILENT]; how to handle attachments | | sandbox rules | instructions | With "sandbox": true: working copies, file contents as data, no network. | | sandbox cleanup | hooks | Deletes the session's sandbox when its history is cleared, such as by sessions.idleResetHours. |

Sandbox

@segmentstream/agent/sandbox exports environment, a Vercel Sandbox environment, and a default selector. eve build prepares the environment's snapshot with a fixed toolset: poppler-utils, qpdf, tesseract (eng, rus), ghostscript, ImageMagick, 7zip, unar, python3 with pandas, openpyxl, pypdf and pdfminer from Ubuntu 26.04, and pdfplumber 0.11.10, Pillow 12.2.0 and pypdfium2 5.9.0 pinned by hash. eve keys snapshots on the authored sandbox.ts and the environment's options, not on a packaged prepare, so the segmentstreamToolset tag carries SANDBOX_TOOLSET_VERSION, a digest of the toolset; a changed toolset is a new snapshot. The environment has no network policy: it would also apply while the toolset installs.

The selector opens each session's sandbox with networkPolicy: "deny-all" and 2 vCPUs, writes /workspace/.segmentstream/sandbox.json (session id, opening time, toolset) and records the opening in session state. The file tools pass a gate first: the marker must name the current session, or the call fails before eve's executor runs, so an agent without sandbox.ts never gets a shell in eve's default sandbox, whose network is open. The first call of a turn replaces a sandbox older than 24 hours, or one without a marker, through its handle, carrying over the attachments eve staged in the last 24 hours. eve keeps a session's sandbox across deployments, so a session that opened one before the agent adopted @segmentstream/agent/sandbox gets the SDK's on its first file-tool call; a reopened sandbox without the marker fails the call, and another session's marker fails it without deleting anything. The cleanup hook deletes a sandbox only when the session recorded opening one, so it never opens a sandbox to delete it; compaction.completed is ignored, because eve also compacts mid-turn. eve's session-end cleanup deletes whatever remains.

segmentstream__attachments_promote takes path, a file inside /workspace, instead of id while the sandbox is on. The runtime reads it through the gate, refusing a real path outside /workspace, anything but a non-empty regular file and more than 20 MB, stores it through the attachment route with the source sandbox and the reference sandbox:<session>:<sha256>, and promotes it like any attachment. The ontology accepts that source from Ontology SDK 0.53.0.

For a script-generated JSON payload, use segmentstream__ontology_call_file with the bare tool name and arguments_file (a JSON object inside /workspace, at most 1 MiB). It snapshots the payload before approval and dispatches that snapshot once, preserving exact computed values. Eve has no public connection API for reading arguments from a sandbox file. This narrow bridge has no discovery or inline-call API; ordinary calls use Eve. Chat posting is excluded. File-taking actions remain reachable through segmentstream__attachments_promote.

Set save_result: true on that file call to save a permitted read-only tool's exact JSON response in the same session's verified sandbox. The response gives a generated result_file, byte size, SHA-256 and query/paging metadata, without copying rows through the model. This mode refuses write tools before dispatch and checks their read-only annotation again at execution. Files are limited to 8 MiB; the query's existing permissions and bounds still apply. A file's row count covers that result only: follow continuations or split truncated aggregate groups into disjoint filters before combining and deduplicating them with a script. The file is temporary, like the sandbox itself.

Under eve dev the sandbox is a Vercel Sandbox too and needs local Vercel credentials; without them the sandbox tools, and turns carrying files, fail.

The ontology

Agent SDK 0.5.3 uses progressive discovery with ontology SDK 0.27.0: a compact entity index or metadata search, followed by complete inspection of relevant refs. Definitions already in context can be reused for the same ontology hash. Older ontologies without discover use their legacy describe catalog; operation authorization is still checked on every call.

Agent SDK 0.5.0 adds outbound attachments. It needs ontology SDK 0.21.0 for file grants and document requests; requests without documents from older ontologies remain compatible.

The connection calls {origin}/mcp with the deployment's Vercel OIDC token as the bearer and the member's id in X-SegmentStream-Act-As. The ontology service verifies the token, refuses a member who is inactive or has no live pairing, lists only the tools the member may run and records the member in history. A turn whose principal carries no memberId, such as an unpaired caller, gets no ontology connection at all. Each member gets their own connection instance, so members who share a group chat's session never share a client.

A task session sends the task's secret in X-SegmentStream-Agent-Task instead of X-SegmentStream-Act-As, never both, on /mcp and on the metadata reads of its approval checks. The ontology resolves the workflow's user from the secret, so the session sees and runs exactly what that user may. Its instance key is {origin}/mcp#task:<task id>, so a task never shares a client with a member, and it negotiates neither chat posting nor the chat's runtime source.

Workflow tasks

A workflow hands the workspace's agent a prompt with agentTask from @segmentstream-cloud/ontology/workflows and waits for its answer (the specification). The task runs in an ordinary session of the same agent: its instructions, model choice from the principal's roles, tools, guides and sandbox. The SDK adds no text to the prompt.

• Start. The ontology calls POST /eve/segmentstream/v1/tasks with { "task_id", "secret" } and the deployment's runtime token. The route admits only functions of the same Vercel project in production, as the health route does; elsewhere it answers 404. It reads GET /api/ontology/agent/tasks/current with the secret and answers 409 when the task is not live or is another task. Otherwise it ends any earlier session at task:<task id>, starts the session with the prompt as its first message and answers 202 once the session accepted it.
• Principal. segmentstream-agent-task, a service principal with the workflow user's id and taskId, workflow, roles and name, all from the ontology's answer, never from the request.
• The secret. The route hands the secret to the session in the channel's adapter state; the session's first event moves it into the session's durable task state, and it is forgotten once the outcome is reported. It is never in the principal, the model's context, tool arguments or results, the session's events, logs or the conversation archive.
• No person is present. Ontology actions the workflow's role permits run without confirmation cards. Attachments, member files, chat history and chat_post_message are not offered. A question from questions_ask, an approval a tool asks a person for, eve's own questions or a sign-in end the task as needs_input with the question's text; the per-session cost cap ends it as failed.
• Report. When the turn ends, the channel posts the outcome to POST /api/ontology/agent/tasks/current/result: completed with the turn's reply text, needs_input, or failed with agent_failed. Text longer than 65,536 characters is cut and marked truncated. A 409 means the task was already reported. Network failures, 429 and 5xx retry with backoff for up to two minutes and never past the task's deadline. After the report, or when the ontology refuses the secret, the session ends: eve has no command for a session to retire itself, so it asks its own deployment's POST /eve/segmentstream/v1/tasks/end, which ends only the session that still owns the task's address.
• Logs. The conversation archive keeps a task's events under provider task and the task's id, with no member taking part, so segmentstream agent logs --chat <task id> shows them.

Telegram

• Credentials. eve's Telegram channel puts the bot token in the URL path. The adapter gives it a placeholder token and an origin that does not resolve, and its fetch sends each method call to the connection's bot API and each download to its file API through SegmentStream's connection proxy. The token stays in SegmentStream.
• Webhook. A webhook is accepted only with X-Telegram-Bot-Api-Secret-Token equal to SEGMENTSTREAM_AGENT_WEBHOOK_SECRET, compared in constant time. The platform generates the secret and registers the webhook after each production deployment. A workspace-supplied menuButton.text enables the app menu button; omitting it resets the provider menu to its default for Agent SDK 0.13 or later. Pre-0.13 archives without menu copy retain their existing provider menu.
• Production only. Outside VERCEL_ENV=production the route answers 200 and does nothing, so a preview never answers a real chat. Without channels.telegram it answers 404.
• Pairing. Bots and channel posts are archived without starting turns. The sender's Telegram account is checked on every message, so revocation takes effect immediately. Unlinked group speech and attachment metadata remain raw context, including edits; no unlinked voice is decoded or gets a reply. An unknown private sender gets the workspace pairing prompt and button to /agent/connect?code=…, valid for ten minutes; revoked or inactive private senders receive configured refusal copy, or no message when it is absent.
• Who the agent is. A paired sender's turn runs with the principal id telegram:<user> (telegram:<chat>:<user> in groups) and the attributes memberId, memberName, roles, chatId, chatType and username.
• Routing. Private chats and messages explicitly addressed to the bot retain direct admission. Other group messages, including captionless attachments, go through one decide question with the agent's purpose, recent conversation, open questions and confirmation state. The legacy mention policy observes unaddressed messages without paid work. Attachments never force a turn. Human reply edges and other recipients are untrusted context for inference. Failed decisions observe. Unanswered messages join the bounded backlog; answered messages remain in recent shared context without rejoining that backlog on retries. Decision events retain the reason, short audit note, model and latency, plus probabilities when available.
• Conversation addressing. Opt in with groups.smart.engagement.policy: "conversation". Smart replies to an explicit invitation, a live back-and-forth, or context clearly addressed to the agent. Human addressees end the conversation; uncertainty observes. The router uses three addressing/response/closing questions, excludes workspace agent instructions, and applies per-path probability thresholds. Pending questions are hints inside a live conversation, never a bypass or another session. Untagged bursts wait 6 seconds (maximum 15); same-caller messages steer one chat/topic session and other callers are rechecked when dequeued. A durable burst fence prevents a second accepted turn. Claimed mentions and commands survive later human-directed messages in the burst, recorded as dequeue_explicit_kept; archive failures and duplicate claims still cancel. Duplicate and observe cancellations remain silent at Telegram delivery, including callback replay. Other turn cancellations use texts.notice.cancelled with a symbol fallback; an unconfigured failure notice never consists only of a reference. Diagnostics remain in operator logs. See group configuration.
• Voice. Voice messages, audio files and video notes are downloaded through the connection and transcribed with models.stt through AI Gateway before the turn. The transcript becomes the message, with a note such as [voice message, 12 s] in context. stt: false skips transcription. Transcription accepts at most 20 MiB and ten minutes of declared duration, with a 60-second inference timeout. A skipped or failed transcription reaches the agent as a note. Originals are stored as ontology attachments. Eve 0.85.0 only hydrates inbound images and PDFs; it does not deliver native audio bytes to the chat model. Even an audio-capable chat model therefore uses the configured STT fallback, with an explicit host limitation note. Native audio requires a compatible Eve host change.
• Attachments. Photos, documents and other Telegram files up to 20 MB stream through the connection proxy (its framed responses are not bound by the 2 MiB buffered limit) and are stored, before the turn, as the member's temporary attachments in the ontology's file storage (POST /api/ontology/agent/attachments), for this chat, for 24 hours. Uploads start a turn after participant registration; attachment tools can also list the stored files after a cold start. Eve supplies native file parts for photos and documents; other media reaches the turn through stored attachment IDs in context. The turn's file parts point at the stored copies, which eve reads through the channel's fetchFile as the turn's session, which then owns them, and an <attachments> block names each attachment's id. A file that is too large or could not be stored becomes a note. The segmentstream__attachments_* tools derive the member, chat and session from the turn, never from their arguments; they are extension tools, so defaultTools: false keeps them. attachments_promote runs an ontology action that takes a file, such as document.upload, on the stored file as the member, and answers an earlier promotion of the same SHA-256 instead of storing a duplicate. A group's history is shared, so private files belong in a direct message. attachments_read detects formats from MIME, extension and signatures: spreadsheets become paged sheet CSV, DOCX/PPTX become text, ZIP becomes an entry listing, EML becomes headers/body/attachment names, and other files return metadata. Continue with pagination.next_offset (and next_character_offset for a partial row). attachments_table parses the entire CSV/TSV/XLSX/XLS/ODS for typed dates and decimals, filters, grouping and exact sums/counts/extrema/first/last by date. Decimal results are strings. Use it for totals and balances rather than a read preview; there is no need to split files.
• Chat registry. Every group message reports its chat to the ontology's agent_chat registry (POST /api/ontology/agent/chats/seen) with its title and, when the sender is paired, the member's id: at most once per chat every ten minutes per process, and at once for a member not yet reported there. The report runs in the background; a failure is logged and changes nothing, so the next message reports again. An answered group turn waits for its report, bounded by the call's ten-second timeout, and carries the registry's title, purpose and whenToUse as a <chat> block. When a paired member leaves the group, or is removed, the agent reports it with leftMemberId, so the registry unlinks them; it starts no turn. A later message from them links them again. When the agent itself leaves or is removed, it reports agentLeft: true, so the registry unlinks every member and keeps the chat; members link again when they write after the agent is back. When a group becomes a supergroup, either notice (migrate_to_chat_id in the old chat, migrate_from_chat_id in the new one) reports the new id with migratedFrom: <old id>, starts no turn and drops the old chat from the memo. The <chat> block carries at most 1000 characters of whenToUse.
• Cross-chat drafts. The agent uses chat_post_message through its existing ontology connection. The exact full text, destination and authenticated attribution appear on a deterministic Confirm / Edit / Cancel card in the requesting group or DM. Only the requester can use it; the card expires after ten minutes. Edit cancels the original call and asks for a revision. Oversize previews are refused, and uncertain delivery is never automatically retried. Sent receipts retain the preview and show a Telegram link when available; archive failure is distinct from send failure. Agent SDK 0.6.0 prepares this flow; delivery becomes available when the ontology advertises the protocol below. The old instruction receiver /eve/v1/telegram/messages is removed. Telegram routes now serve only the inbound webhook and authenticated health; old instruction relay calls fail instead of starting an agent turn.
• Silence. When the model answers a routed message with exactly [SILENT] (or NO_REPLY), in any case, nothing is sent to the chat. Empty and whitespace-only final text also send nothing. The turn stays in the session. Archive decisions distinguish silence markers, empty text and whitespace, retaining finish reason and character count without logging raw output. Model reply decisions do not prove transport delivery; completed outbound events carry that evidence.

The observed ring and chat-report caches live in process memory. On a cold start, the adapter restores recent conversation context from the platform archive and reports its chats afresh. The archive survives independently of eve session state and these caches.

Text and captions retain their Telegram entity annotations for addressing. Rich-message blocks are projected to plain text before capture and dispatch, bounded to 16,384 UTF-16 code units, 2048 traversal steps and 24 nested levels. The archive records content shape and truncation without retaining raw webhook payloads. Edits remain separate audit events, but replace the earlier message in observed/restored model context. An edit alone never starts an agent turn. Entity offsets cannot reconstruct a missing body; unsupported empty content is recorded with shape diagnostics.

Image context and capabilities

Before handling media, the adapter reads the selected model's declared input modalities from AI Gateway's public model metadata. Existing sessions use the model recorded in their latest available step.started event, rather than reselecting by the current speaker's roles. A new session uses models.chat with the normal role and preview rules. An inline model choice in agent.ts cannot be inspected before its first session step; capabilities then stay unknown.

For group messages reaching the classifier, models.imageDescription produces short image text even if the main model sees native images. Native image input remains available to the main model when native ingestion is selected (the omitted/default setting). Explicit image-model overrides retain their existing fallback ingestion behavior. false disables image descriptions. On text-only models the same setting supplies fallback image context; unknown capabilities remain explicit.

Routing decodes at most one file (5 MiB/15 seconds): images yield up to 1,000 characters; PDFs use plain extraction in a terminable worker for the first two pages and 2,000 characters. No-text PDFs report no_extractable_text; other documents supply metadata. Group voice/audio/ video-note transcripts use models.stt (60 seconds/5 MiB/15 seconds, 1,000 characters). Derived evidence is untrusted and retained with the message under argsSummary.routing.decoded, including kind, model, status and voice duration. It survives cold reconstruction and is visible in archive event output. An attachment alone never starts a turn; chat purpose and open asks remain classifier context. Routing records provider choice probabilities when available, model and latency; missing probabilities are omitted.

Fallback descriptions outside classifier routing cover JPEG, PNG, WebP and GIF, at most one per Telegram message, 5 MiB of input, 256 output tokens, 512 retained characters and a 15-second inference timeout. Albums are multiple Telegram messages and can incur multiple calls. These ingestion calls are separate from the chat model's session spending limit. The descriptor treats image text as untrusted evidence. Failure, unsupported format and limit notes retain the original attachment reference. Metadata lookups have a five-second timeout, a five-minute success cache and a 30-second failure cache, with at most 128 model entries. A lookup failure never asserts that the model saw pixels.

Eve's native image input limit is 3 MiB per image. On the next reply, the adapter also appends eligible observed images as file parts, up to five images and 3 MiB total after accounting for current images. It creates attachment URLs for the current requester, and storage checks that member's access again. Expired or omitted images get notes. A text-only reply receives generated descriptions and references, never hidden image bytes. Other formats remain attachment-tool inputs.

The process-local ring retains 30 observations per chat/topic, with separate budgets of 1000 UTF-16 code units of text and 2000 of media notes per observation, plus at most five image references. The recent routing window also retains 30 entries. A process retains at most 500 chats, evicting the chat least recently written. Reading does not refresh its position. A reply drains the observation backlog; an idle reset clears both windows; daily compaction leaves the ring intact. Image bytes are not kept in this cache.

Archive restoration can recover raw text/captions after restarts and compaction: the webhook warms an empty ring from up to 30 inbound events, and each turn refreshes a separate bounded archive transcript. The latter reads at most 100 events, 4000 text characters and ten raw attachment references per event, within 30,000 serialized characters overall. Generated transcripts/descriptions and native-image ring references are currently not archived. They can be lost on restart or eviction before the next reply; the originals of stored images remain discoverable with attachment tools until expiry. Already dispatched native images follow Eve's sandbox/session history and may be omitted by compaction. Durable enriched media context and native audio are follow-up work, not guarantees of this adapter.

The webhook subscribes to message, edited_message, callback_query, my_chat_member and chat_member. A bot join reports the group immediately, including when the inviter is unpaired; membership updates do not start turns. Report tasks participate in the wrapper's tracked promises and Eve's waitUntil, and concurrent callers await a pending report.

Registry failures carry structured status, body code, request id and counters in operator logs. The first two consecutive failures per chat log warnings, the third logs one error, and further failures increment counters without repeating that error. A success clears the consecutive count and logs recovery. GET /eve/v1/telegram/health, authenticated with a production runtime token of this Vercel project, returns 503 when any chat reaches three failures, else 200. It exposes only process-local counters and safe codes, is uncached, and resets on a cold start. A 404 report indicates an older/missing ontology route; 401 indicates runtime authentication failed. Chat reports need the runtime token without an act-as member.

Send stored files

Call segmentstream__attachments_send with documentIds (1–10 document record UUIDs), an optional caption (at most 1024 characters), and optional positive replyToMessageId. For other entities use { entity: "invoice", id: "<record UUID>", property: "pdf" } in the same array. Omit property only when the entity has exactly one file property. These are ontology record ids, distinct from temporary attachment ids and blob revision ids. The member, chat and forum topic come from the authenticated turn. The tool is available only in production Telegram turns with a paired member and configured Telegram connection.

The tool gets a pinned file grant from GET /api/ontology/agent/files/:entity/:id[/:property] with the runtime token and X-SegmentStream-Act-As. The ontology checks the member's read permission before returning metadata, and checks live permissions again before granting download. Storage receives only its grant headers. JPEG/PNG/WebP use sendPhoto (Telegram limit 10 MiB); other files use sendDocument (50 MiB). Compatible files form albums of 2–10; mixed types and albums that exceed the proxy limit are sent sequentially. A caption appears once, on the first successful upload. Each result in { files: [...] } has id, filename, and either telegramMessageId or error with detail; filenames are empty when lookup was refused. Errors include not_found, forbidden, too_large and unsupported.

Effective upload limit: the connection proxy buffers multipart bodies inside its base64 JSON envelope and limits the decoded body to 2 MiB including multipart overhead. It does not stream uploads. Files whose multipart body exceeds this limit return too_large; the larger Telegram limits do not bypass it. Downloads are bounded even if storage understates size.

Exact chat posting accepts text only through the ontology's chat_post_message. There are no document_ids relay arguments, AgentMessage transport or <agent_request> turns. Stored file tools keep their member authorization in the current conversation; group purpose and document restrictions still apply.

Telegram answer delivery

Agent replies render Markdown as visible text and Telegram entities. Headings, emphasis, links, lists, quotations and code are supported; decorative separator lines are removed. The customer-text guard checks the rendered prose and link destinations. If rendering fails, delivery uses clean plain text. Only an explicit Telegram entity-formatting refusal permits a plain-text retry; timeouts and other uncertain outcomes must never trigger an automatic resend. Attachment captions use the same rendering and fallback rules.

Replies split at paragraph, line, sentence, word or UTF-16 boundaries within Telegram's 4096-unit limit. Entities are rebased or clipped on each part, and surrogate pairs stay intact. The whole reply or attachment batch occupies one shared FIFO slot per workspace/connection/chat, including all forum topics. A failed slot releases the next one. Sends require a positive Telegram message id; missing acknowledgements are errors and archive as unknown. Partial-delivery errors retain the message ids already acknowledged.

Only final replies are delivered by default: the text of a step that ends in tool calls (finishReason: "tool-calls") stays in the session. channels.telegram.interimMessages: true delivers it as an interim message through the same path as a reply (rendering, splitting, the queue slot, the reply anchor and the outbound record), never for empty, whitespace-only or silent text. A runtime remembers the interim texts of its recent turns, so a repeated completion or a final reply with the same text is not sent twice. Interim messages are not added to group routing memory, which, like its recovery from the session stream, holds only final replies.

This queue lives in one SDK runtime. It does not coordinate separate serverless instances or the ontology service's exact chat_post_message provider. Ordering all of those together needs a shared provider sequencer. Exact posting retains its approved literal text and durable receipt; this formatter does not rewrite approved posts or their confirmation cards. The removed agent_chat.send_message instruction relay must not be restored as a delivery fallback.

Local development

eve dev in agent/ runs the agent locally. The Telegram channel stays silent outside production. To reach a local ontology service, set SEGMENTSTREAM_AGENT_ONTOLOGY_URL=http://localhost:3000. vercel env pull writes a VERCEL_OIDC_TOKEN for the token. The ontology service admits only production tokens of the workspace's project, though, so a local run gets the ontology only from a local ontology service configured to accept that token.

Building the package needs Node.js 24, as eve does: pnpm run build runs eve extension build, which writes the agent-shaped dist/extension tree, its declarations and eve's compatibility manifest.

SDK-authored tool results and conversation archive events normalize runtime values to JSON before returning or encoding them. Exact decimal/bigint strings and binary byte views are preserved; conversion diagnostics identify the boundary and value path without logging payloads. Maps use entry arrays so distinct key types and object keys retain every entry. Native brand and binary getters avoid provider accessors; projection failures retain their redacted cause graph through the shared operator diagnostics reporter. This guard does not intercept Eve's own workflow streams or third-party tools.

Environment

The platform sets every variable on the workspace's Vercel project. None of them is a model key or a provider token.

| Variable | Used for | | ------------------------------------- | ------------------------------------------------------------------------------- | | VERCEL_PROJECT_PRODUCTION_URL | The app's origin: /mcp, /api/ontology/agent/* and the /agent/connect link | | SEGMENTSTREAM_AGENT_ONTOLOGY_URL | Overrides that origin in local development | | SEGMENTSTREAM_URL | backend-next, where the connection proxy answers | | SEGMENTSTREAM_WORKSPACE_ID | The workspace in the proxy's path | | SEGMENTSTREAM_ORGANIZATION_ID | The tenant the proxy selects | | SEGMENTSTREAM_AGENT_WEBHOOK_SECRET | The Telegram webhook secret; the only secret | | SEGMENTSTREAM_RUNTIME_OIDC_ISSUER | The team's OIDC issuer, which signs the tokens the health and task routes take | | SEGMENTSTREAM_RUNTIME_OIDC_AUDIENCE | The audience those tokens must name | | VERCEL_PROJECT_ID | The project those tokens must name | | VERCEL_ENV | Channels run only in production; preview selects the preview model | | VERCEL_OIDC_TOKEN | The runtime token outside a request; Vercel sends it on each request otherwise | | AI_GATEWAY_API_KEY | Optional. AI Gateway uses the OIDC token when it is unset |

License

Apache-2.0

Durable conversations

Production Telegram webhooks archive every receivable message in the platform database, including automated posts and unanswered messages. The extension also records replies, reply/silent decisions, tool names, redacted arguments/results/errors, and attachment references. It reloads stored context for the same chat/topic each turn, including across cold starts and compaction. File bytes are replaced by markers.

The Eve hook sanitizes tool arguments, results and failure messages in argsText, resultText and errorText; ingestion re-sanitizes them. UTF-8 caps are 16/32/4 KiB, with explicit JSON truncation markers. Content and argument summaries expire within 30 days of the original event timestamp (clamped to server time); replay cannot renew that window after archive deletion. Embedded JSON objects and arrays are sanitized recursively. Strings under text/content/body/csv/tsv and file/email body fields are omitted unless they parse as JSON objects or arrays. This includes all CSV/TSV documents, with or without headers. argsSummary keeps its pre-log semantics: the SDK applies bounded credential and file-byte redaction, and ingestion preserves the supplied summary. Deploy backend migration 0043/API before upgrading to agent SDK 0.6.10. segmentstream agent logs and its show/call commands read this content with a human Cloud session; all members with workspace access can read DMs too. Runtime/model/MCP conversation readers never receive these new fields. Telegram delivery diagnostics retain their correlation reference, operation and redacted error in the same archive before the channel callback returns, so a notice remains diagnosable after runtime logs expire.

Archive metadata and message text retain the workspace policy (365 days by default). Platform REST reads, the workspace ontology's chat_history and chat_search, and the member REST facade enforce workspace/group permissions and DM participant/admin scope. Deploy the backend migration/API and upgrade the ontology SDK along with this extension. See the repository's conversation storage guide for APIs, retention, eve's existing storage, and Telegram's delivery limits.

Exact chat posting protocol

An authenticated tools/list advertises _meta: { "segmentstream/chat-post": 1 } on chat_post_message. Without this marker, ordinary tools use the compatible connection and no runtime arguments are added. Posting with missing/incomplete preview metadata is denied. The versioned connection has its own instance key.

The runtime reads me for the server-derived attribution, and chat_list({ chat }) for exactly one eligible destination: { items: [{ id, title, provider, external_id }], next_cursor: null }. Provider is currently telegram. The final plain text is text + separator + attribution, with a blank line as the neutral separator; reply_to is optional. A preview digest is SHA-256 of the JSON array of chat ID, provider, external ID, title, attribution, text, final text and nullable reply target, in that order. The target/attribution are resolved again immediately before executing the frozen call.

The SDK constructs model-hidden _segmentstream metadata for approved runtime posts. It contains version: 1, session_id and call_id. A consumed, unexpired chat approval additionally supplies operation_id, source provider/chat/topic coordinates and expected: { chat_id, digest }. The operation ID is hq: plus SHA-256 of the JSON array [session_id, call_id]; it stays stable on replay and changes for a revision. The model cannot provide an idempotency key or this envelope. The ontology must authenticate runtime provenance, consume/remove the envelope before strict input validation for ordinary tools, derive the posting key from the operation, and recompute attribution. External OAuth/service callers still supply their own idempotency_key and cannot use this envelope. The coordinated ontology SDK implements this protocol; deploy the compatible ontology before using agent exact posting.

Source previews, accepted decisions and receipts use existing archive events with operation-linked toolCallId/event IDs. A cancelled Edit carries its pending revision question in the authoritative Eve resolution, so member/topic routing survives a cold start. No second provider sender lives here.

Release and migration

The original exact-post release used Agent SDK 0.17.13, Eve 0.76.3, Ontology SDK 0.56.0 and App SDK 0.25.2. Current builds require Eve 0.85.0 and a rebuilt Agent SDK extension. In the next release, feedback file guidance follows the visible tools. The compatible Cloud archive/API must be deployed first; migrations run only through its CI/CD gate. Publish the ontology SDK before the agent SDK. Upgrade the workspace ontology before its agent, with the declared Telegram provider mapping and explicit agent_chat:post_message grant. Supply the required workspace presentation before upgrading the agent to 0.13.x. Older ontologies keep ordinary tools working without the extra envelope. An older archive returning only {stored:true} cannot authorize a send.

Rollback disables posting while preserving archive claims and deduplication tombstones; it never restores relay or retries uncertain deliveries. See migration and verification.

Deleted session deployments

If Eve's session lookup raises RunKeyFetchError with HTTP 404 and Vercel's not_found / Deployment not found. response, the Telegram adapter records the original diagnostic and uses a fresh continuation scoped to the serving deployment. The smart gate still decides whether to reply; observing a message creates no session and sends no failure notice. Independent requests and cold starts select the same replacement. Healthy sessions continue across deployments as before. Once an address needs recovery, later deployments start it fresh; archived conversation context remains available, but the inaccessible session's model history and pending inputs cannot migrate.

Recovery applies only to session reads. Dispatch failures are never retried because the inbox may already have accepted them. Other provider errors propagate to operator diagnostics. A group receives a failure notice only after routing selected a reply or a workspace hook acknowledged the task. Notice wording comes from workspace texts.notice.

Failed model calls with an explicit budget-exhaustion reason select optional texts.notice.modelBudgetExceeded; known temporary provider failures (rate limits, timeouts and unavailable services) select texts.notice.modelUnavailable. Both use the usual locale fallback and then the existing failure notice (failed for turns, recovery for session failures) when unconfigured. Supply budget copy that explains that an administrator must restore the budget before retrying, and temporary-outage copy that asks the member to wait before retrying. Unknown model failures retain the ordinary notice; raw provider messages and hints never become user-facing text. Session recovery failures use the same classification.

The conversation recorder also writes a model_service_alert decision for recognized model service failures on every channel it can archive, including private HTTP sessions. This durable operator alert contains only the safe category, failure code, retry policy and ordinary event coordinates. A workspace-wide archive event key limits it to one alert per category per UTC day, including across chats, concurrent workers and replays. The first insert also emits [segmentstream] model service alert in runtime logs. Per-turn diagnostics remain available; alerting failures do not replace the original error. Read alerts with segmentstream agent logs --query model_service_alert --all. This does not submit member feedback or send an operator chat message. See the incident and verification procedure.

Opt-in attachment lifecycle

channels.telegram.attachmentLifecycle: { entity, action } selects a permitted, routine ontology action. There is no hook or collection delay by default. The workspace action owns grouping, collection deadlines, language, retention and completion; the SDK supplies no business policy or acknowledgement copy.

The action receives phase (admitted, stored, retained, ready, completed), a source object (chat, topic, message, sender, locale) from the authenticated Telegram request, and JSON data. Admission precedes downloads and history. data.attachments records every expected Telegram file ID and filename; stored and retained events record every permanent result or actionable error separately. Writes execute as the current member; original source provenance grants no access.

Return JSON {key, reply_to, acknowledgement, ready_at, retain, context} as needed: key identifies durable workspace state, reply_to fixes its source anchor, ready_at is an epoch-millisecond collection deadline, and context is the current durable inventory for the turn. Acknowledgement copy is workspace supplied; claim it durably before returning it so webhook retries do not repeat it. retain: {entity, action, input} requests immediate permanent storage of the current file. Both the hook and retention target must resolve to permitted routine actions. Record incomplete sources at admission and require all expected files before business completion. Persist bill/result receipts and resume by business identity after later uploads; a collection timer alone is insufficient.

The adapter retains the queued-turn policy, checks durable inbox acknowledgement, and anchors delivery to the immutable source message/topic. Every admitted text or intake webhook uses its host waitUntil lifetime to follow the durable session stream and refresh Telegram typing every four seconds; completion, human input, cancellation, failure or abort close the reader and timer. There is no fixed ten-minute cutoff; monitoring remains bounded by the host request lifetime. This is not a separate durable heartbeat workflow. Safe phase telemetry distinguishes admission, storage, queue acceptance and the owning turn without recording file contents or grants.

Before a final reply, an active lifecycle receives completed. Return {complete, message} after checking durable results. The adapter uses this workspace-owned message for both complete and incomplete outcomes; a failed completion check cannot send an unchecked model success claim. Questions retain the owning task and reply anchor across continuation turns.

Use attachments_read with entity, record id, and optional blob property to resume from a permanent file after temporary attachment expiry. The SDK obtains a fresh current-member files_download grant, streams only that grant's headers, bounds the transfer and verifies immutable size/SHA before parsing. The app bearer token and download grant never reach model output or diagnostics.

Returning a lifecycle key explicitly admits that source as a task, bypassing the ordinary routing classifier. Omit the key to retain ordinary routing. Collection honours the returned deadline within a 120-second processing budget; expiry fails closed without queueing an incomplete turn. Authenticated runtime health includes the built SDK version and deployment identity. Phase telemetry records webhook receipt, lifecycle/queue admission, model/tool events and typing API acceptance.

A configured hook failure, or a processing failure after it admits a task, returns HTTP 503 with a correlation reference before durable queue admission. Telegram can redeliver the same source; the hook and attachment reservation must preserve idempotent source membership, storage results and the single acknowledgement. Such a failure must not become an HTTP success with no queued work.

Native MCP tools

Every paired member reaches the ontology through the segmentstream__ontology MCP connection. Eve owns catalog loading, keyword ranking, schema presentation and direct tools/call execution through eve__search / eve__tool. There is no mode setting or gateway alternative. Stored entities use <entity>_find, <entity>_get and <entity>_history; reuse known names.

Upgrade the workspace ontology together with the agent: the connection requires the current runtime-envelope and exact chat-post contracts. Member authentication, provider provenance, trusted runtime arguments and exact chat-post confirmations remain in the adapter. The server authorizes each call against current permissions; connections are member-specific. Runtime catalogs omit optional output schemas. External MCP clients receive the same direct operations through standard cursor-paginated tools/list; the ontology search/describe/dispatch helpers are no longer exposed.

Consequential actions pause the Eve turn until the requester approves or cancels. Other members in a shared chat may wait for its outcome. Eve caches discovery within a connection's lifetime; the approval policy still reads fresh metadata independently. A cold connection downloads all input schemas. This change does not upgrade Eve or change the existing Telegram transport.

Remove workspace instructions for the four retired ontology gateway tools. Migrate script-generated payloads to the file-only segmentstream__ontology_call_file. Remove confirmations.mode from workspace configuration; legacy "deferred" is accepted and ignored so existing configs can upgrade. It cannot restore the old gateway.

Existing confirmation cards

Cards issued before the upgrade still use their original ledger, requester binding, execution fence and ten-minute expiry. Callbacks and the minute sweep remain to settle them; new ontology calls never create deferred cards. Keep their database tables while old cards or audit records remain. Legacy result events still correlate to the original call and never trigger a retry of an uncertain action.

With attachmentLifecycle.contextVersion: 2, lifecycle data includes source_at when Telegram provides its original message timestamp, so a workspace can group related uploads despite serial webhook delivery delays. A completion hook receives the model's reply alongside its intake key, allowing workspace-owned status copy to retain an actionable explanation. Reply delivery carries the owning turn's reply anchor explicitly after restore. Omitting the version preserves existing closed lifecycle data schemas; workspaces opt in only after their hook accepts the additional fields.

Runtime health reports native approvals and attachment-lifecycle presence behind its existing runtime authentication; source pins and public health alone are not adoption proof.

Telegram 0.13.0 requires workspace-authored pairing and confirmation text. See workspace Telegram text for keys, limits, neutral fallbacks and unsupported Eve buttons.

Workspace callback actions and structured Telegram input are separate opt-in mappings. They require the ontology runtime-source capability and dispatch only a currently permitted routine action as the verified member. The workspace owns callback tokens, presentation and business idempotency.

Durable typing

The webhook monitor ends with its host request. For turns that can outlive that request, an application's authored Workflow can call telegramTypingBlock from @segmentstream/agent/extension in a "use step" helper. Each activity runs for at most about thirty seconds, follows a bounded cursor through one session's durable stream, and sends only sendChatAction through the declared connection. It stops for its source turn's completion, failure, cancellation or input wait, or when the source session is terminal. Pass an exact turnId from a real turn.started hook, or a unique marker in a private acceptance message.

The application owns the Workflow directives: Eve does not permit them in an SDK extension. Its step reads getRun(source.sessionId).status through workflow/api and calls telegramTypingBlock(source, cursor, status). Its workflow carries the returned cursor between activities and ends when done is true. Start a continuation periodically to bound the workflow history, then retire the previous monitor. recordTelegramTypingStart(source, workflowRunId) records the monitor reference in the conversation archive. Source arguments contain transport coordinates and a member ID, never credentials or message contents. Repeated typing actions are idempotent; the activity never posts text.

Replication lifecycle and freshness

After verified initial publication, Cloud keeps a replication stream active while later changes are delivered. Active means its published generation is usable; it does not prove that the warehouse has every recent write. Use the reported source freshness watermark when reading aggregates. Initial copies and replay remain unavailable until their required verification and publication finish.