@cuylabs/agent-physical

Physical-world contracts for @cuylabs/agent-core.

agent-core owns the generic agent loop, tools, sessions, middleware, model calls, and tracing. agent-physical adds a small vocabulary for embodied systems: physical sessions, observations, capabilities, safety policies, execution traces, and tool adapters.

The package is intentionally backend-neutral. It defines shared contracts for physical sessions, while adapter packages connect those contracts to concrete robotics frameworks, simulators, middleware, and hardware stacks.

agent-physical is not the robotics runtime and it is not a replacement for a backend's native environment. It gives an external agent harness a common way to observe a physical session, inspect artifacts, request policy-code execution when supported, and apply safety metadata before side-effecting tools run.

Package Boundary

Use @cuylabs/agent-physical when you need:

• a common PhysicalSession interface for simulator and hardware backends
• typed observations, artifacts, execution traces, and task outcomes
• safety metadata that can be consumed by agent-core approvals and middleware
• generic status, observe, stop, artifact, and policy-code tools

Do not put backend-specific launchers, prompts, simulators, perception models, or robot APIs here. Those belong in adapter packages and runtime packages such as @cuylabs/agent-physical-capx and capx-agent-runtime.

Mental Model

A physical integration has three layers:

1. The agent harness decides what to do next.
2. The physical adapter translates shared session/tool calls into a concrete robotics backend.
3. The robotics backend owns the simulator, robot, sensors, task state, and low-level execution.

agent-physical is only the middle contract. It makes physical state and side-effecting actions legible to agent-core without forcing every robotics backend to expose the same native APIs.

Layering

agent-core
  -> agent-physical
    -> agent-physical-capx   (CaP-X Code-as-Policy robotics framework)
    -> agent-physical-ros2   (ROS 2 robot middleware)
    -> agent-physical-isaac  (NVIDIA Isaac Sim robot simulation)

agent-physical defines the shared shape. Adapter packages provide concrete connections to robotics frameworks, simulators, robot middleware, and hardware control stacks.

Quick Start

npm install @cuylabs/agent-core @cuylabs/agent-physical

import { createAgent } from "@cuylabs/agent-core";
import { createPhysicalSessionTools, type PhysicalSession } from "@cuylabs/agent-physical";

const session: PhysicalSession = await createYourPhysicalSession();

const agent = createAgent({
  model,
  tools: createPhysicalSessionTools(session),
  systemPrompt: "You are controlling a physical session. Observe before acting.",
});

For sessions that support direct code execution, the generated toolset includes physical_run_policy_code. For sessions that only expose observation/status or external trial control, that tool is omitted.

Backend adapters can add extra tools next to the generic set. For example, a CaP-X adapter may expose turn history while still using the same generic status, observe, artifact, stop, and policy-code contracts. Backend-specific helper libraries should normally appear as observation/context for generated policy code, not as extra model-facing robot tools.

Design Notes

• A physical capability is not just a normal tool. It may depend on calibrated sensors, timing, world state, hardware readiness, and explicit safety policy.
• Physical tool outputs include structured metadata so UIs and middleware can inspect observations, traces, artifacts, and outcomes.
• Direct physical execution is marked dangerous by default and uses manual replay semantics because repeated execution can move hardware or mutate a simulator.

Docs

• docs/README.md for the documentation map
• docs/architecture.md for package boundaries
• docs/safety.md for approval and replay expectations

License

Apache-2.0