TurboWarp-Time-Space-Sync

English | 日本語

Optical time correspondence and camera placement calibration for TurboWarp. This repository is based on turbowarp-extension-template v0.4.0.

Minimal demo

Open in TurboWarp Editor · SB3

Requires two physical screens/computers, a camera, HTTPS/permission, compatible exposure and matching pattern profile. Set Display refresh us in Stage code to the actual display interval (16667 is only a 60 Hz starting value). The decoder uses the extension minimum calibration duration. This sample explicitly enables only opticalTimeSyncV1 while constructing the embedded extension and restores host globals; package defaults remain OFF. Camera Source is embedded with revision/license/checksum. Tests cover idle, cancel, and stop only: flashing display, camera decoding/calibration, observation rate and time correspondence are unverified. No audio is used. Display consent is required every time; do not type SHOW for people who have not agreed. S/Stop hides the pattern and stops the decoder; Escape hides display.

Editable source and regeneration

What it does

Shows a time coded pattern on screen and decodes it back out of a camera, so a recorded frame can be placed against the moment the pattern was drawn. It also solves where cameras stand against a measured physical reference, and with the twtss.pattern.v2 profile the same projected pattern serves as that reference.

Everything is behind startup-fixed feature flags in config/feature-flags.ts, off by default: an extension loaded without them publishes no blocks. Set them before the project starts:

globalThis.__TWTSS_FEATURE_FLAGS__ = {opticalTimeSyncV1: true, placementSolveV1: true};

One projected pattern for time and placement

With use time pattern profile twtss.pattern.v2, the panel has four corner cells that are lit in every code. Each camera computer runs the decoder as usual, then measure pattern corners for [SECONDS] seconds: the corners found in the downscaled decoder frame only say where to look, and each outer corner is measured again in a small window of the full-resolution frame as the meeting point of the two outer edges of its corner cell, each edge located along its length at the halfway level and fitted as a line. Frames are combined by median; a spread above half a pixel is refused. The result is a twtss/placement-observation with points tl, tr, br, bl, named in the orientation the pattern is drawn. That naming is only proven while the camera is rolled less than 45 degrees and does not see the pattern mirrored; a turned view never decodes, and a mirrored one is refused because its readings do not advance with real time.

The operator measures the same four corners (the outer corners of the lit corner cells) on the wall once, and pattern reference JSON turns them into a twtss/placement-reference. Send each camera's observation and its camera model JSON to the computer solving the rig. Camera Source's own camera intrinsics JSON carries no distortion and is refused by set camera model.

What a measurement does and does not say

An optical reading constrains the sum of three unknowns: the offset between the two clocks, the camera's capture-to-timestamp delay, and the display's draw-to-photons delay. It cannot separate them. Results are therefore reported as displayToTimestampDelayUs with the folded-in components listed, never as a clock offset, and a time correction never implies that two cameras were exposed at the same instant.

A pattern code stays on screen for one display refresh, so the effective resolution is the refresh interval; the one millisecond step is only the quantisation of the displayed value. A reading also only decodes when the whole camera exposure falls inside one displayed code, which makes the decode rate roughly 1 - exposure / refresh. A low rate usually means the exposure is too long rather than the panel too dim, and the decoder says which.

If the camera's frame interval is a whole multiple of four pattern steps, the two lowest cells never change state, calibration learns no contrast for them and fails. Choose a calibration window whose frame interval is not such a multiple.

Photosensitivity

The pattern covers a large area and reverses many cells every refresh. Showing it requires an explicit acknowledgement from the operator, the default panel covers 35% of the shorter screen edge, and Escape removes it at any time. The guidance this is measured against depends on the viewer's distance from the screen, which nothing here knows, so these are mitigations and not a claim of compliance.

Planned implementation

See the Japanese implementation proposal for responsibilities, dependencies, acceptance criteria and rollback. The proposal is also copied below so this entrypoint records the intended work.

目的

時刻対応とカメラ配置の校正を提供するTurboWarp拡張として開発する。

実装予定

• 光学時刻パターンの生成・読取りと、共通時刻への変換（差・ドリフト・不確かさ）。
• スクリーン等の既知の実寸基準からカメラ配置を推定し、固定rigの相対姿勢を管理する。
• 同一PCの複数カメラと、WebRTCで接続された複数PCの観測を同じAPIで扱う。

パッケージ間の関係

• camera-source: 校正済み画像・撮影時刻。内部校正は本パッケージに重複実装しない。
• time-space-sync-app: QR搬送ペアリング、WebRTC接続、表示・実寸入力・観測収集の操作。既存realtime-motion-capture-appから共通処理を抽出する。
• AR / visual-tracking / photogrammetry / realtime-motion-capture: 同期・配置校正の利用側。

設計上の条件

時計差と表示・撮影遅延を区別する。時刻補正は同時露光を保証しない。投影歪み、測定寸法の精度、変換方向・軸・単位を記録する。

段階導入・受け入れ基準

1. 既存実装の所在・APIと校正形式を確認し、関連GitHub Issueでスコープを確定する。
2. 互換性を保った最小経路を実装する。新経路のフィーチャーフラグは既定OFFとし、導入時に設定場所を定義する。
3. 静止基準を撮影した2台の配置を独立した寸法測定で検証し、共通時刻の残差・再投影誤差を報告できる。
4. 単体検証に加えて実カメラによる統合検証を記録する。

ロールバック

抽出元の旧経路を移行中は保持し、フラグOFFで切り戻す。保存済み校正形式の互換読取りを保持する。

タスク管理

この文書はローカルの提案草案。実装着手前に本リポジトリのGitHub Issuesへ依存・DoD・チェックリスト・start/done/blockedログを記録する。Issueの作成・投稿は今回の初期配置には含まない。

Using the results from another extension

The results are plain JSON documents with a schema and a version. Their JSON Schemas are in schemas/, and fixtures/ holds one worked sample of each, produced by running the estimator and the solver rather than written by hand:

| Contract | Sample | |---|---| | twtss/optical-time-observation | fixtures/optical-time-observation-v1.json | | twtss/time-correspondence | fixtures/time-correspondence-v1.json | | twtss/placement-reference | fixtures/placement-reference-v1.json | | twtss/placement-observation | fixtures/placement-observation-v1.json | | twtss/placement-result | fixtures/placement-result-v1.json |

Check your reader against the samples rather than against a document you wrote to match your reader. Three things in them are easy to read past:

A delay is an interval. displayToTimestampDelayUs is the middle of delayLoUs..delayHiUs, not a measurement with noise around it. Carry the interval; a consumer that keeps only the midpoint has thrown away the part that says how much to trust it.

ippeErrorRatio is absent when there was no second solution. Absent is not a very large ratio — it says the question did not arise, because the target showed enough perspective to determine its pose. Present and near one is the case to refuse.

A pose direction cannot be validated. cameraFromReference maps reference points into the camera. Its inverse is also orthonormal, right-handed and affine, so no check you can write will catch it arriving the wrong way round; the sample is what pins the direction. toFromFrom in pairs is cameraB_from_reference * reference_from_cameraA.

The adapter belongs in the consuming repository. This package names no consumer, so that nothing it depends on can end up depending on it.

Planned architecture

flowchart TD
    Camera["camera-source: 画像・内部校正"] --> Sync["time-space-sync: 時刻・配置"]
    Camera --> Tracking["visual-tracking: 姿勢・疎な地図"]
    Sync --> Tracking
    Tracking --> AR["AR: セッション・アンカー"]
    Tracking --> Reconstruction["photogrammetry: 深度・形状"]
    Camera --> Reconstruction
    Sync --> Reconstruction
    Camera --> Motion["realtime-motion-capture: 身体姿勢"]
    Sync --> Motion
    AR --> Rendering["aframe: 描画"]
    Reconstruction --> Rendering
    Motion --> Rendering

Arrows indicate provider → consumer. This is a proposal; these integrations are not implemented in the scaffold.

Requirements and safety

Node.js >=24 and pnpm 11.28.8. The extension runs unsandboxed because it leases a camera through turbowarp-camera-source and draws a full screen overlay. Its only dependency is Camera Source, and only for the declarations describing a camera lease -- a handful of constants and one narrowing function, imported rather than copied so that a change on that side stops this build instead of surfacing in a browser. The corner measurement also reads Camera Source's calibration capability at runtime, when it is published, for the lens a camera's pixels were measured through. It does not bundle OpenCV.

Development

npm install --global corepack@latest
corepack enable
pnpm install --frozen-lockfile
pnpm run build
pnpm run typecheck
pnpm run lint
pnpm run test
pnpm run repo:check

Package identity: @kubohiroya/[email protected]. Bundle: dist/time-space-sync.js. Contract: dist/extension-manifest.json.

pnpm run check additionally checks generated files against Git.

Block reference

acknowledge that the time pattern flashes

Records that the operator was warned the full screen pattern flashes. The pattern will not be shown until this runs, and the acknowledgement lasts until the project stops.

| Property | Value | |---|---| | Type | Command | | Opcode | acknowledgePatternFlashing |

show time pattern

Covers the screen with the time coded pattern. The panel stays blank until the display refresh interval has been measured, so nothing decodable is shown from a guess.

| Property | Value | |---|---| | Type | Command | | Opcode | showTimePattern |

hide time pattern

Removes the time pattern overlay. Escape also removes it, and so does the page ceasing to be shown.

| Property | Value | |---|---| | Type | Command | | Opcode | hideTimePattern |

time pattern shown?

Reports whether the time pattern overlay is on screen.

| Property | Value | |---|---| | Type | Boolean | | Opcode | timePatternShown |

time pattern stable?

Reports whether the display refresh interval has been measured. While false the panel is blank and no camera can read a time from it.

| Property | Value | |---|---| | Type | Boolean | | Opcode | timePatternStable |

time pattern refresh us

Returns the measured display refresh interval in microseconds, or 0 before it has been measured.

| Property | Value | |---|---| | Type | Reporter | | Opcode | timePatternRefreshUs |

time pattern wrap us

Returns the period after which the encoded display time repeats, in microseconds.

| Property | Value | |---|---| | Type | Reporter | | Opcode | timePatternWrapUs |

time pattern profile id

Returns the identifier of the pattern profile in use. The decoder must be given the same profile or no reading will ever decode.

| Property | Value | |---|---| | Type | Reporter | | Opcode | timePatternProfileId |

use time pattern profile [PROFILE_ID]

Chooses the pattern the display draws and the decoder reads: twtss.pattern.v1 (the default) or twtss.pattern.v2, which keeps the panel's brightness constant and has lit corner cells that placement can be measured from. Run it before showing the pattern or starting the decoder; once either exists with another profile the choice is refused and nothing changes. The choice lasts until the project stops.

| Property | Value | |---|---| | Type | Command | | Opcode | setTimePatternProfile | | PROFILE_ID | String, default: twtss.pattern.v2 |

time pattern profile error

Returns why the last profile choice was refused -- unknown-pattern-profile or pattern-profile-in-use -- or an empty string when it was accepted.

| Property | Value | |---|---| | Type | Reporter | | Opcode | timePatternProfileError |

start optical time decoder for camera [CAMERA_ID] reference [REFERENCE_ID] calibrating [SECONDS] seconds at [REFRESH_US] us refresh

Leases the camera, locates the pattern, learns each cell light and dark level, and refuses when readings do not decode often enough. The display refresh interval must be supplied because it sets how long each code stays on screen.

| Property | Value | |---|---| | Type | Command | | Opcode | startOpticalTimeDecoder | | CAMERA_ID | String, default: default | | REFERENCE_ID | String, default: screen | | SECONDS | Number, default: 8 | | REFRESH_US | Number, default: 16667 |

calibrate optical time decoder for [SECONDS] seconds

Runs calibration again on a running decoder, for example after the camera or the display moved.

| Property | Value | |---|---| | Type | Command | | Opcode | calibrateOpticalTimeDecoder | | SECONDS | Number, default: 8 |

stop optical time decoder

Stops decoding and releases the camera lease.

| Property | Value | |---|---| | Type | Command | | Opcode | stopOpticalTimeDecoder |

optical time decoder state

Returns idle, acquiring-camera, calibrating, ready, or error.

| Property | Value | |---|---| | Type | Reporter | | Opcode | opticalTimeDecoderState |

optical time decoder error

Returns the last decoder error code, or an empty string when there is none.

| Property | Value | |---|---| | Type | Reporter | | Opcode | opticalTimeDecoderError |

optical time decode rate

Returns the share of recent camera frames the decoder could read, between 0 and 1. It is governed by the camera exposure: a reading only decodes when the whole exposure falls inside one displayed code.

| Property | Value | |---|---| | Type | Reporter | | Opcode | opticalTimeDecodeRate |

optical time decode margin

Returns how much room the weakest cell of the last reading had to spare, in luminance units. A panel drifting out of readability shows here as a falling margin before it starts failing.

| Property | Value | |---|---| | Type | Reporter | | Opcode | opticalTimeDecodeMargin |

optical time observation available?

Reports whether a decoded observation is waiting to be taken.

| Property | Value | |---|---| | Type | Boolean | | Opcode | opticalTimeObservationAvailable |

optical time observation count

Returns how many observations are currently held. Observations are kept for a fixed stretch of time rather than a fixed count, so the window does not change length with the decode rate.

| Property | Value | |---|---| | Type | Reporter | | Opcode | opticalTimeObservationCount |

optical time dropped count

Returns how many observations were discarded for leaving the retention window or exceeding the memory cap.

| Property | Value | |---|---| | Type | Reporter | | Opcode | opticalTimeDroppedCount |

optical time rejected count

Returns how many readings decoded but could not follow the previous one in real time. A frozen panel and a reading that slipped past the check bits both land here.

| Property | Value | |---|---| | Type | Reporter | | Opcode | opticalTimeRejectedCount |

take next optical time observation

Removes the oldest observation from the queue and exposes it to the observation reporter.

| Property | Value | |---|---| | Type | Command | | Opcode | takeOpticalTimeObservation |

latest optical time observation JSON

Returns the taken observation as twtss/optical-time-observation version 1 JSON, or an empty string before one is taken.

| Property | Value | |---|---| | Type | Reporter | | Opcode | latestOpticalTimeObservationJson |

optical time minimum calibration seconds

Returns the shortest calibration window in which every pattern cell is guaranteed to change state at least once.

| Property | Value | |---|---| | Type | Reporter | | Opcode | opticalTimeMinimumCalibrationSeconds |

estimate time correspondence

Intersects the observations collected so far into one delay. A reading bounds the delay rather than measuring it, so the result is the set of delays every reading allows; when a strict majority cannot agree on any, nothing is published.

| Property | Value | |---|---| | Type | Command | | Opcode | estimateTimeCorrespondence |

time correspondence JSON

Returns the last estimate as twtss/time-correspondence version 1 JSON, or an empty string when none has been made.

| Property | Value | |---|---| | Type | Reporter | | Opcode | timeCorrespondenceJson |

time correspondence error

Returns why the last estimate could not be made, or an empty string when it succeeded.

| Property | Value | |---|---| | Type | Reporter | | Opcode | timeCorrespondenceError |

display to timestamp delay us

Returns how much later a frame is stamped than the pattern it shows was drawn, in microseconds. This is not a clock offset: an optical reading constrains only the sum of the clock offset and the two pipeline delays, and the components folded in are listed in the JSON.

| Property | Value | |---|---| | Type | Reporter | | Opcode | displayToTimestampDelayUs |

time correspondence uncertainty us

Returns half the width of the delays the readings agree on. It shrinks as readings accumulate and never reaches zero, because one refresh of the display is never resolved.

| Property | Value | |---|---| | Type | Reporter | | Opcode | timeCorrespondenceUncertaintyUs |

time correspondence current?

Reports whether the last estimate still describes the present. Every estimate expires: one measured before the camera refocused or the clock was re-estimated is not a smaller measurement but a measurement of something else.

| Property | Value | |---|---| | Type | Boolean | | Opcode | timeCorrespondenceCurrent |

define reference [REFERENCE_JSON]

Stores a twtss/placement-reference version 1 document: the measured position of each point on the reference, with how well each was measured. Corners are listed individually because an image thrown obliquely onto a wall is a general quadrilateral, and describing it by a width and a height bakes a scale error into every pose solved from it.

| Property | Value | |---|---| | Type | Command | | Opcode | defineReference | | REFERENCE_JSON | String, default: {} |

add placement observation [OBSERVATION_JSON]

Stores where one camera saw the reference points, in unmirrored source pixels. Marks are matched to the reference by name; a mark whose name is unknown is dropped rather than matched by position.

| Property | Value | |---|---| | Type | Command | | Opcode | addPlacementObservation | | OBSERVATION_JSON | String, default: {} |

set camera model for [CAMERA_ID] to [MODEL_JSON]

Supplies the intrinsics and distortion to interpret one camera pixels with, as {intrinsics, distortion} with optional intrinsicProfileId, imageWidth and imageHeight, which a solve checks each observation against. Build it with camera model JSON on the camera computer rather than scaling a calibration profile yourself: only Camera Source can tell a scaled capture from a cropped one.

| Property | Value | |---|---| | Type | Command | | Opcode | setCameraModel | | CAMERA_ID | String, default: default | | MODEL_JSON | String, default: {} |

camera model JSON for [CAMERA_ID]

Returns the camera model to pass to set camera model: the intrinsics Camera Source adapted to the current frame, joined with the distortion, profile id and frame size from the calibration profile it holds. Camera Source's own camera intrinsics JSON leaves the distortion out and is refused by set camera model. Empty when Camera Source publishes no calibration, has no profile for the camera, or cannot adapt it to the frame.

| Property | Value | |---|---| | Type | Reporter | | Opcode | cameraModelJson | | CAMERA_ID | String, default: default |

measure pattern corners for [SECONDS] seconds

Measures the four outer corners of the projected time pattern in this camera's full-resolution image and builds a twtss/placement-observation from them. Needs the optical time decoder running on twtss.pattern.v2 and a calibration profile for the camera in Camera Source. Each frame's corners are found from the two outer edges of each lit corner cell and the frames are combined by median. The corners are named in the orientation the pattern is drawn, which is only proven while the camera is rolled less than 45 degrees and not seeing the pattern mirrored. A failed measurement clears the previous observation.

| Property | Value | |---|---| | Type | Command | | Opcode | measurePatternCorners | | SECONDS | Number, default: 3 |

pattern corner observation JSON

Returns the last corner measurement as twtss/placement-observation version 1 JSON, with points tl, tr, br and bl in unmirrored source pixels as observed, or an empty string when there is none.

| Property | Value | |---|---| | Type | Reporter | | Opcode | patternCornerObservationJson |

pattern corner error

Returns why the last corner measurement failed -- for example decoder-not-running, profile-without-fiducials, intrinsic-profile-missing, too-few-corner-frames, corners-unstable or orientation-unproven -- or an empty string when it succeeded.

| Property | Value | |---|---| | Type | Reporter | | Opcode | patternCornerError |

pattern corner spread px

Returns how far the per-frame corners scattered around the measured ones, as an RMS distance in source pixels. Reported for a measurement refused as corners-unstable as well, so the operator can see by how much.

| Property | Value | |---|---| | Type | Reporter | | Opcode | patternCornerSpreadPx |

pattern reference JSON [REFERENCE_ID] corners [CORNERS] sigma [SIGMA_METERS] measured by [MEASURED_BY]

Builds the twtss/placement-reference for the projected pattern from the outer corners of its lit corner cells, measured on the wall: tlX,tlY;trX,trY;brX,brY;blX,blY in metres on the wall plane, with any right-angled axes. The rectangularity residual is the largest distance from a corner to the best-fitting rectangle. Returns an empty string when a value is not a number, the corners do not trace a convex outline in that order, or measured by is not tape, laser or nominal.

| Property | Value | |---|---| | Type | Reporter | | Opcode | patternReferenceJson | | REFERENCE_ID | String, default: wall-projection | | CORNERS | String, default: 0,0;0.7,0;0.7,0.7;0,0.7 | | SIGMA_METERS | Number, default: 0.002 | | MEASURED_BY | String, default: tape |

solve placement for rig [RIG_ID]

Places every camera that observed the reference and works out where they stand relative to each other. A camera whose view fits two poses about equally well is refused rather than placed: a small reprojection error says the pose explains the image, not that the image chose it.

| Property | Value | |---|---| | Type | Command | | Opcode | solvePlacement | | RIG_ID | String, default: rig |

placement result JSON

Returns the last placement as twtss/placement-result version 1 JSON, or an empty string when none has been solved.

| Property | Value | |---|---| | Type | Reporter | | Opcode | placementResultJson |

placement error

Returns why the last placement could not be solved, or an empty string when it succeeded.

| Property | Value | |---|---| | Type | Reporter | | Opcode | placementError |

placement reprojection RMS for [CAMERA_ID]

Returns the reprojection error of one camera placement, in pixels.

| Property | Value | |---|---| | Type | Reporter | | Opcode | placementReprojectionRms | | CAMERA_ID | String, default: default |

clear placement observations

Forgets the stored reference, observations, camera models and result.

| Property | Value | |---|---| | Type | Command | | Opcode | clearPlacement |

License

MPL-2.0. See LICENSE.

Fixed input editing uses the optional editor API present at extension construction; late API replacement requires GUI reload and instance reconstruction. Dialogs and FixedInputs are startup flags, both off by default. Unchanged literals stay raw; reporters are never evaluated. Device/session prerequisites remain runtime checks. The canonical fixed field schema is schemas/block-editor-fixed-inputs.schema.json; docs/schemas/ is generated by scripts/generate-fixed-input-docs.mjs. The schema URI identifies the contract and is not fetched during editing.

The existing ./runtime TypeScript declarations require the host ambient TurboWarpRuntime interface. Strict consumer verification keeps skipLibCheck: false and supplies the unchanged host declaration; standalone use without that ambient remains an existing limitation.