Social media plugin

Fetches and imports posts from several social media platforms through the installed yt-dlp CLI. Version 0.1.2 supports YouTube and TikTok. It is a self-describing plugin package (contrib.socialmedia); each platform owns one schema module, its migrations and its source/fetcher/importer exports.

Replacing the YouTube plugin

contrib.socialmedia replaces contrib.youtube (@semverse/plugin-youtube). The package declares semantic.package.replaces: ["contrib.youtube"], and installing it hands everything the YouTube plugin imported to this plugin:

• the YouTube activation is removed; its configuration, priority and auto-upgrade setting are inherited unless the install supplies its own;
• the YouTube plugin's service account is relinked to this plugin, so every semantic:owner / semantic:creator and grant keeps pointing at the plugin that now imports those items (if this plugin already had its own account, owned rows move to it and the old account is suspended);
• semantic:import:imported_by is rewritten, so Re-import uses this plugin;
• module youtube carries the YouTube package's migrations byte for byte. The host adopts the already applied ones instead of running them again, keeping the contrib:youtube:channel / contrib:youtube:video classes and all entity IDs, so re-imports update the same entities.

Afterwards contrib.youtube can no longer be installed. A dormant (uninstalled) YouTube plugin is absorbed the same way. This needs a Semantic server with package succession support (semantic.package.replaces).

Platforms

YouTube

Classes contrib:youtube:channel and contrib:youtube:video (subclasses of the shared social media account and post), plus the contrib:youtube:channel_import_options options class.

• Channel roots, @handles, /channel/ID, /browse/ID, /c/name, /user/name and legacy root vanity channel URLs are supported. Roots, featured/about/community/posts/playlists channel tabs select videos and Shorts in round robin order under one limit. /videos, /shorts, /streams select that tab alone. Standalone playlist/search URLs are rejected. Watch, youtu.be, Shorts, live and embed video URLs identify the same video; playlist and tracking query parameters never reach yt-dlp.
• IDs: platform youtube, channel youtube:channel:CHANNEL_ID, post youtube:video:VIDEO_ID, file youtube:video:VIDEO_ID:file.
• Default format: the best video excluding YouTube's AI-upscaled Super Resolution formats plus the best audio, merged without re-encoding (bv*[format_id!*=-sr]+ba/b[format_id!*=-sr]). If no matching format is available, the import fails instead of silently selecting an upscale. This preserves a YouTube-encoded rendition, not the uploader's original file. See yt-dlp's guidance.

TikTok

Classes contrib:tiktok:user and contrib:tiktok:video (module tiktok, migration 001_init), plus the contrib:tiktok:user_import_options options class with Also import all of the user's videos (import_videos) and Maximum videos (max_items).

• URLs: https://www.tiktok.com/@user/video/ID (also without www. or on m.), /embed/ID, /embed/v2/ID, /share/video/ID, short links (vm.tiktok.com, vt.tiktok.com, www.tiktok.com/t/CODE, resolved by yt-dlp) and profiles https://www.tiktok.com/@user. Photo/slideshow posts, lives, sounds, tags and collections are rejected.
• Users are keyed by secUid, which yt-dlp reports for profiles and videos and which survives handle renames: tiktok:user:SEC_UID. Their re-import source is https://www.tiktok.com/@SEC_UID. Videos are tiktok:video:ID, files tiktok:video:ID:file, the platform tiktok.
• Videos map title, message, published_at, view/like/comment counts, reposts as share_count, and save_count, music_track and music_artists.
• Default format b: TikTok serves progressive files and yt-dlp ranks watermarked and unplayable renditions last. TikTok usually reports exact sizes, so the download_signature is a single FORMAT:SIZE.

Install and build

Run the plugin on the Semantic server/node with Node 22+, a recent yt-dlp CLI, and ffmpeg for merging YouTube video and audio formats. Temporary disk space must hold a completed video and its intermediate streams.

npm install --global @semverse/[email protected]
semantic api plugin describe --stdio semantic-socialmedia
semantic api plugin install --stdio semantic-socialmedia
# Optional configuration:
semantic api plugin install --stdio semantic-socialmedia --config socialmedia-config.json

For a source build, run npm ci && npm run build in semantic-contrib and use --stdio node -- /absolute/path/to/dist/src/main.js. In Data → Plugins → Add plugin, enter the same launch, choose Describe, review the package and schema migrations, then Install. Executable paths, paths inside configuration and environment refer to the selected server. Use semantic api plugin upgrade contrib.socialmedia for an explicit upgrade and semantic api plugin uninstall contrib.socialmedia to remove the installation while keeping its package, entities and files.

Configure

All options are optional; paths refer to the server:

{
  "yt_dlp_path": "/absolute/path/to/yt-dlp",
  "max_items": 50,
  "cookies": "/absolute/path/to/cookies.txt"
}

yt_dlp_path defaults to yt-dlp on the server's PATH. cookies is a Netscape cookies file available for every platform. Set the server environment variable YT_DLP_COOKIES_FROM_BROWSER (for example, firefox or chrome:Default) to configure --cookies-from-browser. Each yt-dlp invocation first runs without cookies. If it exits unsuccessfully before emitting any records, the plugin retries once with the configured cookie file and/or browser cookies. Cancellation, launch failures, malformed output and partial streams are not retried. yt-dlp's exit codes do not reliably distinguish missing content from authentication failures, so either can trigger this fallback. Account URLs advertise their platform's options class; without import_videos (default off), account fetches and imports cover only the platform and account. max_items caps the selected posts (installation default if configured, otherwise unlimited). An explicit format overrides every platform's default and is passed as one argv value, never shell code. Requests may set only import_videos, max_items and format; executables and credential files are installation-only.

semantic api import candidates 'https://www.tiktok.com/@example'
semantic api import fetch 'https://www.youtube.com/watch?v=jNQXAC9IVRw'
semantic api import start 'https://www.tiktok.com/@user/video/ID' --plugin contrib.socialmedia --export tiktok_importer
semantic api import start 'https://www.youtube.com/@YouTube' --plugin contrib.socialmedia --export youtube_importer

Shared behavior

• Fetch: metadata only, no video downloads or temporary video files. Videos and Shorts produce a platform, an account and a post with empty contents. Channels produce their account, then full metadata for each selected video/Short.
• Skip contents: requests with skip_contents (Semantic's Save details only, semantic api import start --skip-contents) import metadata without downloads. Posts then omit contents, so a re-import never detaches a previously stored File. Semantic records the flag as import_skip_contents on each item and reuses it when the item is re-imported.
• Re-download avoidance: platforms publish no content hashes, and a merged video's size is only known after muxing. Each download therefore first runs a format probe (--skip-download --write-info-json) and records the selected streams' exact upstream sizes as download_signature (e.g. 137:1000+140:200). On re-import, the plugin reads the stored post and File through its own host access (semantic.db.get). If the File exists and the probed signature is unchanged, the download is skipped. Otherwise the download reuses the probe's info JSON (--load-info-json) and, if its SHA-256 equals the stored File's content_hash_sha256, the upload is skipped. Unknown stream sizes, lookup failures or missing files simply download as before.
• Import source: a completed video is read in 64 KiB chunks through SDK content events. The host persists the File; the wrapper post follows its file_end. Platforms and accounts are emitted before posts. No full video is buffered in memory. The host uploads files normally, then commits the hierarchy and file parent updates together in one ordinary database batch.
• Import fetched: advertised and implemented per platform: representation youtube.metadata.v1 / namespace youtube, and tiktok.metadata.v1 / tiktok. Pass the fetched entity stream to semantic.import/v1.Application.start_import_fetched, retaining the fetch's source identity. It re-extracts/downloads precisely those post URLs, refreshing metadata; it does not enumerate newly uploaded channel videos. Platform/account/post entities are accepted, with URL validation, duplicate checks and configured limits. This operation needs network access and is available through the streaming interface API; the current CLI has no import-fetched subcommand.
• Account enumeration uses lazy NDJSON and bounded per-listing limits. Missing YouTube tabs are skipped; other extraction/auth/network errors fail the operation. If the stream or final batch fails, no pending hierarchy changes are published. Completed file uploads (including replacements) remain; new uploads may be unattached. Active cancellation terminates/reaps yt-dlp, closes file readers and removes temporary files. Closing the stream runs the same cleanup. On POSIX, a supervisor owns the process group through cleanup and kills remaining descendants even if yt-dlp exits first, including after normal completion. Cancellation sends SIGTERM, allows a 750 ms grace period, then escalates to SIGKILL with at most 100 ms more for reaping before killing the remaining group. The supervisor signals its own group so a reused group ID cannot target unrelated processes. Descendants that deliberately start a separate session or process group escape this boundary. Windows cleanup kills only the direct process. Abrupt OS termination cannot run application cleanup.

Identities

The hierarchy is platform → account → post → File, using the parent attribute. Posts also reference their account and contain their File IDs in contents. The post/File references are cyclic and require atomic batch publication by the host.

Entities use explicit, readable IDs derived from upstream YouTube identifiers:

• Platform: youtube.
• Channel: youtube:channel:CHANNEL_ID.
• Post: youtube:video:VIDEO_ID.
• Video file: youtube:video:VIDEO_ID:file.

Watch/Short aliases, channel URLs, and fetched imports reuse these identities. Accounts, posts and Files name their own channel or watch URL as the item's re-import source, so re-importing a channel's video refreshes only that video. The import's source URL remains provenance; it does not determine the entity ID. These changes require a server supporting explicit import IDs and base migration 016_social_media_hierarchy. Existing records imported under the older URL-hashed IDs are retained; they are not automatically merged or deleted.

Schema IDs are semantic:base:social_media_platform, semantic:base:social_media_account and semantic:base:social_media_post. Posts use message, account, account_name, contents, published_at and available nonnegative counters. Counters are typed u64; timestamps are Semantic datetimes. In the current crates/data/src/entity_ref.rs, semantic:EntityRef is an untagged union of an ID string or embedded object. Thus contents: [fileIdString] is the defined ID representation; {id: ...} is not its tagged representation. account is a regular entity Ref. Video descriptions never become channel biographies.

Adding a platform

Implement Platform (src/core/platform.ts) in src/platforms/NAME/: URL identification, metadata mapping, entity keys and a schema module with its own migrations, then add it to src/platforms/index.ts. Downloads, signatures, skip-contents, account listings, fetched replay and exports are shared.

Tests and real smoke verification

npm run check
npm test
npm run fmt:check
# Optional real host test; creates a temporary, isolated unauthenticated local server:
SEMANTIC_BIN=/absolute/path/to/semantic npm run test:host

Tests use an executable deterministic fake yt-dlp (YouTube and TikTok modes) and cover schema mappings, video/Short/channel and TikTok video/user selection, limits, file streaming and references, fetched metadata replay, malformed/oversized output, process errors, cancellation, early stream close and temporary-file cleanup. test/package.test.ts checks that the adopted YouTube migrations equal the published @semverse/[email protected] package's. The host test uses the actual CLI and self-described package installation, fetches metadata, imports posts/accounts/Files, reads persisted file bytes, retries imports and checks failures.

The hierarchy is covered by authenticated Rust plugin-flow tests in semantic/crates/app/tests/import_hierarchy.rs, including cyclic references, ordinary file uploads, shared IDs, cancellation, and batch failure. The optional CLI host test above has been updated for this hierarchy but has not been rerun for these changes. Earlier versions passed it. A real yt-dlp 2026.10.21 run through the former YouTube plugin fetched @YouTube as a valid account without video bytes; lazy Shorts listing also succeeded. The public jNQXAC9IVRw video required YouTube bot-check login from this environment; BaW_jenozKc was unavailable. A successful real video download has not been verified here. Cookies and yt-dlp updates may be necessary for your network/account; errors are surfaced with yt-dlp's bounded stderr.

TikTok support was written against the yt-dlp 2026.10.26 TikTok, TikTokUser and TikTokVM extractor sources; a real TikTok fetch or download has not been verified here.