xhoard

v1.0.2

Published

5 days ago

Archive Twitter/X bookmarks to markdown with link expansion, content extraction, and AI-powered categorization

0High
0Medium
0Low

tudorandrei

twitter x bookmarks archive markdown opencode twitter-archiver bookmark-manager obsidian

Xhoard

Archive your Twitter/X bookmarks (and/or optionally, likes) to markdown. Automatically.

Xhoard collects the valuable things you bookmark and like.

Multi-model support: Xhoard runs through OpenCode using the OpenCode SDK, so you can pick any model available in your OpenCode setup.

Quick Start (5 minutes)

Option 1: Using bunx (Recommended for CI/GitHub Actions)

No installation needed! Works great in GitHub Actions:

# Fetch bookmarks immediately (creates files in bookmarks/)
bunx xhoard fetch

# Run with environment variables
AUTH_TOKEN=xxx CT0=xxx bunx xhoard fetch

# Process with OpenCode
bunx xhoard run

# Full job with all options
AUTH_TOKEN=$AUTH_TOKEN CT0=$CT0 bunx xhoard run

Option 2: Local Installation

# 1. Install bird CLI (Twitter API wrapper)
# See https://github.com/steipete/bird for installation

# 2. Clone and install Xhoard
git clone https://github.com/TudorAndrei/xhoard
cd xhoard
npm install

# 3. Run the setup wizard
npx xhoard setup

# 4. Run the full job (fetch + process with OpenCode)
npx xhoard run

The setup wizard will:

Create required directories
Guide you through getting Twitter credentials
Create your config file

Manually Getting Twitter Credentials

Xhoard uses the bird CLI which needs your Twitter session cookies.

If you don't want to use the wizard to make it easy, you can manually put your session info into the config.

Copy the example config:

cp xhoard.config.example.json xhoard.config.json

Open Twitter/X in your browser
Open Developer Tools → Application → Cookies
Find and copy these values:
- auth_token
- ct0
Add them to your xhoard.config.json:

{
  "twitter": {
    "authToken": "your_auth_token_here",
    "ct0": "your_ct0_here"
  }
}

Note: xhoard.config.json is gitignored to prevent accidentally committing credentials. The example file is tracked instead.

What Xhoard Actually Does

Fetches bookmarks from Twitter/X using the bird CLI (can also fetch likes, or both)
Expands t.co links to reveal actual URLs
Extracts content from linked pages:
- GitHub repos (via API: stars, description, README)
- External articles (title, author, content)
- X/Twitter long-form articles (full content via bird CLI)
- Quote tweets and reply threads (full context)
Invokes OpenCode to analyze and categorize each tweet
Saves to markdown as one file per bookmark (organized by date folders)
Files to knowledge library - GitHub repos to knowledge/tools/, articles to knowledge/articles/

Running Manually

# Full job (fetch + process with OpenCode)
npx xhoard run

# Fetch from bookmarks (default)
npx xhoard fetch 20

# Fetch ALL bookmarks (paginated - requires bird CLI from git)
npx xhoard fetch --all
npx xhoard fetch --all --max-pages 5  # Limit to 5 pages

# Fetch from likes instead
npx xhoard fetch --source likes

# Fetch from both bookmarks AND likes
npx xhoard fetch --source both

# Process already-fetched tweets
npx xhoard process

# Force re-process (ignore duplicates)
npx xhoard process --force

# Check what's pending
node -e "console.log(require('./.state/pending-bookmarks.json').count)"

Fetching All Bookmarks

By default, Twitter's API returns ~50-70 bookmarks per request. To fetch more, use the --all flag which enables pagination:

npx xhoard fetch --all              # Fetch all (up to 10 pages)
npx xhoard fetch --all --max-pages 20  # Fetch up to 20 pages

Note: This requires bird CLI built from git (not the npm release). See Troubleshooting for installation instructions.

Cost warning: Processing large bookmark backlogs can consume significant model tokens. Each bookmark with content-heavy links (long articles, GitHub READMEs, etc.) adds to the context. Process in batches to control costs:

npx xhoard run --limit 50 -t    # Process 50 at a time with token tracking

Use the -t flag to monitor usage. See Token Usage Tracking for cost estimates by model.

Bookmark Folders

If you've organized your Twitter bookmarks into folders, Xhoard can preserve that organization as tags. Configure folder IDs mapped to tag names:

{
  "folders": {
    "1234567890": "ai-tools",
    "0987654321": "articles-to-read",
    "1122334455": "research"
  }
}

How to find folder IDs:

Open Twitter/X and go to your bookmarks
Click on a folder
The URL will be https://x.com/i/bookmarks/1234567890 - the number is the folder ID

When folders are configured:

Xhoard fetches from each folder separately
Each bookmark gets tagged with its folder name
Tags appear in bookmark files and knowledge file frontmatter

Note: Twitter's API doesn't return folder membership when fetching all bookmarks at once, so Xhoard must fetch each folder individually.

Automation

Run Xhoard automatically every 30 minutes:

Option A: PM2 (recommended)

npm install -g pm2
pm2 start "npx xhoard run" --cron "*/30 * * * *" --name xhoard
pm2 save
pm2 startup    # Start on boot

You can also use the included PM2 config example:

cp ecosystem.example.json ecosystem.json
# Edit "cwd" in ecosystem.json to your project path
pm2 start ecosystem.json --update-env
pm2 save

Option B: Cron

crontab -e
# Add:
*/30 * * * * cd /path/to/xhoard && npx xhoard run >> xhoard.log 2>&1

Option C: systemd

# Create /etc/systemd/system/xhoard.service
# See docs/systemd-setup.md for details

Option D: GitHub Actions (Serverless)

Use bunx to run Xhoard in GitHub Actions without installing anything:

Copy .github/workflows/archive-bookmarks.yml from this repo to your own
Add your Twitter credentials as repository secrets:
- TWITTER_AUTH_TOKEN
- TWITTER_CT0
The workflow runs daily at 6 AM UTC (or trigger manually)

Example workflow:

name: Archive Bookmarks
on:
  schedule:
    - cron: '0 6 * * *'  # Daily at 6 AM UTC
  workflow_dispatch:      # Manual trigger

jobs:
  archive:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: oven-sh/setup-bun@v1
      - run: bunx xhoard fetch --all
        env:
          AUTH_TOKEN: ${{ secrets.TWITTER_AUTH_TOKEN }}
          CT0: ${{ secrets.TWITTER_CT0 }}
      - run: |
          git config user.email "[email protected]"
          git config user.name "GitHub Action"
          git add bookmarks/ knowledge/
          git diff --staged --quiet || (git commit -m "Archive bookmarks $(date +%Y-%m-%d)" && git push)

Output

bookmarks/YYYY-MM-DD/*.md

Your bookmarks saved as individual files, organized by date folders:

---
tweet_id: "123456789"
author: "simonw"
date: "2026-01-02"
tweet_url: "https://x.com/simonw/status/123456789"
---

# Gist Host Fork for Rendering GitHub Gists

> I forked the wonderful gistpreview.github.io to create gisthost.github.io

- **Link:** https://gisthost.github.io/
- **Filed:** [gisthost-gist-rendering.md](./knowledge/articles/gisthost-gist-rendering.md)
- **What:** Free GitHub Pages-hosted tool that renders HTML files from Gists.

knowledge/tools/*.md

GitHub repos get their own files:

---
title: "whisper-flow"
type: tool
date_added: 2026-01-02
source: "https://github.com/dimastatz/whisper-flow"
tags: [ai, transcription, whisper, streaming]
via: "Twitter bookmark from @tom_doerr"
---

Real-time speech-to-text transcription using OpenAI Whisper...

## Key Features
- Streaming audio input
- Multiple language support
- Low latency output

## Links
- [GitHub](https://github.com/dimastatz/whisper-flow)
- [Original Tweet](https://x.com/tom_doerr/status/987654321)

Configuration

Copy the example config and customize:

cp xhoard.config.example.json xhoard.config.json

Example xhoard.config.json:

{
  "source": "bookmarks",
  "archiveMode": "files",
  "archiveDir": "./bookmarks",
  "archiveFile": "./bookmarks.md",
  "pendingFile": "./.state/pending-bookmarks.json",
  "stateFile": "./.state/bookmarks-state.json",
  "timezone": "America/New_York",
  "twitter": {
    "authToken": "your_auth_token",
    "ct0": "your_ct0"
  },
  "autoInvokeOpencode": true,
  "opencodeModel": "opencode/glm-4.7-free",
  "opencodeTimeout": 900000,
  "webhookUrl": null,
  "webhookType": "discord"
}

| Option | Default | Description | |--------|---------|-------------| | source | bookmarks | What to fetch: bookmarks (default), likes, or both | | includeMedia | false | EXPERIMENTAL: Include media attachments (photos, videos, GIFs) | | archiveMode | files | Archive format: files (per bookmark) or single | | archiveDir | ./bookmarks | Where per-bookmark markdown files are written | | archiveFile | ./bookmarks.md | Legacy single-file archive (used when archiveMode: single) | | timezone | America/New_York | For date formatting | | autoInvokeOpencode | true | Auto-run OpenCode for analysis | | opencodeModel | opencode/glm-4.7-free | OpenCode model (see OpenCode docs) | | opencodeTimeout | 900000 | Max processing time (15 min) | | webhookUrl | null | Discord/Slack webhook for notifications |

Environment variables also work: AUTH_TOKEN, CT0, SOURCE, INCLUDE_MEDIA, ARCHIVE_MODE, ARCHIVE_DIR, ARCHIVE_FILE, TIMEZONE, AUTO_INVOKE_OPENCODE, OPENCODE_MODEL, OPENCODE_TIMEOUT, etc.

Experimental: Media Attachments

Media extraction (photos, videos, GIFs) is available but disabled by default. To enable:

# One-time with flag
npx xhoard fetch --media

# Or in config
{
  "includeMedia": true
}

When enabled, the media[] array is included in the pending JSON with:

type: "photo", "video", or "animated_gif"
url: Full-size media URL
previewUrl: Thumbnail (smaller, faster)
width, height: Dimensions
videoUrl, durationMs: For videos only

⚠️ Why experimental?

Requires bird with media support - PR #14 adds media extraction. Until merged, you'll need a fork with this PR or wait for an upstream release. Without it, --media is a no-op (empty array).
Workflow still being refined - Short screengrabs (< 30s) don't need transcripts, but longer videos might. We're still figuring out the best handling.

OpenCode SDK Integration

Xhoard now invokes OpenCode programmatically through @opencode-ai/sdk.

It starts an isolated OpenCode server/client session per run.
It executes the project command at .opencode/commands/process-bookmarks.md.
It uses your configured model (opencodeModel) and respects opencodeTimeout.

You can also run processing manually inside OpenCode:

/process-bookmarks

Token Usage Tracking

Track your API costs with the -t flag:

npx xhoard run -t
# or
npx xhoard run --track-tokens

This displays a breakdown at the end of each run:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📊 TOKEN USAGE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Main (opencode/glm-4.7-free):
  Input:               85 tokens
  Output:           5,327 tokens
  Cache Read:     724,991 tokens
  Cache Write:     62,233 tokens
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Troubleshooting

"No new bookmarks to process"

This means either:

No bookmarks were fetched (check bird CLI credentials)
All fetched bookmarks already exist in your archive (bookmarks/ by default)

To start fresh:

rm -rf .state/ bookmarks/ bookmarks.md knowledge/
mkdir -p .state knowledge/tools knowledge/articles
npx xhoard run

Bird CLI 403 errors

Your Twitter cookies may have expired. Get fresh ones from your browser.

Processing is slow

Try a faster OpenCode model in opencodeModel for quicker processing
Make sure you're not re-processing with --force (causes edits instead of appends)

Only ~50-70 bookmarks fetched

The npm release of bird CLI (v0.5.1) doesn't support pagination. To fetch all bookmarks, install bird from git:

# Clone and build bird from source
cd /tmp
git clone https://github.com/steipete/bird.git
cd bird
pnpm install    # or: npm install -g pnpm && pnpm install
pnpm run build:dist

# Link globally (may need sudo or --force)
npm link --force

# Verify
bird --version  # Should show a newer commit hash
bird bookmarks --help  # Should show --all flag

Then use npx xhoard fetch --all to fetch all bookmarks with pagination.

Credits

bird CLI by Peter Steinberger
This project is a fork of smaug
Built with OpenCode SDK

License

MIT