mikser-io-provider-github

GitHub repositories as a content source for mikser-io. Configure a repo + ref; every file (filtered by include/exclude/path) becomes a mikser entity. Refs, manifest, layouts, renderers, vector indexing, MCP queries all work as if the docs were checked out locally.

Two surfaces, one package — same v9 provider convention as mikser-io-provider-gdrive:

• Lifecycle plugin (providerGithub) — sits in plugins[], authenticates against the GitHub API, polls configured repos, emits createEntity / updateEntity / deleteEntity.
• read(entity) named export — what mikser's engine dispatches into when an entity's uri is github://.... You don't import it directly; the engine does the dynamic import by package-name convention.

GitHub repo
   │
   ▼
providerGithub() → emits entities with uri = "github://<owner>/<repo>/<ref>/<path>"
                  + meta { githubBlobSha, githubModified, githubPath, ... }
   │
   ▼
catalog (mikser-io entities) — refs, manifest, lifecycle, render all work normally
   │
   ▼
readEntityContent(entity) → dispatches into THIS package's `read(entity)`
                          → base64-decode contents API for text,
                            mirror to runtime/github-cache/ for binaries,
                            error cleanly on LFS-tracked files

Install

npm install mikser-io-provider-github

Peer dep on mikser-io ^9. Hard deps on octokit and minimatch.

Setting up access — 30 seconds

GitHub access uses a Personal Access Token (PAT). Much simpler than the Drive setup. Either flavour works; fine-grained is recommended for new projects (scoped to specific repos).

Fine-grained PAT (recommended)

1. https://github.com/settings/personal-access-tokens/new
2. Token name: mikser-sync (or whatever)
3. Expiration: pick a duration that works for you (max 1 year)
4. Repository access: Only select repositories → pick the repos you want mikser to read
5. Repository permissions:
   • Contents → Read-only
   • Metadata → Read-only (auto-selected; required)
6. Generate token
7. Copy the token (github_pat_...); GitHub shows it once.

Classic PAT (works on every repo your account can see)

1. https://github.com/settings/tokens/new
2. Note: mikser-sync
3. Expiration: pick something
4. Scopes: check repo (private repos) or public_repo (public only)
5. Generate token
6. Copy the token (ghp_...)

Store it

# Common: env var in your shell profile / CI secrets
export GITHUB_TOKEN=github_pat_...

Or use a secret manager and pass it inline via auth: { token: ... } in your config.

Find the repo's owner / name / ref

From the repo URL https://github.com/almero-digital-marketing/mikser-io:

• owner: almero-digital-marketing
• repo: mikser-io
• ref: a branch (main), tag (v9.0.0), or commit sha. Required. If you want "always the default branch", set ref: 'main' explicitly — null / omitted is rejected.

Configure

// mikser.config.js
import { documents, frontMatter, yaml, renderHbs } from 'mikser-io'
import { layouts }        from 'mikser-io-layouts'
import { providerGithub } from 'mikser-io-provider-github'

export default {
    plugins: [
        documents(),
        frontMatter(),
        yaml(),
        layouts({ autoLayouts: true }),
        renderHbs(),

        providerGithub({
            auth: {
                token: process.env.GITHUB_TOKEN,
                // Also accepted: process.env.GH_TOKEN (matches gh CLI).
            },
            repos: [
                {
                    owner:      'almero-digital-marketing',
                    repo:       'mikser-io',
                    ref:        'main',
                    collection: 'documents',
                    prefix:     '/github/mikser-io/',     // mikser-side id prefix
                    path:       'documentation/',          // optional: only sync this subtree
                    include:    ['**/*.md'],
                    exclude:    ['_drafts/**'],
                },
                {
                    owner:      'almero-digital-marketing',
                    repo:       'mikser-io',
                    ref:        'v9.0.0',                  // tagged release docs
                    collection: 'documents',
                    prefix:     '/github/mikser-io@v9/',
                    path:       'documentation/',
                    include:    ['**/*.md'],
                },
            ],
            // Optional knobs (defaults shown).
            // pollIntervalMs: 60_000,
            // cacheFolder:    'runtime/github-cache',
        }),
    ],
}

Run mikser:

export GITHUB_TOKEN=github_pat_yourtoken
mikser --watch

Expect:

github: authenticated as <your-github-handle> (cache: /path/to/runtime/github-cache)
github: cold-scanned almero-digital-marketing/mikser-io@main @ a7bf170 — 47 files emitted (12 skipped by include/exclude/path)
github: watch poll every 60000ms

Subsequent polls log only when something changed:

github: polled almero-digital-marketing/mikser-io@main — 3 files changed between a7bf170 and 9365ebe

How it works

Cold scan (first run, or after --clear): one Git tree API call — GET /repos/{owner}/{repo}/git/trees/{sha}?recursive=1 — returns every blob at the ref. For each blob the plugin runs the path-subtree and include/exclude filters, then emits createEntity. Stashes the head commit sha so the next tick can go incremental.

Incremental polling (every pollIntervalMs in watch mode): GET /repos/{owner}/{repo}/compare/{lastSha}...{ref} returns the union of files changed between the last-synced commit and the current ref head. Each file carries status (added / modified / removed / renamed) and sha (the new blob hash). The plugin dispatches:

• added → createEntity
• modified → updateEntity only if the blob_sha changed. GitHub's blob sha is git's content-addressed hash, so identical content → identical sha → cheapest possible skip
• removed → deleteEntity
• renamed → delete old + create new (the old path lives in previous_filename)

Cache for binaries: PDFs, images, video, etc. land at runtime/github-cache/<path> with a sidecar <path>.sha recording the blob sha they came from. Re-reads hit the cache and skip the round-trip when the sha hasn't changed; a blob_sha mismatch triggers a re-download.

LFS: v1 detects LFS-tracked files and returns a clear error. The Contents API returns the LFS pointer file (~133 bytes starting with version https://git-lfs.github.com/spec/v1) as the "content" — that's not the real file, so we don't pretend. Following LFS pointers requires the separate Git LFS API and is v2 territory.

URI scheme

Every entity emitted by this plugin has:

entity.uri = 'github://<owner>/<repo>/<ref>/<path>'
entity.meta = {
    githubOwner:    'almero-digital-marketing',
    githubRepo:     'mikser-io',
    githubRef:      'main',
    githubPath:     'documentation/plugins.md',
    githubBlobSha:  'abc123...',           // git content-addressed hash
    githubMode:     '100644',              // file mode
    githubSize:     12_345,
    githubHtmlUrl:  'https://github.com/almero-digital-marketing/mikser-io/blob/main/documentation/plugins.md',
}

When any downstream plugin calls readEntityContent(entity), mikser-io parses github out of the scheme, dynamic-imports mikser-io-provider-github, and calls read(entity) — exactly the way it dispatches renderers and postprocessors. No registry, no descriptor coupling.

Content mapping

| File shape | What read(entity) returns | |---|---| | Text by extension (.md, .json, .yml, .html, .css, .js, .ts, .py, .go, …) | { content: <utf8> } — base64-decoded contents API; fallback to raw download for files >1MB | | Extensionless name in caps (README, LICENSE, MAKEFILE) | { content: <utf8> } — treated as text | | Binary by extension (.png, .pdf, .mp4, .zip, …) | { contentSkipped, cachedAt: <path> } — mirrored to local cache | | LFS-tracked file | { contentError: "LFS-tracked files not supported in v1" } | | Path is a directory (you pointed at a folder) | { contentError: "path is a directory, not a file" } |

Filtering — include / exclude / path

Three layers of selection, in order:

1. path: — only entries under this subtree at the repo root pass. e.g. path: 'documentation/'.
2. exclude: — patterns that drop matching paths. Default excludes always stack on top:

   node_modules/**    .git/**    .DS_Store    Thumbs.db    *.lock    .env    .env.*

3. include: — when set, ONLY matching paths pass. Empty/unset means "everything that survived path + exclude."

Patterns match against the repo-relative path (e.g. documentation/plugins.md). Standard minimatch globs. exclude always wins over include when both match.

Defaults exclude the common operator noise; you usually don't need to repeat node_modules/** in every config. Override the defaults by listing the same pattern in include if you genuinely want it.

Multiple refs of the same repo

Each {owner, repo, ref} tuple is independent — separate entity streams, separate state row, separate cache namespace. Pin docs to a tag while tracking dev on main:

repos: [
    { owner: 'me', repo: 'docs', ref: 'main',   prefix: '/docs/dev/',   ... },
    { owner: 'me', repo: 'docs', ref: 'v2.0.0', prefix: '/docs/v2/',    ... },
    { owner: 'me', repo: 'docs', ref: 'v1.4.0', prefix: '/docs/v1.4/',  ... },
],

Watch mode

mikser --watch schedules the polling loop. Default pollIntervalMs is 60s. The compare API call when nothing changed is a single round-trip that returns immediately with total_files: 0 — cheap. Set pollIntervalMs: 0 to disable polling (one-shot semantics).

For sub-minute latency, GitHub webhooks are the v2 answer. The substrate is ready (runtime.options.url is exposed, the route would mount on the engine-supplied Express app), but registering the webhook end-to-end isn't shipped in v1. See "What v1 doesn't do" below.

State persistence

mikser_provider_github_state (
    repo_key          TEXT PRIMARY KEY,    -- "<owner>/<repo>@<ref>"
    last_commit_sha   TEXT,
    last_synced_at    INTEGER
);
mikser_provider_github_files (
    repo_key          TEXT NOT NULL,
    path              TEXT NOT NULL,
    entity_id         TEXT NOT NULL,
    blob_sha          TEXT NOT NULL,       -- git content-addressed hash
    mode              TEXT,
    size              INTEGER,
    last_seen_at      INTEGER,
    PRIMARY KEY (repo_key, path)
);

Lives in mikser's main sqlite database (runtime/mikser.sqlite) via registerSchema. --clear wipes it; the next run does a fresh cold scan.

What v1 does NOT do

• Issues / PRs / comments / discussions / wikis / project boards — different domain, different schema. Probably lands as mikser-io-provider-github-issues with a different URI scheme (github-issue://owner/repo/123). Not in scope for the file-contents plugin.
• GitHub App auth. PAT only. Apps are better at the organization level (higher rate limits, scoped per repo, installation tokens) but add the cert flow. Deferred to v2.
• Webhooks for sub-minute push. Polling at pollIntervalMs is v1. The substrate (runtime.options.url) is ready; the webhook-receive route + signature verification + GitHub-side webhook registration is the work to do. Deferred to v2.
• LFS-tracked files. Detected and surfaced as a clear error. Following LFS pointers requires a second API client and authenticated raw fetches; deferred.
• Write-back (commit via PR). Read-only.
• Submodules. Not followed; submodule entries in the tree are surfaced as their own blob rows but the linked content isn't pulled. v2 if anyone needs it.
• Truncation handling for >100k file repos. GitHub truncates the recursive tree response above 100k entries; the plugin logs a warning and tells you to narrow with path: or include:. v2 could paginate via subtree walks.

Rate limits

PAT-authenticated calls get 5,000 requests/hour shared across your token. Conditional requests via If-None-Match (etag) — which Octokit handles automatically when available — return 304 and don't count against the quota.

The realistic per-repo budget:

• 1 cold scan = 2 calls (commits + tree)
• 1 incremental poll = 1 call when nothing changed (compare returns immediately)
• 1 incremental poll with N changes = 1 call (compare) + N calls (content fetch on demand via read(entity))

A repo with 100 files changed in the last interval is ~101 calls — comfortably under 5k/hr for tens of refs polled at 60s intervals.

Troubleshooting

github: token validation failed (HTTP 401)

Token is wrong, expired, or missing the scopes. Re-generate with Contents: Read-only + Metadata: Read-only (fine-grained) or repo / public_repo (classic).

github: tree response was truncated by GitHub (>100k entries)

The repo is too big for a single tree call. Narrow via path: (only sync documentation/ instead of the whole tree) or specific include globs.

github: <file> is an LFS-tracked file

Working as intended — v1 doesn't follow LFS pointers. Either exclude the file via exclude:, or wait for v2's LFS support.

github: 404 on cold scan

Either the repo doesn't exist at that path or the token doesn't have access to it. For fine-grained PATs, double-check the repo is in the "Selected repositories" list.

github: poll tick failed — secondary rate limit

GitHub's secondary rate limits kick in when you make many requests in a short window. Increase pollIntervalMs or add the affected repo to a less-frequent polling schedule.

Cache fills up

runtime/github-cache/ mirrors binaries the first time something reads them. Stale entries (blobs from old refs you no longer track) aren't auto-pruned. mikser --clear wipes it, or delete the folder manually.

Files don't update after a push

• Check that the ref in your config matches the branch you pushed to (main vs master, etc.).
• Verify pollIntervalMs isn't set too high.
• For watch mode, the poll runs in-process — restart mikser if it was killed.

License

MIT