KBGate (Knowledge Base Gate)

This document describes not existing reality but what should be at the very end.

A Markdown knowledge base which acts as universal content management system with a toolset that keeps those Markdown files in sync with other systems.

List of other systems

GitHub is supported natively since git tool can download and upload everything in this repo

• Confluence: Atlassian Wiki which allows to host documents
• Jira: Task Management system (download connector working — see below)
• Google Docs: Google documents which allows to host any documents
• Google Sheets: spreadsheets, mirrored either as forms or as tables (two-way — see below)
• Google Slides: presentations, mirrored as the text of the deck (download only — see below)
• Gmail: pulls unread mail (grouped by conversation) into Markdown (download only — see below)
• Any internal web app behind a corporate login: pulled through a browser session instead of an API token (download only — see below)

The repository is two things in one:

• The docs — admin and end-user documentation for the target project written as plain Markdown so it is easy to read, review, and diff.
• The tooling — Node.js scripts that facilitate upload and download from/to this repository to various systems.

New here? See USAGE.md for full setup, commands, and troubleshooting. The quickest start is npm run init: it lists the systems below, explains how to get the credentials for the one you pick, and writes the .remote.yml client for it.

How it works on example of Confluence

Confluence stores pages in its own rich format (ADF). This project treats the local Markdown as the source of truth and converts in both directions:

• Download: Confluence page (ADF) → adf-to-markdown → <Title>/readme.md. Images are saved inside the page folder and internal links point to local paths.
• Upload: <Title>/readme.md → markdown-to-adf → Confluence page.

The filesystem is the source of truth: each page is a folder, its identity (pageId + title) lives in metadata.md, and the page hierarchy is folder nesting. See USAGE.md for the layout.

A page's writable properties are plain YAML frontmatter, reconciled with the live page on every push:

| Key | What it is | How it is stored in Confluence | | ------------ | ------------------------------ | -------------------------------------------- | | owner | page owner | the page's ownerId | | labels | labels | content labels | | emoji | the emoji before the title | the emoji-title-published content property | | properties | the Page Properties macro rows | the details macro in the page body | | status | the status badge | content state — read-only, see below |

An absent key leaves the live value alone rather than clearing it: frontmatter states what a field should be, not everything it isn't. To say a field should be nothing, write emoji: null (a bare emoji: means the same in YAML) — pull then drops the key again, so it clears itself.

properties: is a mirror of an existing Page Properties macro, not a way to create one. The macro itself is never converted to Markdown — it stays a parked grommet (see below) — and push copies that parked node with only its value cells rewritten, so the table's structure and column widths survive. A macro that holds anything richer than plain key/value text (a link, a mention, several paragraphs in one cell) is left entirely alone and gets no frontmatter.

The content state (the status badge) is the exception: like a Jira status it is written through a trailing ! directive instead, so there is one writable path per field:

---

!In review

<optional free-text comment, after a blank line>

• !<state name> — set the page's content state.
• !- — remove it.
• status: in the frontmatter is written by pull and ignored by push.
• Unlike Jira's, a Confluence directive takes no property lines — every other writable field already lives in frontmatter.

Confluence has no workflow and no transitions: a state is a plain value, and any allowed state can replace any other. So the scan trans writes for a Confluence client is just the allowed list per space, not a graph:

npm run trans wiki_dev ENG        # one space
npm run trans wiki_dev ENG DOCS   # several

A space admin defines up to five space states and may also let authors add their own custom states; trans records both, and a space where states are switched off is omitted from <client>.trans.yml entirely. push resolves the directive's name against that file — and against the space's real settings — before anything is sent, so a typo or a stale scan aborts the push rather than being silently mis-resolved. Setting a state publishes a new page version with an unchanged body, so a state that already matches is never re-sent.

How it works on example of Jira

Jira issue description (and comments) come back as ADF natively in REST API v3, so the same adf-to-markdown converter used for Confluence handles the issue body. Unlike Confluence pages, issues have no hierarchy — a project is a flat list of issues, and each issue folder is named after its stable, unique issue key (no rename-tracking needed):

• Download: Jira issue → adf-to-markdown → <ProjectFolder>/<ISSUE-KEY>/readme.md, with summary/type/priority/labels/components/team/Prioritization_comment as YAML frontmatter properties ahead of the description body.

• Tracked history (status transitions, changes to the configured property/payload fields and description, comments) is clustered into sessions — events no more than 20 minutes apart — and written next to the readme. Edits made within 20 minutes of the issue's own creation are still drafting the ticket, not history, so they are dropped rather than recorded as a session. A shared .history.md index (a ## heading, then one list item per session, newest first, no blank lines between them) linking one .<YYYYMMDD_HHMM>.<client>.md detail file per session, dot-prefixed like the index, named after the session's end time. The index is shared across every remote the issue is mirrored to — each download only regenerates its own client's sessions and merges them back into the same chronology, reverse-sorted regardless of which remote a session came from. Details carry property changes as code blocks, comment bodies, and git-style diffs for rich-text fields. Each comment is its own event, headed ### @friendlyrhino, 2026-05-17 13:45 UTC, reply to @mega-builder — author and reply tags resolve through the remote's Users: dictionary, the "reply to" annotation coming from Jira Cloud's convention of opening a reply with an @mention (its comments have no real threading). History files live at the issue level (same place Jira puts attachments added through a comment) and are regenerated on every download; they are download-only, like comments.

• Uploading a new comment is supported: a trailing \n\n---\n\n<text> block at the very end of the readme (see bugs/JM-1/readme.md) is posted as a Jira comment on push, then the issue is re-downloaded so the local separator+text disappear. Headings weaker than ### are not allowed in that trailing text (outside fenced code). Re-uploading an existing comment (editing a comment already folded into .history.md) is not implemented.

• That same trailing block can instead be a ! directive, which push executes as a Jira workflow transition (and/or a plain field change) rather than posting it as a comment — status can never be written as a plain field PUT (a transition is a different kind of write), so this is the only way to move an issue's status through push:

  ---
  
  !<transition name or target status name>
  prop1: new value1
  prop2: new value2
  
  <optional free-text comment, after a blank line>

  • !<name> — resolves <name> against the issue's current status: either a transition name, or (if unambiguous) the name of the status it leads to.
  • ! alone (nothing after it) — no status change, just apply the properties below.
  • !! alone — change the issue's type; the first two property lines must be, in that order, type: <new type> then status: <new status>. Not every type change is a field write: Jira accepts only a type at the same hierarchy level whose workflow already holds the issue's current status. Neither is visible in the local files, so push asks the issue's own edit screen before sending anything and reports what is actually on offer instead of Jira's flat The issue type selected is invalid. 400 — see below.
  • Properties and the trailing comment are both optional in every form; a markdown image as the very first line (![alt](src)) is never mistaken for a directive, even though it also starts with !.

  Resolving a directive needs to know the target workflow's statuses and transitions, scanned ahead of time with trans:

  npm run trans jira_dev JM        # every issue type in project JM
  npm run trans jira_dev JM Bug    # only JM/Bug

  trans writes <client>.trans.yml next to the .remote.yml file — a flat, deduplicated list of workflows by name, each with its statuses and transitions (name/from/to). It never touches .remote.yml itself; a project's types.<type> entry must separately name which of those scanned workflows it uses:

  Projects:
    JM:
      types:
        Bug:
          workflow: Software Simplified Workflow for Project JM

  With that in place, push resolves a directive in two steps: it first validates the named workflow's own shape (exists, every transition's from/to is one of its statuses, no duplicate (from, name) pair, and no status/transition name collision — e.g. a transition literally named "Done" that doesn't actually lead to the status "Done"), then resolves the directive's target against that validated workflow and the issue's actual current status. Either step failing aborts the push with a clear error before anything is sent to Jira — a typo'd status/transition name, or a stale .trans.yml, is caught rather than silently mis-resolved.

  A !! type change is the one directive move no local file can predict, since Jira's answer depends on hierarchy levels and workflows the scan does not see. So push reads the issue's edit screen first and refuses locally, naming the types Jira does offer:

  ✗  GFENG-90254: Jira will not change "Task" to "Service Request" — from
     status "In Progress" it offers only: Task, Spike. A type at another
     hierarchy level, or one whose workflow lacks that status, needs Jira's
     own Move.

  When Jira offers no type at all — an Epic (hierarchy level 1) is the usual case, since no standard type shares its level — the message says so and points at Jira's Move, which is a different operation than a field write and has no REST equivalent here. The check runs in --dry-run too: learning that a !! cannot work is exactly what dry-running one is for.

  A transition can also carry a screen — its own form of fields, which Jira accepts only together with the transition, never through a plain field update. What is on that screen is not in the workflow scan (reading screens needs Jira admin), so push asks the live issue for the fields of the one transition it is about to run, and routes each directive property by membership: on the screen → sent with the transition, otherwise → the normal field update. Requiredness does not decide this — an optional screen field such as resolution still has to ride the transition.

  To pin that split by hand instead — a transition Jira reports differently than it behaves, or a screen you want spelled out in the file — give the transition either list in .trans.yml; naming any field at all switches that transition off the live metadata:

  transitions:
    - name: Resolve
      from: In Progress
      to: Verify
      required_fields: [resolution]
      optional_fields: [priority]

  Both lists take property names as written in the readme, and a name may appear in only one of them. required_fields does not change routing; it is used to explain a rejected transition, naming the screen fields the directive left unset.

Custom field IDs (customfield_NNNNN) differ per Jira site, so they are never hardcoded — a remote's Fields: block maps our property name to that environment's field ID (see jira_dev.remote.yml's Projects:/Fields: blocks for the pattern any future custom-field-bearing connector should follow).

A field's type is not declared anywhere: push reads it from Jira and sends each value in the shape that type wants — a number as a number, a date as YYYY-MM-DD, a select as { value: … }, a single label as a one-element array. This matters because a value typed in a ! directive is plain text: LoE_Estimate: 5 is the string "5" until the field's type says otherwise, and a number field answers a string with "Specify a number for the custom field". A value that cannot be that type is rejected locally, naming the property as the readme spells it, before anything is written.

The types ride along with the issue the push is about to change (expand=schema on the issue fetch), so they cost no extra call and need no special permission. Only creating an issue has no issue to read them off: that path asks for the site-wide field list (GET /rest/api/3/field) once per run instead. When neither can be read, push falls back to sending values as typed, which is what it always did.

A component is the one value a push must never invent. Jira does not reject an unknown component name when the caller may administer the project — it creates the component — so a typo would quietly add one to the project. push reads the project's own components instead and rejects anything else, naming what it does know:

✗  GFENG-90254: Components: "Okta CIS" is not a component of GFENG —
   known: Jira, Okta WIC, Okta CIC, SRE-1P&Cloud, SRE-Enterprise

That list is read once per project per run, and only when a readme names the field at all.

What the readme says is checked whether or not it differs from the live issue. Checking only what changed is how the original bug hid: once a push had created [SRE-1P&Cloud] and put it on the issue, the readme and the issue agreed, so the next push reported no changes and never looked at the value again.

A list value belongs in the frontmatter, written as a YAML list. A ! directive line is plain text, so Components: [SRE-1P&Cloud] there is a single value whose name includes the brackets — which is exactly how a component named [SRE-1P&Cloud] comes into existence. Any list-typed field written that way is rejected, with the form to use instead:

✗  GFENG-90254: Components is a list — a directive line is plain text, so
   "[SRE-1P&Cloud]" is one literal value. Write it in the frontmatter as:
   Components:
     - SRE-1P&Cloud

Some fields hold an opaque id rather than the name they display. Jira's Team field is one: sending FamilyTeam fails with "Cannot assign a non-existing team", because the field wants the team's UUID. A remote's optional Dictionaries: block supplies it:

Dictionaries:
  Team: # a property_fields name
    FamilyTeam: 55e351a0-f4ed-4f34-8b7e-3e59a287b5f4

push then sends the id in place of the name, and rejects a value the block does not list — naming the ones it does — instead of letting Jira answer with an opaque 400. pull writes whatever name the API reports, so a key here must be spelled exactly as that name: this is a lookup table, not a place for aliases. The ids for a field are quickest to read off GET /rest/api/3/jql/autocompletedata/suggestions?fieldName=<field>.

Issue links are frontmatter lists of issue keys. A link is one record in Jira that both issues show, each from its own side: "GF-113 implements GF-111" is Implements on GF-113 and Implemented By on GF-111. A remote's optional Links: block names one key per side of a link type:

Links:
  Implements: { type: Implements, direction: outward }
  Implemented By: { type: Implements, direction: inward }
Projects:
  GF:
    folder: gf
    property_fields: [priority, Implements, Implemented By]

# gf/GF-111/readme.md       # gf/GF-113/readme.md
Implemented By:             Implements:
  - GF-112                    - GF-111
  - GF-113

• type is the link type's name; direction is the side the issue itself is on. Link types differ per site: npm run trans <client> <PROJECT> prints a ready Links: block with both sides of every type. Paste it, delete what you do not need. Configuring only one side is fine — the other is then neither read nor written.
• A key works only where a project (or issue type) lists it in property_fields, like any other field.
• push compares the list with the live links, ignoring order: a new key creates a link, a removed key deletes one. An absent key leaves the links alone; Implemented By: [] removes all links of that side. Links are also made on a newly created issue.
• Each entry must be an issue key, and not the issue itself. Links are set in the frontmatter only, never under a ! directive.
• The other issue's readme does not change until it is pulled again — push names those issues. Link changes are not part of the tracked history.

Children of an issue (the issues whose parent it is — stories of an epic, sub-tasks of a story) are a frontmatter list of issue keys too. There is no config block: list the reserved name children in property_fields.

Projects:
  GF:
    folder: gf
    property_fields: [priority, parent, children]

# gf/GF-10/readme.md  (an epic)
children:
  - GF-11
  - GF-12

• pull finds the children with the JQL parent = GF-10, in the same project only. Children in other projects are not shown and not changed.
• push compares the list with the live children, ignoring order. A new key sets that issue's parent to this issue — this moves it away from its old parent. push reads the old parents first and warns (moving GF-12 (now under GF-20) here), also on --dry-run. A new key that Jira does not find stops the push before anything is written. A removed key clears that issue's parent. Jira refuses to clear the parent of a sub-task; the push then stops with Jira's error.
• An absent children leaves the children alone; children: [] detaches all. Each entry must be an issue key of the same project, and not the issue itself. Set it in the frontmatter only, never under a ! directive.
• The children's readmes, and the old parents of moved children, are behind until they are pulled again — push names them.
• ⚠️ If one push sets parent: GF-20 on GF-12 and also lists GF-12 under children of GF-10, the issue pushed last wins. When GF-10 is pushed last, the move warning shows it; the other order is not detected.
• ⚠️ Checked on Jira Cloud only. On Data Center, epic children use the Epic Link field, which children does not read or write.
• The name children cannot be used in payload_fields, Links: or Fields:.

A remote's optional Filters: block maps a name to either a JQL string (shorthand, all other settings default) or an object:

Filters:
  Quick_Filter: labels = urgent # shorthand
  Epics_In_Progress: # full form
    jql: issuetype = Epic AND status in ("Analyzing", "Approved") ORDER BY Rank ASC
    property_fields: [type, status, summary] # default: [type, status, summary]
    swimlanes: assignee # optional; any scalar field name; default: none
    instructions: Action for each ticket might be added below # this is also the default

A filter target is project-scoped as <PROJECT>:<name> (or <PROJECT>:filter:<id> for a saved Jira filter by id): its JQL runs as project=<KEY> AND (<jql>), the matched issues are downloaded normally, and a folder named after the filter is written next to them holding a ## Filter <name> (N) index, N the total matched issue count. Each matched issue is its own #### heading, #### [KEY](../KEY/readme.md): v1 | v2 | ... — one value per property_fields name, pipe-separated. When swimlanes is set, tickets are grouped first under ### <field>: <value> (N) headings (### is reserved for these swimlanes, N that group's issue count), ordered by first appearance, with a trailing ### <field>: n/a (N) group for issues with no value. The folder's .<client>/metadata.yml records the full project-scoped JQL that was actually run (project=<KEY> AND ...) and when it was downloaded.

A folder is read as such an index — by pull and push alike — only through links that point at this remote: a relative ../KEY/readme.md, or an absolute URL on the remote's own host. A key-shaped link to another Jira site, or to any other system, is ordinary body text, so an issue can cite [JRACLOUD-81901](https://jira.atlassian.com/browse/JRACLOUD-81901) and still be pulled, pushed, and created like any other.

npm run pull jira_dev CJ-27                 # one issue
npm run pull jira_dev CJ                    # whole project
npm run pull jira_dev CJ:Epics_In_Progress  # a configured filter
npm run pull jira_dev CJ:filter:10042       # a saved filter by id

How it works on example of Google Docs

Google Docs has no ADF. Instead of a custom converter pair, the connector delegates conversion to Drive itself: pull asks Drive to export the document as Markdown (files.export?mimeType=text/markdown), and push sends the local Markdown back as a media upload that Drive converts into a Google Doc. The interim dump is therefore Markdown, kept in the gitignored .inbox//.outbox/ folders like the ADF dumps of the other connectors.

• Download: Google Doc → Drive Markdown export → <Doc_Name>/readme.md. The Drive file name becomes the # title heading; inlined base64 images are extracted into real files next to the readme; body # headings are shifted down one level (recorded as headingShift in metadata and reversed on push). A Drive folder tree pulls as nested local folders — a folder dir carries only a folderId in its .<client>/metadata.yml, a doc dir carries docId, modifiedTime, and contentHash.
• Upload: <Doc_Name>/readme.md → local images re-inlined as data URIs → Drive Markdown import. A dir with a docId is updated (renamed too when the # title changed); one with a readme but no docId is created under the nearest ancestor Drive folder, and new local dirs on the way become Drive folders. No-op pushes are detected by comparing the readme's sha256 against the contentHash recorded at pull time — the round-trip is lossy, so re-export comparison would always differ. If the remote modifiedTime moved since the last pull, push overwrites it with a warning.

One-time setup: create a Desktop app OAuth client in Google Cloud Console, enable the Drive API, run node src/tools/gdocs-auth.mjs, and put GDOCS_CLIENT_ID/GDOCS_CLIENT_SECRET/GDOCS_REFRESH_TOKEN into .env (see gdocs_dev.remote.yml and USAGE.md).

Decision: round-trip through Drive's own Markdown converter is lossy

Because a push replaces the whole document with re-imported Markdown, any Google-side formatting that Markdown cannot express is destroyed on push — the same "filesystem is the source of truth" stance as the Confluence connector.

| Construct | Survives the round-trip? | | --------------------------------------------- | ----------------------------------------------------------------- | | Headings, lists | ✅ verified live | | Images | ✅ verified live — re-inlined as base64 data URIs, layout reset | | Bold/italic/strikethrough | expected ✅ (not yet live-verified) | | Tables (structure), links, blockquotes, rules | expected ✅, column widths/cell styling reset (not live-verified) | | Code blocks | expected structure ✅; fence language likely lost (not verified) | | Comments / suggestions | ❌ not exported, gone after push | | Fonts, colors, page breaks, headers/footers | ❌ no Markdown representation | | Drawings, charts, footnotes | ❌ dropped or rasterized (not yet live-verified) | | Tabs beyond the first (multi-tab docs) | ❌ only the first tab is exported (not yet live-verified) |

The export is capped at 10 MB by Google; larger docs fail with the raw HTTP error. Documents and folders are matched by id, so local renames survive a re-pull, mirroring the Confluence layout logic.

How it works on example of Google Sheets

A spreadsheet is not one document but a workbook of tabs, and a tab is used in one of two ways, which one connector cannot serve. So there are two remote types, and a client picks one:

• GOOGLE_SHEETS_FORM — the sheet is a form: individual cells or named ranges hold the values of one record. Each property names a range, and the whole sheet becomes one readme.md.
• GOOGLE_SHEETS_TABLE — the sheet is a table: the header row names the properties and every row below it is a record. Each property names a column, and each row becomes its own folder.

Drive nesting becomes directory nesting, exactly as in the Google Docs connector:

Summit_2026/                      <- Drive folder   (.<client>/metadata.yml: folderId)
  Ukraine_Action_Summit/          <- spreadsheet    (spreadsheetId, modifiedTime)
    To_do_list/                   <- sheet          (spreadsheetId, sheetId, sheetTitle)
      Book_the_venue/             <- row (TABLE)    (+ rowKey, rowIndex)
        readme.md
      row_0007/                   <- a row whose key is empty or duplicated
        readme.md
    Delegate_Profile/             <- sheet (FORM)
      readme.md                   <- the record itself; no row level

A row's folder is named after its key column, which is what lets a re-pull find the same row after it moved, was re-sorted, or the folder was renamed locally. A key that is empty — or shared with another row, and therefore identifying nothing — falls back to row_<NNNN>, the spreadsheet's own row number.

Nothing is discovered at pull time. Which sheets are mirrored, and how their cells map to properties, comes from the .remote.yml Spreadsheets: block, validated against a scan written by trans:

npm run trans gsheets_dev https://docs.google.com/spreadsheets/d/<id>/edit
npm run trans gsheets_dev https://drive.google.com/drive/folders/<id>

trans writes <client>.trans.yml next to the .remote.yml — every sheet's sheetId and header-row columns, plus the workbook's named ranges resolved to A1 — and prints a paste-ready Spreadsheets: block, because wiring a 40-tab workbook by hand is the real cost of this connector. It never touches .remote.yml itself. A sheet the config does not name is skipped with a warning; a column or named range the scan does not know is an error before any request goes out, rather than a silently empty property.

Rows that vanish upstream are listed, never deleted, and push creates and deletes none either: it writes values into cells that already exist. The local copy may be the only one left of a row somebody removed.

One-time setup: create a Desktop app OAuth client in Google Cloud Console, enable the Sheets API and the Drive API, run node src/tools/gsheets-auth.mjs, and put GSHEETS_CLIENT_ID/GSHEETS_CLIENT_SECRET/GSHEETS_REFRESH_TOKEN into .env (see examples/gsheets-form.remote.yml and examples/gsheets-table.remote.yml). The scopes are read-only by default — a token that cannot write is one less way to damage a live spreadsheet — so a client that pushes needs --write as well (see below).

Writing back: only what a human changed

push is off unless the client says Push: true, and it never sends a cell nobody edited. That is not caution for its own sake: a pull renders formattedValue, the text on screen rather than the value in the cell, so re-sending an untouched cell is a lossy write. Which cells changed is answered by .<client>/baseline.yml — one sha256 per field, written at pull time — and each field gets a three-way verdict:

| Local vs baseline | Sheet vs baseline | What happens | | ----------------- | ----------------- | ------------------------------------------- | | same | — | never sent | | changed | same | written | | same | changed | left alone; pull brings the change in | | changed | changed | refused, naming both values; --force wins |

The unit of atomicity is the record. If any field of a readme cannot be pushed, nothing of that record is written — not the other fields, not the metadata, not the baseline. Half a record upstream is worse than none, and the operator gets one list to fix rather than a drip.

Everything is checked locally, before the first write, because Sheets answers 200 to a write that silently destroys a formula or a chip — the API's own answer proves nothing. A record is refused when a changed field is a formula cell or holds a smart chip (pull mirrors both as text, and a write would replace them), when a property value would become a formula, when a ## heading names no configured field, when the local Markdown would not mirror back identically, or when a TABLE row has no unique key to find it by. --force lifts only the both-sides-changed refusal; the rest mean the write would damage data.

Two endpoints do the writing, for a reason: property cells go through values.batchUpdate as USER_ENTERED, so a number stays a number and a date stays a date, while payload cells go through spreadsheets.batchUpdate, the only call that can carry textFormatRuns. They share no transaction, so the baseline advances per call and a half-finished push resumes without re-sending.

Push resolves every address against the live workbook and never reads <client>.trans.yml: a named range moves when rows are inserted, and a stale address would not mis-read a value — it would write into the wrong cell. For the same reason a named range is the safer thing to point a two-way FORM property at; a literal C1:C2 in the config rots silently. A range covering several cells is written at its top-left, and push says so when the value it read came from elsewhere in that range.

Decision: rich text only in the body, never in the frontmatter

Cells are read through spreadsheets.get?includeGridData=true, not through values.batchGet: the values endpoint returns bare strings, with no formatting, no links and no chips. Everything rendered as Markdown lives on CellData.

Where that richness is allowed is a deliberate split:

• Property fields are always plain. YAML frontmatter has to stay scalar — a **bold** value there is noise for a human and an LLM alike — so a property cell contributes its displayed text and nothing else.
• Payload fields become Markdown. textFormatRuns turn into emphasis, links into links. If a property turns out to need a link, move it to payload_fields rather than loosening the frontmatter rule.

Decision: images are not mirrored, in any form

CellData has no image field of any kind — its full field list is userEnteredValue, effectiveValue, formattedValue, userEnteredFormat, effectiveFormat, hyperlink, note, textFormatRuns, dataValidation, pivotTable, dataSourceTable, dataSourceFormula, chipRuns. A picture placed through Insert → Image, whether "in cell" or "over cells", is therefore invisible to the API: it cannot be downloaded, and it cannot even be detected well enough to warn about.

An =IMAGE("…") formula could be read, since the URL sits in formulaValue. Covering only that path would make the mirror sometimes right about images, which is worse than one that is consistently silent — so no image files are written and no ![](…) is emitted. An =IMAGE() cell keeps its formula as text, which at least leaves the URL readable.

| Construct | Survives the pull? | | ---------------------------------------------------------------------------- | ----------------------------------------------- | | Cell values (dates, currency, percentages as displayed) | ✅ via formattedValue | | Bold / italic / strikethrough in a payload cell | ✅ from textFormatRuns | | Links (a whole-cell hyperlink, or one run) | ✅ as [text](uri) | | Smart chips (person, rich link) | ✅ as their email or link | | Formatting in a property cell | ⚠️ dropped by design — frontmatter stays scalar | | Images, every form | ❌ not supported — see above | | Charts | ❌ sheets.charts is a spec, not a rendering | | Cell notes | ❌ fetched but dropped | | Formulas, merged cells, fill colour, conditional formatting, data validation | ❌ no Markdown representation |

Everything that survives the pull can be pushed back, with two exceptions: a formula cell mirrors as its formula text and a smart chip mirrors as its label, and neither can be rebuilt from what the Markdown holds, so a changed one is refused rather than flattened. A cell's own formatting — number format, colour, width — is untouched by a write; only its value and, for a payload cell, its inline marks are replaced.

How it works on example of Google Slides

Google Slides pull is download-only (no push), and it deliberately mirrors only what Markdown can hold: the text of the deck. A slide is a canvas — positions, sizes, layers, animations — and a push would have to rebuild all of that from a Markdown file, which the "filesystem is the source of truth" model cannot do. So the deck is read, never written.

One presentation is one publication entry: a folder with a readme.md and the pictures next to it. Drive nesting becomes directory nesting, exactly as in the Docs and Sheets connectors.

Board_decks/                        <- Drive folder   (.gslides/metadata.yml: folderId)
  Q3_Business_Review/               <- presentation   (presentationId, modifiedTime)
    readme.md
    Q3_Business_Review_10.png

• Download: presentation → presentations.get → <Deck_Name>/readme.md. The Drive file name is the # title, every slide is a ### heading, and a slide's subtitle placeholder is ####. Speaker notes follow the slide in a fenced block tagged notes — they carry no formatting, and a fence keeps them from being read as Markdown nobody wrote.
• Upload: not supported.

Decision: heading levels stop at ### per slide

The mapping is # deck → ### slide → #### subtitle. ## is unused. Google Slides has no notion of a section: presentations.get returns a flat list of slides, and the only thing that hints at a break is a layout name, which a custom theme is free to change. Rather than guess wrong on half the decks, the level is left empty for a future signal that actually exists.

Inside a slide there is likewise no heading structure to read — a text box is a text box — so only the SUBTITLE placeholder becomes #### and everything else is paragraphs, lists and tables.

Two things the API does not order for you, and this connector does:

• pageElements come back in z-order, not reading order, so elements are sorted top-to-bottom then left-to-right before conversion.
• A group nests its children under its own transform, so groups are flattened with the transforms composed; a grouped label lands where it is seen, not where the API happens to list it.

Decision: a picture is content only if it is not furniture

Everything a slide inherits from its layout or master is invisible in slide.pageElements, so a template's frame, rules and page numbers cost nothing. What remains are pictures placed on the slide itself, and those are judged by three thresholds, all configurable per remote in an Images: block because every corporate template has different proportions:

| Rule | Config key | Default | | ----------------------------------------------------------- | -------------- | ------- | | The slide's own pageBackgroundFill picture | — | always | | Covers this share of the slide area or more (a backdrop) | max_area | 0.9 | | Covers less than this share of the slide area (an icon) | min_area | 0.02 | | The same bytes on this share of the slides or more (a logo) | repeat_ratio | 0.5 |

The repeat rule needs the bytes, so it runs after download: pictures are hashed, and identical bytes appearing on more than one slide and on enough of the deck are dropped. Every run prints how many pictures each rule rejected, so a threshold that is wrong for a template is visible rather than silent.

Identical bytes that survive are written once and linked wherever they appear.

One-time setup: create a Desktop app OAuth client in Google Cloud Console, enable the Slides API and the Drive API, run node src/tools/gslides-auth.mjs, and put GSLIDES_CLIENT_ID/GSLIDES_CLIENT_SECRET/GSLIDES_REFRESH_TOKEN into .env (see examples/gslides.remote.yml and USAGE.md). Both scopes requested are read-only — the connector never writes, so a token that cannot write is one less way to damage a live deck.

npm run pull gslides https://docs.google.com/presentation/d/<id>/edit
npm run pull gslides https://drive.google.com/drive/folders/<id>
npm run pull gslides Board_decks/Q3_Business_Review

A deck that vanishes upstream is listed, never deleted — and only a folder pull can tell, since pulling one deck says nothing about its neighbours.

| Construct | Survives the pull? | | ---------------------------------------------------- | ------------------------------------------------------------------ | | Slide titles, subtitles, paragraphs | ✅ as ### / #### / text | | Bold, italic, strikethrough | ✅ | | Bulleted and numbered lists, nested | ✅ ordered vs bulleted is read off the rendered glyph | | Links | ✅ as [text](url) | | Tables | ✅ as GFM; the first row is taken as the header | | Pictures and linked Sheets charts | ✅ downloaded next to readme.md, unless a threshold rejects them | | Speaker notes | ✅ in a fenced block tagged notes | | Videos | ✅ as a link — the file itself is not mirrored | | Underline, superscript, subscript | ⚠️ kept as literal <u>/<sup>/<sub> (no Markdown syntax) | | Formatting in a slide title | ⚠️ dropped by design — a heading is a label, like a property | | Merged table cells | ⚠️ text kept in the top-left cell, covered columns left blank | | Links from one slide to another | ⚠️ plain text — there is one file, so there is nothing to point at | | Background and decorative pictures | ⚠️ dropped by the Images: thresholds, and counted in the report | | Sections | ❌ the API has no such concept — see above | | Layout, sizes, positions, colours, fonts, animations | ❌ no Markdown representation | | Comments, revision history | ❌ not exposed by presentations.get |

How it works on example of Gmail

Gmail pull is download-only (no reply/send). The unit of pull is the thread, not the message: an unread message's whole thread is fetched in full and rendered as one conversation transcript, so a reply on an already-pulled thread updates the same readme.md instead of creating a second folder.

• Download: unread messages under a configured Gmail label (Labels: in the .remote.yml, default { INBOX: Inbox }) are grouped by thread; each thread's messages (oldest first) become ## <From> — <date> sections in <label folder>/<YYYY-MM-DD-subject-slug>/readme.md, dated by the first message. The HTML body is converted with turndown (headings, bold, italic, lists, blockquotes, links, images); inline images are extracted next to the readme (<slug>_10.png, …) and real attachments are saved as plain files. .<client>/metadata.yml carries the threadId, used to match the same folder on re-pull across renames.
• Mark as read: off by default — pulling never mutates the mailbox. Setting MarkAsRead: true clears a pulled thread's UNREAD label in Gmail after it is written to disk.

One-time setup: create a Desktop app OAuth client in Google Cloud Console, enable the Gmail API, run node src/tools/gmail-auth.mjs, and put GMAIL_CLIENT_ID/GMAIL_CLIENT_SECRET/GMAIL_REFRESH_TOKEN into .env (see examples/gmail.remote.yml and USAGE.md).

Decision: the HTML→Markdown conversion is lossy

Only the constructs turndown can already express survive the download; Gmail quoted-reply blockquotes convert to > rather than being stripped.

| Construct | Survives the pull? | | ------------------------------------------- | --------------------------------------------------------------------- | | Headings, bold, italic, lists, links | ✅ | | Blockquotes (incl. Gmail's quoted replies) | ✅ rendered as > | | Inline images | ✅ extracted to real files, links rewritten | | Real attachments | ✅ saved as files next to readme.md | | Underline | ⚠️ kept as literal <u>…</u> inline HTML (no Markdown syntax exists) | | Fonts, colors, signatures-as-images styling | ❌ no Markdown representation | | Read/unread and other Gmail labels | not represented in readme.md (see MarkAsRead) |

How it works on example of a browser session

Every other connector needs a credential the operator can put in .env. A corporate internal system usually hands out neither an API token nor an OAuth client — access goes through the company SSO, and the only thing that proves identity is a session cookie inside a browser. A BROWSER remote turns that cookie into a credential:

• Login (once): npm run login <client> opens the remote's own browser profile in a visible window at its BASE_URL. The operator passes the SSO by hand and presses Enter; the cookies stay in the profile directory named by PROFILE:. ⚠ That directory is a live corporate session — it is gitignored and must never be committed.
• Download: npm run pull <client> <URL> opens the same profile headless, renders the page, takes the first node matching the remote's Selectors.content, removes its Strip: selectors, and converts the result with turndown into <Title>/readme.md. Images are downloaded through the same session (so authenticated attachments work) and rewritten to local files. .<client>/metadata.yml carries url, title, pulledAt and a contentHash; a re-pull matches the folder by url and skips writing when the Markdown has not changed.
• Upload: not supported. Writing back would mean driving each system's own editor UI.

A URL outside the remote's BASE_URL origin is refused before the browser starts — the corporate session is never sent to a host the remote was not configured for. If a navigation ends on another origin, that is the SSO bounce, reported as an expired session with the login command to fix it.

The connector knows nothing about the target system: what it takes and what it throws away lives in the .remote.yml, not in the code (see examples/browser.remote.yml).

Decision: rendered DOM, not the system's API

Reading a system's REST API converts better — it is how the Confluence and Jira connectors get real structure. It also has to be written per system, and needs an API credential the operator does not have. This connector deliberately takes the lowest common denominator instead: whatever the browser has already rendered. When one system dominates in practice, a connector speaking its API through the same session is the better next step, not more selectors.

| Construct | Survives the pull? | | --------------------------------------------- | ------------------------------------------------------------------------------ | | Headings, bold, italic, lists, links | ✅ | | Tables, strikethrough | ✅ via turndown-plugin-gfm; a cell holding a list or a nested table flattens | | Images (including behind the login) | ✅ downloaded next to readme.md, links rewritten | | Relative links | ✅ made absolute, pointing back at the source system | | Underline | ⚠️ kept as literal <u>…</u> inline HTML (no Markdown syntax exists) | | Page furniture (nav, sidebars, footers) | ⚠️ removed by Strip: — a site-specific list | | Content rendered after a click (tabs, expand) | ❌ only what is in the DOM after load is seen | | Embedded viewers (PDF, video, dashboards) | ❌ no Markdown representation | | Page identity, versions, comments, labels | ❌ not exposed by the rendered page |

What tooling should provide for each system

What is I need not just Confluence and GitHub? Be my guest.

The tooling can be developed for any systems.

System definition

System is defined by so called remote. Remote configuration consist of type (ATLASSIAN_CONFLUENCE, ATLASSIAN_JIRA, GOOGLE_DOCS, GMAIL, GOOGLE_SHEETS_FORM, GOOGLE_SHEETS_TABLE, GOOGLE_SLIDES and BROWSER are supported today), but might be supported more once it will be developed. Configuration is defined by .remote file. Which contains type and access.

Besides name and access any system configuration should define:

• Define main publication entry scope. e.g. for Confluence it is the page with attachments, labels and comments. Different pages or folder or whiteboard are different entries.
• Define main content of the publications. e.g. for Confluence it is a text and MD-relevant formatting on the page and labels.
• Define decorators: what is system specific decorators are that doesn't have representation in MD but exist in e.g. sizes of the columns in Confluence are in decorators, while table itself is stored in MD.
• Define grommets in Markdown: what should be left in MD to put decoration on it. E.g. tables can be referenced by captions, columns in Confluence can be converted to tags
• Define backstage: which elements in MD won't be stored to the target system and will be replaced by posters. E.g.
• Define how posters are created from MD content and how they are recognized during the download and omit overriding the backstage. e.g. mermaid diagrams will be converted to image and uploaded as image with attachment that describes mermaid as text. And during download it will be automatically picked up
• Define user dictionary: what should be the user tag which is used in GitHub to be transpiled to in the target system e.g. friendlyrhino --> [email protected]. The Users: block in a .remote.yml can either be inline or a reference to a shared file (e.g. Users: users.yml), so several remotes pointed at the same site don't have to repeat the same list.

Tooling should cover following scenarios:

• Download a publication with full scope to the interim dump (for Confluence it is ADF)
• Download BLOB Attachments from the publication (download everything that enhances content of the individual publications, e.g. images and video)
• Upload the interim dump to the target system
• Update BLOB Attachment
• Extract Markdown with grommets from the interim dump
• Extract decoration to transpiler
• Compare two interim dumps with ignoring format and volatiles
• Stage Markdown to the interim dump

How to implement connector with the system?

Basically each scenario should be tested with round-trip. Initially you should have content locally (so called local mirror) We have it in Test Mirror. The round trip works as contract and unit tests:

• check connection to the target system
• download mirror
• format resulted MD files by using prettier
• check that id is retrieved
• upload mirror
• ensure no new version is created, because no change was made
• do minor change to the local MD in mirror
• run verify and make sure no mistake is found
• upload mirror and ensure that new version is created
• change structure by subordinating Test mirror: Part 2 to Test mirror: Part 1 and recover references.
• run upload and see that Test mirror: Part 2 is now under Test mirror: Part 1

This at first run manually, but later can be automated.

How the data is stored on the local machine

• Each publication entry represented by the folder
• the folder contains readme.md which represents everything that is most important to human operator and to LLM is stored in file readme.md
• the folder contains all attachments related to the publication entry

Browse the docs

Read the .md files directly, or run the local docs server (docsify):

npm install
npm run serve      # then open http://localhost:3000

Sync with Confluence

You need Node.js 18+ and a .env file with your Atlassian credentials (ATLASSIAN_URL, ATLASSIAN_USER, ATLASSIAN_TOKEN) — copy the format from env.example. Never commit .env; it is already ignored.

Two commands, both run as npm run <name>:

Both take a client (a <client>.remote file) as the first argument, which selects the target system (Confluence or Jira, dispatched from the client's .remote.yml type) and its credential set:

| Command | What it does | | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | pull | Download into folders. Confluence: space keys or page URLs (npm run pull confl eng). Jira: project key, issue key, or filter (npm run pull jira_dev CJ-27). Gmail: unread threads under a label (npm run pull gmail_dev INBOX). Google Slides: a presentation or folder URL (npm run pull gslides <url>). | | push | Upload folders back to the remote. Confluence: page folders (npm run push confl eng/Test_mirror [--dry-run]). Jira: issue folders (npm run push jira_dev bugs/JM-27 [--dry-run]). |

Full step-by-step setup, workflows, and troubleshooting are in USAGE.md.

Tests and formatting

npm run test        # unit tests for the ADF <-> Markdown converters
npm run test:e2e    # full Playwright tests
npm run format      # format everything with Prettier

Architecture

src/ is split by how well-tested and system-specific each part is:

• src/backend/ (adf-to-markdown.mjs, markdown-to-adf.mjs, adf-compare.mjs, page-store.mjs, frontmatter.mjs, transpile.mjs, remote.mjs, cell-markdown.mjs, markdown-to-cell.mjs, sheet-record.mjs, sheet-map.mjs, sheet-baseline.mjs, slides-markdown.mjs, markdown-inline.mjs) — the ADF ⇄ Markdown core (shared by both the Confluence and Jira connectors — Jira's description/comment fields return ADF natively, same as Confluence page bodies), the filesystem-backed page store/frontmatter/macro-decorator parking, .remote.yml credential resolution, and the pure converters for the systems that have no ADF: cell-markdown.mjs (a Sheets cell) and slides-markdown.mjs (a whole presentation). markdown-to-cell.mjs is the inverse of the first — Markdown back into a cell's text and textFormatRuns, which is what makes the Sheets connector two-way — and sheet-baseline.mjs remembers, as one hash per field, what the mirror last agreed with the spreadsheet, so a push can tell a local edit from someone else's. markdown-inline.mjs holds the inline emphasis/escaping both of those share, over a neutral mark set each connector maps its own API format onto.
• src/gateway/ (confluence-api.mjs, jira-api.mjs, gdocs-api.mjs, gmail-api.mjs, gsheets-api.mjs, gslides-api.mjs, google-oauth.mjs) — the raw REST clients: stateless functions that take baseUrl/authHeader (or, for Google, the OAuth creds object) and talk to each remote system's API. google-oauth.mjs is the shared refresh-token→access-token exchange used by every Google gateway. gdocs-api.mjs doubles as the generic Drive client — getFile/listChildren/FOLDER_MIME are not Docs-specific, and the Sheets and Slides connectors import them from there rather than growing a second copy. No local-folder or CLI concerns live here.
• src/frontend/ (pull.mjs, push.mjs, confluence-upload.mjs, confluence-download.mjs, jira-download.mjs, jira-upload.mjs, gdocs-download.mjs, gdocs-upload.mjs, gmail-download.mjs, gsheets-download.mjs, gsheets-upload.mjs, gsheets-targets.mjs, gslides-download.mjs, drive-mirror.mjs, concurrency.mjs, test-full.mjs) — the CLI orchestration that walks local folders and wires the gateway + backend pieces together. pull.mjs/push.mjs are the npm run pull/npm run push entry points: they load the selected client's .remote.yml (remote.mjs) and dispatch on its type to the matching connector's exported runPull/runPush — Confluence's in confluence-download.mjs/confluence-upload.mjs, Jira's in jira-download.mjs/jira-upload.mjs, Google Docs' in gdocs-download.mjs/gdocs-upload.mjs, Gmail's (pull-only) in gmail-download.mjs, Google Sheets' (both remote types, two-way) in gsheets-download.mjs/gsheets-upload.mjs, Google Slides' (pull-only) in gslides-download.mjs. drive-mirror.mjs is what the three Drive-backed connectors share: the CLI argument shape, the target-URL parser, the scan of what is already mirrored, and the collision-free rule for naming a new directory. Each connector module stays directly runnable on its own (node src/frontend/jira-download.mjs …) for debugging.
• src/tools/ (init.mjs, init-types.mjs, confluence-folder-to-page.mjs, google-consent.mjs, gdocs-auth.mjs, gmail-auth.mjs, gsheets-auth.mjs, gslides-auth.mjs, browser-login.mjs) — separate utilities: setting a client up (init), converting local folders into Confluence documents/pages, and the one-time Google OAuth consent flows that produce GDOCS_REFRESH_TOKEN/GMAIL_REFRESH_TOKEN/GSHEETS_REFRESH_TOKEN/GSLIDES_REFRESH_TOKEN. The four auth tools differ only in scope and .env prefix, so the loopback consent flow itself lives once in google-consent.mjs. init.mjs is the npm run init wizard — it writes a <client>.remote.yml and the .env keys that client needs; everything that differs from one target system to the next (what it mirrors, how to get its credentials, the YAML to write) is data in init-types.mjs, so adding a connector means adding a recipe, not a branch. Not part of the frontend, gateway, or backend.

Where does a new file go?

Ask these questions, in order, about what the file does:

1. Does it talk to the network (a remote system's REST API) and nothing else — no local folders, no CLI args? → src/gateway/.
2. Does it work only with local data (ADF, Markdown, YAML, the filesystem) with no network calls of its own? → src/backend/.
3. Does it orchestrate a CLI command — parse process.argv, walk local folders, and call into gateway//backend/ to do the real work? → src/frontend/.
4. Is it a standalone utility that doesn't fit any of the above (its own CLI, its own concerns, not reused by the other connectors)? → src/tools/.

This is why, for example, jira-download.mjs and jira-upload.mjs live in src/frontend/ (they parse CLI args and orchestrate a full download/upload run) while jira-api.mjs lives in src/gateway/ (it only wraps Jira's REST endpoints) and remote.mjs lives in src/backend/ (it only reads local .remote.yml/.env files, no network calls).

Conventions

• Every readme.md starts with a single # Heading; the authoritative page title is title in metadata.md and the H1 is stripped on upload.
• Links between pages use relative paths to the target's readme.md; on upload they are rewritten to Confluence URLs (reconstructed from the target pageId).
• Project rules live in CLAUDE.md; Definitions of Done live in definitions_of_done/.

Decision: native Confluence Markdown export is NOT a replacement

Question evaluated: Confluence Cloud now exposes a native Markdown body (GET /wiki/api/v2/pages/{id}?body-format=markdown). Can we drop our custom src/backend/adf-to-markdown.mjs converter and pull pages as Markdown directly?

Verdict: No. As of July 2026 the endpoint works and is API-accessible, but the native output is lossy on exactly the constructs our converter exists to handle. Switching would silently lose all images and all internal cross-references. Keep the ADF → Markdown pipeline as the primary path.

Evidence — probed against a torture-test page carrying every edge case:

| Construct | Native body-format=markdown | Our adf-to-markdown | | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------- | | Bold / italic / strikeout, tables, nested lists, JQL autolinks | ✅ correct | ✅ | | Images / attachments | ❌ dropped entirely — only the caption text survives, no ![]() and no attachment link | ✅ downloads the binary, writes ![alt](local.png) | | Cross-references to our pages | ❌ flattened to plain text, so remapLinks has nothing to rewrite into a local file link | ✅ preserved as a link and remapped | | Underscore / superscript / subscript | ❌ escaped as literal text (\<u\>…\</u\>, \<sup\>…\</sup\>) | ✅ rendered | | Info / warning / error panels | ❌ flattened, structure lost | ✅ | | Layout columns / [TOC] | ❌ flattened | ✅ | | Mentions @User | ~ bare @Name, no account link | ✅ | | Whiteboard / diagram / children macros | ❌ empty | — |

Where the native export is still useful (narrow):

1. Regression oracle — pull native Markdown as a second, independent signal in tests/adf-to-markdown.test.mjs and diff against our output to catch drift.
2. Fallback for legacy pages — pages that return no atlas_doc_format body (see src/frontend/confluence-download.mjs) can fall back to native Markdown instead of erroring.

It does not cover the reverse direction (Markdown → Confluence via markdown-to-adf + src/frontend/confluence-upload.mjs); that stays fully on us.