docmark-util-subtokenize

docmark utility to tokenize subtokens.

Contents

• What is this?
• When should I use this?
• Install
• Use
• API
  • subtokenize(events)
  • subcontent(events, eventIndex)
• Types
• Contribute

What is this?

This packages exposes utilities to tokenize embedded content.

When should I use this?

This package is useful when building ecosystems on top of docmark.

Install

This package is ESM only.

In Node.js with yarn:

yarn add @flex-development/docmark-util-subtokenize

In Deno with esm.sh:

import { subcontent, subtokenize } from 'https://esm.sh/@flex-development/docmark-util-subtokenize'

In browsers with esm.sh:

<script type="module">
  import { subcontent, subtokenize } from 'https://esm.sh/@flex-development/docmark-util-subtokenize'
</script>

Use

TODO: use

API

This package exports the identifiers subtokenize and subcontent.
There is no default export.

subtokenize(events)

Tokenize embedded content.

Some tokens declare a ContentType.
These tokens do not contain fully parsed content themselves. Tokens with a chunk* (i.e. chunkMarkdown, chunkDocument, chunkFlow) type and contentType act as containers for another tokenizer.

For example:

comment
└─ chunkComment

A chunkComment token may contain embedded syntax.
This function replaces those chunk tokens with the events produced by their child tokenizer.

Parameters

• events (Event[]) — the current list of events

Returns

(boolean) Whether subtokens (embedded content) were found

subcontent(events, eventIndex)

Tokenize embedded content for a single token.

The algorithm has three phases:

1. Feed linked chunk tokens to a child tokenizer
2. Determine which child events belong to each linked token
3. Replace chunk* events with their corresponding child events

Parameters

• events (Event[]) — the parent event stream
• eventIndex (number) — the index of the corresponding enter event in events

Returns

(undefined) Nothing

Types

This package is fully typed with TypeScript.
It exports no additional types.

Contribute

See CONTRIBUTING.md.

This project has a code of conduct. By interacting with this repository, organization, or community you agree to abide by its terms.