中文文档

Defines the shape of Yozora tokenizers and their lifecycle methods, as well as some utility functions to assist in resolving tokens.

Install

• npm

  npm install --save @yozora/core-tokenizer

Usage

According to the parse strategy, there are two types of tokenizers: Block Tokenizer and Inline Tokenizer.

Block Tokenizer

The parsing steps of the block tokenizer are divided into two life cycles:

• match-block: Match a block node and get a BlockToken.

• parse-block: Parse BlockToken lists into Yozora AST nodes.

match-block phase

In the process of parsing block nodes, the content is read line by line. Block-level nodes have a nested structure:

> This is a blockquote
> - This is a list item in blockquote
> - # This is a setext heading in the list item of the blockquote
> - > ...

As shown in the second line above, when parsing ListItem, it cannot get the first character in the original document line directly, but waits for its ancestor elements along the existing nested structure, such as the Blockquote, to complete matching before it gets an opportunity to match. To make tokenizers work with each other transparently, the nested-structure parsing logic of the match-block phase is lifted into @yozora/core-parser, using IPhrasingContentLine as the actual parsing unit of a line:

export interface IPhrasingContentLine extends INodeInterval {
  /**
   * Array of INodePoint which contains all the contents of this line.
   */
  nodePoints: readonly INodePoint[]
  /**
   * The index of first non-blank character in the rest of the current line.
   */
  firstNonWhitespaceIndex: number
  /**
   * Visual width of the preceding indentation.
   */
  indentWidth: number
}

The lifecycle methods at this stage are subdivided into the following methods (see match-block for the complete type definitions):

• eatOpener: (Required) Try to match a new block node.

• eatAndInterruptPreviousSibling: (Optional) Try to interrupt the previous sibling node and match a new block node.

• eatContinuationText: (Optional) Try to match the continuation text of the current block node; that is, consume the current IPhrasingContentLine with the current block node. There may be many kinds of results at this stage, distinguished by the returned status:

  • notMatched: Not matched.

  • closing: Matched, and this is the last line of the current block node. The current node is saturated and will close.

  • opening: Matched, and the current block node remains open.

  • failedAndRollback: Matching failed, and the content of the previous lines must be rolled back. It is assumed that rollback does not affect the previously satisfied nested structure.

  • closingAndRollback: Matching failed, but only the returned lines need to be rolled back. The current node remains valid and will close.

• eatLazyContinuationText: (Optional) Try to match lazy continuation text. The paragraph and table tokenizers currently implement this method. See https://github.github.com/gfm/#phase-1-block-structure step 3 for details.

• onClose: (Optional) Called before the current node closes to perform cleanup.

• extractPhrasingContentLines: (Optional) Convert a Block Token generated by the current tokenizer to IPhrasingContentLine[]. Override this method when matching this node type may require rollback.

• buildBlockToken: (Optional) Convert IPhrasingContentLine[] into a Block Token. Override this method when matching this node type may require rollback.

parse-block phase

This phase contains the following lifecycle method (see parse-block for the complete type definitions):

• parse: Convert a list of Block Tokens into Yozora AST nodes.

Inline Tokenizer

The parsing steps of the inline tokenizer are divided into two life cycles:

• match-inline: Match inline content and get an InlineToken.
• parse-inline: Parse an InlineToken into a Yozora AST node.

match-inline phase

After a block node closes, matching inline nodes can begin, so inline matching receives continuous text without the concept of lines. Inline nodes also have priorities; for example, links have a higher priority than emphasis (see https://github.github.com/gfm/#example-529). To make tokenizers work with each other transparently, priority-related logic is handled in @yozora/core-parser. Each tokenizer provides four types of delimiters: opener, both, closer, and full. The processor in @yozora/core-parser completes the coordination work.

The lifecycle methods at this stage are subdivided into the following methods (see match-inline for the complete type definitions):

• findDelimiter: (Required) Find delimiters.
• isDelimiterPair: (Optional) Check whether the given two delimiters can pair.
• processDelimiterPair: (Optional) Process a delimiter pair, as in @yozora/tokenizer-emphasis.
• processSingleDelimiter: (Optional) Process a single delimiter, as in @yozora/tokenizer-text.

parse-inline phase

This phase contains the following lifecycle method (see parse-inline for the complete type definitions):

• parse: Convert a list of Inline Tokens into Yozora AST nodes.

Related

• homepage

• Block Tokenizer Lifecycle

  • match-block
  • parse-block

• Inline Tokenizer Lifecycle

  • match-inline
  • parse-inline