marked-token-position

Add position field for each token.

interface Position {
  /**
   * Positions for each line of the token. LinePositions will not include the newline character for the line.
   */
  lines: LinePosition[]
  /**
   * Position at the beginning of token
   */
  start: PositionFields;
  /**
   * Position at the end of token
   */
  end: PositionFields;
}

interface LinePosition {
  /**
   * Position at the beginning of line
   */
  start: PositionFields;
  /**
   * Position at the end of line. Will not include the newline character.
   */
  end: PositionFields;
}

interface PositionFields {
  /**
   * Number of characters from the beginning of the markdown string
   */
  offset: number;
  /**
   * Line number of the token. Starts at line 0.
   */
  line: number;
  /**
   * Column number of the token. Starts at column 0.
   */
  column: number;
}

Usage

Extension

import {Marked} from "marked";
import markedTokenPosition from "marked-token-position";

// or UMD script
// <script src="https://cdn.jsdelivr.net/npm/marked/lib/marked.umd.js"></script>
// <script src="https://cdn.jsdelivr.net/npm/marked-token-position/lib/index.umd.js"></script>
// const Marked = marked.Marked;

const marked = new Marked();

function anotherExtension {
	return {
		walkTokens(token) {
			// token has `position` field
		}
		hooks: {
			processAllTokens(tokens) {
				// tokens have `position` field
			}
		}
	};
}

marked.use(anotherExtension(), markedTokenPosition());

marked.parse("# example markdown");

The position field will be added to the tokens so any other extension can use the position field in a walkTokens function or processAllTokens hook.

[!CAUTION] The processAllTokens hook is used by this extension so any other extension using processAllTokens that requires the position field must be added before this extension because marked calls the processAllTokens hooks in reverse order.

The tokens will look like:

[
  {
    "type": "heading",
    "raw": "# example markdown",
    "depth": 1,
    "text": "example markdown",
    "tokens": [
      {
        "type": "text",
        "raw": "example markdown",
        "text": "example markdown",
        "escaped": false,
        "position": {
          "start": {
            "offset": 2,
            "line": 0,
            "column": 2
          },
          "end": {
            "offset": 18,
            "line": 0,
            "column": 18
          }
        }
      }
    ],
    "position": {
      "start": {
        "offset": 0,
        "line": 0,
        "column": 0
      },
      "end": {
        "offset": 18,
        "line": 0,
        "column": 18
      }
    }
  }
]

addTokenPositions

Calling marked.lexer() will not add the position field with the extension since the extension is only called on marked.parse() and marked.parseInline().

An addTokenPositions function is exported to add the position field to the tokens returned by marked.lexer().

import {Marked} from "marked";
import {addTokenPositions} from "marked-token-position";

// or UMD script
// <script src="https://cdn.jsdelivr.net/npm/marked/lib/marked.umd.js"></script>
// <script src="https://cdn.jsdelivr.net/npm/marked-token-position/lib/index.umd.js"></script>
// const Marked = marked.Marked;
// const addTokenPositions = markedTokenPosition.addTokenPositions;


const marked = new Marked();
const tokens = marked.lexer("# example markdown");

addTokenPositions(tokens);

// tokens now have a `position` field