mpeg-sdl-parser

ISO/IEC 14496-34 Syntactic Description Language (MPEG SDL) parser implemented in TypeScript

Web Editor

A browser based web editor using this module is available at https://github.com/MPEGGroup/mpeg-sdl-editor

Module Usage

Installation

NPM

Add the module to your project:

npm install @mpeggroup/mpeg-sdl-parser

BUN

Add the module to your project:

bun add @mpeggroup/mpeg-sdl-parser

Example Code

import {
  SdlStringInput,
  createLenientSdlParser,
  collateSyntaxErrors,
  buildAst,
  dispatchNodeHandler,
  prettyPrint
} from "@mpeggroup/mpeg-sdl-parser";

// Create a Lezer based SDL concrete syntax parser.
// This will create a lenient parser which recovers from parse errors and places error nodes in the parse tree.
// A strict parser which will throw a `SyntaxError` can be created with `createStrictSdlParser()`.
const parser = await createLenientSdlParser();

// Prepare the SDL input
const sdlStringInput = new SdlStringInput("class A{}");

// Parse SDL input and produce a parse tree
const sdlParseTree = sdlParser.parse(sdlStringInput);

// Traverse and print the parse tree
const cursor = sdlParseTree.cursor();
do {
  console.log(`Node ${cursor.name} from ${cursor.from} to ${cursor.to}`)
} while (cursor.next());

// Print any syntax errors by collating any error nodes in the parse tree
const syntaxErrors = collateSyntaxErrors(sdlParseTree, sdlStringInput);

console.log(JSON.stringify(syntaxErrors);

// Build an Abstract Syntax Tree (AST) of the SDL specification from the concrete parse tree
// The third argument sets `lenient = true`. If this is not set or is `false`
// a `SyntaxError` will be thrown if the sdlParseTree contains parsing errors.
const specification = buildAst(sdlParseTree, sdlStringInput, true);

// Define a simple AST Node handler
class MyNodeHandler implements NodeHandler {
  beforeVisit(node: AbstractCompositeNode) {
    console.log("About to visit child nodes");
  }

  visit(node: AbstractLeafNode) {
    console.log("Visiting leaf node");
  }

  afterVisit(node: AbstractCompositeNode) {
    console.log("Finished visiting child nodes");
  }
}

// Dispatch the handler to visit all nodes in the AST
dispatchNodeHandler(specification, new MyNodeHandler());

// Create an SDL semantic analyser
// This will create a lenient analyser which recovers from semantic errors and stores them in the analysis results.
// A strict analyser which will throw a `SemanticError` can be created with `createStrictSdlAnalyser()`.
const analyser = createLenientSdlAnalyser();

// Analyse the AST
const analysisResult = analyser.analyse(specification);

// Print any semantic errors from the AST analysis
console.log(JSON.stringify(analysisResult.semanticErrors));

// Print any semantic warnings from the AST analysis
console.log(JSON.stringify(analysisResult.semanticWarnings));

// Pretty print the specification (retaining comments and handling parse errors)
let prettifiedSpecification = await prettyPrint(specification, sdlStringInput)

console.log(prettifiedSpecification);

// A Prettier (prettier.io) plugin for SDL is also available:
import * as prettier from "prettier/standalone.js";
import { prettierPluginSdl } from "@mpeggroup/mpeg-sdl-parser"; 

prettifiedSpecification = await prettier.format("class A{}", { 
  parser: "sdl",
  plugins: [prettierPluginSdl],
});

console.log(prettifiedSpecification);

Development

Install dependencies:

bun install

Generate parser from grammar:

bun run generate

Test:

bun test

NOTE: The following tasks use Deno as it excels at these and Bun does not currently provide such functionality:

Format:

deno fmt

Lint:

deno lint index.ts src/ tests/

Generate HTML API Documentation:

deno doc --html --name=mpeg-sdl-parser index.ts

Documentation

Overview

The concrete syntax parser is implemented using Lezer and the Lezer grammar defined in sdl.lezer.grammar. This framework was chosen as it:

• Provides robust error recovery whilst parsing.
• Integrates well with the web based code editor framework Codemirror which is used in the MPEG SDL Editor.

For reference purposes an SDL EBNF grammar is also provided in sdl.ebnf.grammar

The concrete syntax tree is then converted to an Abstract Syntax Tree (AST) which is not tied to any underlying framework. The AST representation is used for pretty printing and as such retains a "full fidelity" represenation of the source i.e. including text location and trivia items (comments and blank lines).

Abstract Syntax Tree Model

classDiagram

  class Trivia {
    <<interface>> 
    text: string;
  }

  class Location {
    <<interface>> 
    row: number
    column: number
    position: number
  }

  class AbstractNode {
    <<Abstract>>
    nodeKind: NodeKind
    isCompositeNode: boolean
  }

  class AbstractLeafNode {
    <<Abstract>>
    isCompositeNode = false
  }

  class AbstractCompositeNode {
    <<Abstract>>
    isCompositeNode = true
  }

  class Token {
    text: string;
    tokenKind: TokenKind;
  }

  class TraversingVisitor {
  }

  class NodeVisitor {
    <<interface>> 
    visit(node: AbstractNode)
  }

  class NodeHandler {
    <<interface>> 
    beforeVisit(node: AbstractCompositeNode)
    visit(node: AbstractLeafNode)
    afterVisit(node: AbstractCompositeNode)
  }

  Trivia --> Location : location

  NodeHandler ..> AbstractCompositeNode : afterVisit
  NodeHandler ..> AbstractCompositeNode : beforeVisit
  NodeHandler ..> AbstractLeafNode : visit

  AbstractNode --> "*" Trivia : leadingTrivia
  AbstractNode --> "*" Trivia : trailingTrivia

  AbstractLeafNode --|> AbstractNode

  AbstractCompositeNode --|> AbstractNode
  AbstractCompositeNode --> "*" AbstractNode : children
  AbstractCompositeNode --> "0..1" Token : startToken
  AbstractCompositeNode --> "0..1" Token : endToken

  Token --|> AbstractLeafNode
  Token --> Location

  NodeVisitor ..> AbstractNode : visit
  TraversingVisitor --|> NodeVisitor
  TraversingVisitor --> NodeHandler

  CompositeNodeXYZ "*" --|> AbstractCompositeNode
  Specification --|> AbstractCompositeNode

Semantic Analyser Model

classDiagram
    class SdlAnalyser {
        configure(options) SdlAnalyser
        analyse(specification) SdlAnalysisResult
    }

    class SdlAnalysisResult {
        <<interface>>
        SemanticError[] semanticErrors
        SemanticWarning[] semanticWarnings
    }

    class SymbolTable {
        addSymbol()
        lookupVariable()
        lookupClass()
        lookupMap()
        enterClassScope()
        enterBlockScope()
        exitScope()
    }

    class AbstractAnalysisNodeHandler {
        <<abstract>>
    }

    class BuildSymbolTableNodeHandler {
    }

    class ValidateScopeNodeHandler {
    }

    class ValidateTypeNodeHandler {
    }

    class SpecificCheckNodeHandler {
    }

    class Check {
        <<interface>>
        string nodeKind
        string subKind
        checkFunc()
    }

    class Specification {
    }

    SdlAnalyser --> SdlAnalysisResult : produces
    SdlAnalyser --> Specification : analyses
    SdlAnalysisResult --> SymbolTable
    SdlAnalysisResult --> Specification
    AbstractAnalysisNodeHandler --> SymbolTable
    BuildSymbolTableNodeHandler --|> AbstractAnalysisNodeHandler
    ValidateScopeNodeHandler --|> AbstractAnalysisNodeHandler
    ValidateTypeNodeHandler --|> AbstractAnalysisNodeHandler
    SpecificCheckNodeHandler --|> AbstractAnalysisNodeHandler
    SpecificCheckNodeHandler --> Check
    SdlAnalyser ..> BuildSymbolTableNodeHandler : pass 1
    SdlAnalyser ..> ValidateScopeNodeHandler : pass 2
    SdlAnalyser ..> ValidateTypeNodeHandler : pass 3
    SdlAnalyser ..> SpecificCheckNodeHandler : pass 4

API

Link to auto-generated API docs:

API Documentation

Debug Logging

Internal framework logging can be enabled by setting the MPEG_SDL_PARSER_DEBUG environment variable. If running in a browser open the developer console and run:

localStorage.setItem("MPEG_SDL_PARSER_DEBUG", "true")

The logging implementation will look for an object conforming to the Logger interface and use it if found. If not found, a simple logging implementation using the console object will be used.

License

MIT © Flowscripter