npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

chain-lexer

v1.10.0

Published

A lexical analyzer based on DFA that is built using JS and supports multi-language extensions

Downloads

182

Readme

1、Background

(1) Situation

Most lexical analyzers are closely coupled with the language, the amount of code is relatively large. It's hard to pay attention to the essential principles of lexical analyzer.

(2) Task

In order to focus on the working principle of lexical analyzer , not to consider the small differences caused by different languages , an idea of making a lexer project that is completely decoupled from the language was born.

(3) Solution

lexer through the following two files, realize the decoupling of lexical analyzer and language

  • src/lexer.js is the core part of lexical analyzer within 300 lines, including ISR and DFA
  • src/lang/{lang}-define.jsis the language extension of lexical analyzer. Support different languages,such as src/lang/c-define.js

2、Features

(1) Complete lexical analysis

From inputting the character sequence to generating token after the analysis, lexer has complete steps for lexical analysis, and 12 token types for most language extensions

(2) Support multi-language extension

lexer supports different language extensions such as Python, Go, etc. How to make different language extensions, please check Contributions

  • C :A popular programming language,click here to see its lexical analysis
  • SQL :A popular database query language,click here to see its lexical analysis
  • Goal :A goal parser problem from leetCode ,click here to see its lexical analysis

(3) Provide state flow log

The core mechanism of lexical analyzer is based on the state flow of DFA. For this reason, lexer records detailed state flow log to achieve the following requirements of you

  • Debug mode
  • Automatically generate DFA state flow diagram

3、Get project

After git clone command, no need for any dependencies, and no extra installation steps

4、Ussage

(1) In your project

If you need use lexer in your project, such as code editor, etc.

Using NPM

npm install chain-lexer
var chainLexer = require('chain-lexer');
let lexer = chainLexer.cLexer;

let stream = "int a = 10;";
lexer.start(stream);
let parsedTokens = lexer.DFA.result.tokens;

lexer = chainLexer.sqlLexer;
stream = "select * from test where id >= 10;";
lexer.start(stream);
parsedTokens = lexer.DFA.result.tokens;

Using Script

Import the package/{lang}-lexer.min.js file, then visit lexer variable to get the object of lexical analyzer,and visit lexer.DFA.result.tokens to get tokens

// 1. The code that needs lexical analysis
let stream = "int a = 10;";

// 2. Start lexical analysis
lexer.start(strem);

// 3. After the lexical analysis is done, get the generated tokens
let parsedTokens = lexer.DFA.result.tokens;

// 4. Do what you want to do
parsedTokens.forEach((token) => {
    // ... ...
});

The Provide state flow log part in features,visit flowModel.result.paths will get the detail logs of state flow inside lexer. The data format is as follows

[
    {
        state: 0, // now state
        ch: "a", // read char
        nextSstate: 2, // next state
        match: true, // is match
        end: false, // is last char
    },
    // ... ...
]

(2) Web preview and testing

In order to preview the process of lexer in real time, to debug and test, there is a index.html file in the root directory of this project. Open it directly in your browser, and after entering the code will automatically output the Token generated after lexer analysis, as shown in the figure below

int a = 10;
int b =20;
int c = 20;

float f = 928.2332;
char b = 'b';

if(a == b){
    printf("Hello, World!");
}else if(b!=c){
    printf("Hello, World! Hello, World!");
}else{
    printf("Hello!");
}

img

or check the online website

5、Contributions

(1) Project Statistics

(2) Source code explanation

Documents about source code development, project design, unit testing, automated testing, development specifications, and how to make extensions in different languages, please read source code explanation

(3) Content contribution

  • Add more new features
  • Add more extensions /src/lang/{lang}-define.js

(4) Release version

The project is released with the version number of A-B-C,regarding release log, you can check the CHANGELOG or the release record

  • A:Major upgrade
  • B:Minor upgrade
  • C:bug fix / features / ...

(5) Q&A

If you have any problems or questions, please submit an issue

6、License

GitHub