npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@rodrigogs/zongji

v0.4.14

Published

A mysql binlog listener running on Node.js

Downloads

4,056

Readme

ZongJi Build Status

A MySQL binlog listener running on Node.js.

ZongJi (踪迹) is pronounced as zōng jì in Chinese.

This package is a "pure JS" implementation based on node-mysql. Since v0.2.0, The native part (which was written in C++) has been dropped.

This package has been tested to work in MySQL 5.5, 5.6, and 5.7.

Quick Start

var zongji = new ZongJi({ /* ... MySQL Connection Settings ... */ });

// Each change to the replication log results in an event
zongji.on('binlog', function(evt) {
  evt.dump();
});

// Binlog must be started, optionally pass in filters
zongji.start({
  includeEvents: ['tablemap', 'writerows', 'updaterows', 'deleterows']
});

For a complete implementation see example.js...

Installation

  • Requires Node.js v4+

    $ npm install zongji
  • Enable MySQL binlog in my.cnf, restart MySQL server after making the changes.

    From MySQL 5.6, binlog checksum is enabled by default. Zongji can work with it, but it doesn't really verify it.

    # Must be unique integer from 1-2^32
    server-id        = 1
    # Row format required for ZongJi
    binlog_format    = row
    # Directory must exist. This path works for Linux. Other OS may require
    #   different path.
    log_bin          = /var/log/mysql/mysql-bin.log
    
    binlog_do_db     = employees   # Optional, limit which databases to log
    expire_logs_days = 10          # Optional, purge old logs
    max_binlog_size  = 100M        # Optional, limit log size
  • Create an account with replication privileges, e.g. given privileges to account zongji (or any account that you use to read binary logs)

    GRANT REPLICATION SLAVE, REPLICATION CLIENT, SELECT ON *.* TO 'zongji'@'localhost'

ZongJi Class

The ZongJi constructor accepts one argument of either:

  • An object containing MySQL connection details in the same format as used by node-mysql
  • Or, a node-mysql Connection or Pool object that will be used for querying column information.

If a Connection or Pool object is passed to the constructor, it will not be destroyed/ended by Zongji's stop() method.

If there is a dateStrings node-mysql configuration option in the connection details or connection, ZongJi will follow it.

Each instance includes the following methods:

Method Name | Arguments | Description ------------|-----------|------------------------ start | options | Start receiving replication events, see options listed below stop | None | Disconnect from MySQL server, stop receiving events set | options | Change options after start() on | eventName, handler | Add a listener to the binlog or error event. Each handler function accepts one argument.

Options available:

Option Name | Type | Description ------------|------|------------------------------- serverId | integer | Unique number (1 - 232) to identify this replication slave instance. Must be specified if running more than one instance of ZongJi. Must be used in start() method for effect.Default: 1 startAtEnd | boolean | Pass true to only emit binlog events that occur after ZongJi's instantiation. Must be used in start() method for effect.Default: false binlogName | string | Begin reading events from this binlog file. If specified together with binlogNextPos, will take precedence over startAtEnd. binlogNextPos | integer | Begin reading events from this position. Must be included with binlogName. includeEvents | [string] | Array of event names to includeExample: ['writerows', 'updaterows', 'deleterows'] excludeEvents | [string] | Array of event names to excludeExample: ['rotate', 'tablemap'] includeSchema | object | Object describing which databases and tables to include (Only for row events). Use database names as the key and pass an array of table names or true (for the entire database).Example: { 'my_database': ['allow_table', 'another_table'], 'another_db': true } excludeSchema | object | Object describing which databases and tables to exclude (Same format as includeSchema)Example: { 'other_db': ['disallowed_table'], 'ex_db': true }

  • By default, all events and schema are emitted.
  • excludeSchema and excludeEvents take precedence over includeSchema and includeEvents, respectively.

Supported Binlog Events:

Event name | Description ------------|--------------- unknown | Catch any other events query | Insert/Update/Delete Query intvar | Autoincrement and LAST_INSERT_ID rotate | New Binlog file Not required to be included to rotate to new files, but it is required to be included in order to keep the binlogName and binlogNextPos properties updated with current values for graceful restarting on errors. format | Format Description xid | Transaction ID tablemap | Before any row event (must be included for any other row events) writerows | Rows inserted, row data array available as rows property on event object updaterows | Rows changed, row data array available as rows property on event object deleterows | Rows deleted, row data array available as rows property on event object

Event Methods

Neither method requires any arguments.

Name | Description -------|--------------------------- dump | Log a description of the event to the console getEventName | Return the name of the event

Important Notes

  • :star2: All types allowed by node-mysql are supported by this package.
  • :speak_no_evil: While 64-bit integers in MySQL (BIGINT type) allow values in the range of 264 (± ½ × 264 for signed values), Javascript's internal storage of numbers limits values to 253, making the allowed range of BIGINT fields only -9007199254740992 to 9007199254740992. Unsigned 64-bit integers must also not exceed 9007199254740992.
  • :point_right: TRUNCATE statement does not cause corresponding DeleteRows event. Use unqualified DELETE FROM for same effect.
  • When using fractional seconds with DATETIME and TIMESTAMP data types in MySQL > 5.6.4, only millisecond precision is available due to the limit of Javascript's Date object.

Run Tests

  • install Docker
  • run docker-compose up and then ./docker-test.sh

Reference

I learnt many things from following resources while making ZongJi.

  • https://github.com/felixge/node-mysql
  • https://github.com/felixge/faster-than-c/
  • http://intuitive-search.blogspot.co.uk/2011/07/binary-log-api-and-replication-listener.html
  • https://github.com/Sannis/node-mysql-libmysqlclient
  • https://kkaefer.com/node-cpp-modules/
  • http://dev.mysql.com/doc/internals/en/replication-protocol.html
  • http://www.cs.wichita.edu/~chang/lecture/cs742/program/how-mysql-c-api.html
  • https://github.com/jeremycole/mysql_binlog (Ruby implemenation of MySQL binlog parser)
  • http://dev.mysql.com/doc/internals/en/date-and-time-data-type-representation.html

License

MIT