mongo-schema-json

mongo-schema-json is a simple and efficient TypeScript utility that connects to a MongoDB database and retrieves the schema of all collections within it. The schema is returned as a JSON object, providing a clear and structured representation of your database's collections and their fields.

Installation

Use it as a library inside your project:

npm install mongo-schema-json

Or as a command line tool, with no install needed:

npx mongo-schema-json --uri mongodb://localhost:27017 --db mflix
# or install globally
npm install -g mongo-schema-json

Requires Node.js >= 16.20.1. TypeScript types are included.

Usage in your code

Here's a basic example of how to use mongo-schema-json to retrieve and log the schema of a MongoDB database:

import { getDbSchema } from 'mongo-schema-json';
// CommonJS: const { getDbSchema } = require('mongo-schema-json');

(async () => {
  const { schema, collections } = await getDbSchema({
    uri: 'mongodb://localhost:27017',
    dbName: 'mflix',
  });

  for (const collection of collections) {
    console.log('--------------------------------------------');
    console.log('Schema for', collection);
    console.log(schema[collection]);
    console.log('--------------------------------------------');
  }
})();

The connection opened by getDbSchema is closed automatically when it finishes.

Reuse your app's existing connection

If your app is already connected, pass the client or db instead of a uri. The package will not close connections that you own.

import { MongoClient } from 'mongodb';
import { getDbSchema } from 'mongo-schema-json';

const client = await MongoClient.connect(process.env.MONGODB_URI!);

await getDbSchema({ client, dbName: 'mflix' });
await getDbSchema({ db: client.db('mflix'), collections: ['users', 'movies'] });

Accurate schemas for NoSQL data (sampling)

In MongoDB, documents in the same collection often have different fields. mongo-schema-json analyses 100 random documents per collection by default and merges all of their fields, so fields that only exist in some documents are still found. When a field has different types in different documents, you get a union such as 'string | null'.

// Analyse more documents for better coverage
await getDbSchema({ uri, dbName: 'mflix', sampleSize: 1000 });

// Scan every document (most accurate, slower on big collections)
await getDbSchema({ uri, dbName: 'mflix', sampleSize: 'all' });

// Pick documents differently: 'random' (default) | 'first' | 'latest'
await getDbSchema({ uri, dbName: 'mflix', sampleStrategy: 'latest' });

• random uses MongoDB $sample, which gives the best coverage across the whole collection.
• latest reads the newest documents (_id descending), which is useful when your schema changed recently.
• first reads documents in natural order and gives the same result on every run. With sampleSize: 1 it behaves like 1.0.0.

The result also includes sampled, the number of documents analysed per collection:

const { sampled } = await getDbSchema({ uri, dbName: 'mflix' });
// { movies: 100, users: 100, comments: 100, theaters: 100 }

Example on the mflix sample database: with a single document, movies misses languages, writers, poster and metacritic. The default of 100 random documents finds them all. users.preferences exists in only 1 of 185 documents, so it is only found with sampleSize: 'all'. Fields that rare need a full scan.

Infer a schema from documents you already have

inferSchema is a pure function with no database access:

import { inferSchema } from 'mongo-schema-json';

inferSchema([{ name: 'a', tags: ['x'] }, { name: 'b', age: 3 }]);
// { name: 'string', tags: 'string[]', age: 'number' }

Usage from the command line

mongo-schema-json --uri <mongodb-uri> [options]

  -u, --uri <uri>            MongoDB connection string (or env MONGODB_URI)
  -d, --db <name>            Database name (defaults to the db in the URI)
  -c, --collections <list>   Comma separated list of collections to inspect
  -s, --sample-size <n|all>  Documents analysed per collection (default: 100, "all" scans everything)
      --strategy <name>      How documents are picked: random | first | latest (default: random)
  -o, --out <file>           Write JSON to a file instead of stdout
      --compact              Print compact JSON (no indentation)
  -h, --help                 Show help
  -v, --version              Show version

Example: write the schema of the mflix database to schema.json, analysing 500 documents per collection:

npx mongo-schema-json -u mongodb://localhost:27017 -d mflix -s 500 -o schema.json

Sample Output

The following is a sample output for the database mflix with the default settings (100 random documents per collection). Fields such as languages, writers, poster, metacritic and most of tomatoes are missing from the first document and only show up through sampling:

--------------------------------------------
Schema for theaters
{
  _id: 'string',
  theaterId: 'number',
  location: {
    address: {
      street1: 'string',
      city: 'string',
      state: 'string',
      zipcode: 'string'
    },
    geo: { type: 'string', coordinates: 'number[]' }
  }
}
--------------------------------------------
--------------------------------------------
Schema for comments
{
  _id: 'string',
  name: 'string',
  email: 'string',
  movie_id: 'string',
  text: 'string',
  date: 'Date'
}
--------------------------------------------
--------------------------------------------
Schema for movies
{
  _id: 'string',
  plot: 'string',
  genres: 'string[]',
  runtime: 'number',
  cast: 'string[]',
  num_mflix_comments: 'number',
  title: 'string',
  fullplot: 'string',
  countries: 'string[]',
  released: 'Date',
  directors: 'string[]',
  rated: 'string',
  awards: { wins: 'number', nominations: 'number', text: 'string' },
  lastupdated: 'string',
  year: 'number',
  imdb: { rating: 'number', votes: 'number', id: 'number' },
  type: 'string',
  languages: 'string[]',
  writers: 'string[]',
  poster: 'string',
  metacritic: 'number',
  tomatoes: {
    viewer: { rating: 'number', numReviews: 'number', meter: 'number' },
    critic: { rating: 'number', numReviews: 'number', meter: 'number' },
    lastUpdated: 'Date',
    dvd: 'Date',
    consensus: 'string',
    rotten: 'number',
    fresh: 'number',
    production: 'string',
    website: 'string',
    boxOffice: 'string'
  }
}
--------------------------------------------
--------------------------------------------
Schema for users
{ _id: 'string', name: 'string', email: 'string', password: 'string' }
--------------------------------------------

API

getDbSchema(options): Promise<{ schema, collections, sampled }>

| Option | Type | Description | | --- | --- | --- | | uri | string | MongoDB connection string. Required unless client or db is given. | | dbName | string | Database name. Defaults to the database in the URI. | | client | MongoClient | Reuse an existing connected client (not closed by this package). | | db | Db | Reuse an existing Db instance (not closed by this package). | | collections | string[] | Only inspect these collections. Default: all collections (views and system.* are skipped). | | sampleSize | number \| 'all' | Documents analysed per collection. 'all' scans every document. Default: 100. | | sampleStrategy | 'random' \| 'first' \| 'latest' | How documents are picked. Default: 'random'. | | mongoClientOptions | MongoClientOptions | Extra options for new MongoClient() when uri is used. |

Returns

• schema: An object where the keys are collection names and the values are JSON representations of the schema for each collection. Empty collections are listed in collections but have no entry in schema.
• collections: An array of collection names that were inspected.
• sampled: Number of documents analysed per collection (0 for empty collections).

inferSchema(docs): IDataType

Builds a schema from a document or an array of documents.

Type mapping

| BSON / JS value | Schema type | | --- | --- | | ObjectId | 'string' | | Date | 'Date' | | Int32, Long, Double, Decimal128, number | 'number' | | Binary, Buffer | 'Buffer' | | null | 'null' | | array of primitives | 'string[]', 'number[]', '(number \| string)[]', ... | | empty array | 'unknown[]' | | array of documents | [{ ...schema }] | | sub-document | { ...schema } |

Exported TypeScript types: IDataType, IFieldType, IGetDbSchemaOptions, IMongoSchemaResponse, ISampleStrategy.

Changelog

1.1.0

• New: analyses 100 random documents per collection by default (was 1) and merges their fields, so fields that only exist in some documents are found.
• New: sampleSize: 'all' for a full scan, and sampleStrategy ('random', 'first', 'latest').
• New: sampled in the result, with the number of documents analysed per collection.
• New: CLI (npx mongo-schema-json) with --sample-size and --strategy.
• New: sampleSize, collections, client, db and mongoClientOptions options.
• New: inferSchema() export and exported TypeScript types.
• Fix: published type declarations referenced a missing types file.
• Fix: calling getDbSchema with a different uri/dbName returned the first database (the connection was cached globally).
• Fix: the connection is now closed after use, so scripts exit on their own.
• Fix: null field values no longer throw. Empty arrays are reported as 'unknown[]'.

Development

npm install
npm test          # builds and runs the tests (integration tests need MongoDB at MONGODB_URI or localhost:27017)
npm run lint

Contributing

Contributions are welcome! Please submit a pull request or open an issue to discuss your ideas.

License

This project is licensed under the MIT License.