npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@skelogh/weaver

v1.1.2

Published

A mongoDB relational document importing tool

Downloads

7

Readme

Athenian weaver

Weaver

Weaver: Clone n-ary relationship sets across distinct databases.

NPM package: @skelogh/weaver

NodeJS 10 npm version codebeat badge Reviewed by Hound Coverage Status CircleCI

Context

Debugging DB objects with n-ary relationships is often done manually, this tool is meant to automate copying these objects into desired databases.

Weaver overview diagram

For example, a database has the following collections/documents:

  // db.users.findOne({_id: ObjectId('abcdef78901234abcdef1234')})
  {
    _id: ObjectId('abcdef78901234abcdef1234'),
    name: 'John',
    orders: [ { orderId: '4321fedcbafedcba67890123' } ]
  }

  // db.orders.findOne({_id: ObjectId('4321fedcbafedcba67890123')})
  {
    _id: ObjectId('4321fedcbafedcba67890123'),
    cartId: 'fedcba67890123fedcba4321'
  }

  // db.carts.findOne({_id: ObjectId('fedcba67890123fedcba4321')})
  {
    _id: ObjectId('fedcba67890123fedcba4321'),
    userId: 'abcdef78901234abcdef1234'
  }

Note how user relates to order, and order relates to cart.

If you wanted to migrate the orders and carts associated to the user to your local environment, you'd need to:

  1. Go to the [users|orders|carts] collection and find the document.
  2. Check if any of the fields is a reference to another collection.
  3. Copy the reference value.
  4. Repeat from step 1.

This tool finds the relationships, and migrates them to a destination db:

| Query | source db | target db | | -------------------------------------------------------- | ----------- | ----------- | | db.users.findOne(ObjectId('abcdef78901234abcdef1234')) | 1 | 1 | | db.orders.findOne(ObjectId('4321fedcbafedcba67890123'))| 1 | 1 | | db.carts.findOne(ObjectId('fedcba67890123fedcba4321')) | 1 | 1 |

Entity-relationship graph

Usage

Requirements

Getting started

Weaver runs on NodeJS >= 10

Install via npm

npm i -g @skelogh/weaver

How to test the CLI in dev mode

Install the CLI tool

  • After cloning the repo and installing dependencies, run npm run build.
  • Run npm pack on the project root.
  • A file weaver-<VERSION>.tgz will be created.
  • Run npm install -g weaver-<VERSION>.tgz, it will install the package globally.

Using the CLI tool

  • After installing, just issue the weaver command, the CLI help will display the CLI help:
Usage: weaver [OPTIONS] COMMAND [ARG...]
       weaver [ --help | -v | --version ]

Commands:
  weaver add        Creation of client, query or ignore
  weaver remove     Removal of clients, queries or ignores
  weaver run        Runs the app with the loaded configuration

Options:
  --version, -v    Print version information and quit
  --config, -c     Read or set path of config file, default: undefined
  --queries, --qq  Document ids to get relationships from, e.g.: 2a3b4c5d6e7f8g9h2a3b4c5d e7f8g9h2a3b4c5d2a3b4c5d6
  --verbose, -V    Enable highest level of logging, same as DEBUG=*
  --help           Show help

1. Add clients

In the CLI, run weaver add client, the following help will display:

Usage: weaver add client -fntou --options.<option_name>

  -f [mongodb]          <String> The client db family, mongodb is only supported for now.
  -n <name>             <String> The client db name.
  -t [source|target]    <String> source if the data will be pulled from it, target otherwise.
  -o [<source.name>]    <String> The target's origin db where the data will be copied from.
  -u <url>              <String> The client db URL.
  --options.<opt_name>  <Any> Client db-specific options, for now MongoClient options, use dot notation to set each option
                        Example: --options.readPreference secondaryPreferred

To pull data from a db into another, you'll need to add both the source and target clients. The source is where the document we want is stored, and target is where you want to copy to.

Example:

Assume the following remote server settings for source:

  • Family: mongodb
  • Type: source
  • IP: 172.17.0.4
  • Port: 27017
  • Database name: production
weaver add client -f mongodb -n production -t source -u mongodb://172.17.0.4:27017

Assume the following local server settings for target:

  • Family: mongodb
  • Type: target
  • IP: 127.0.0.1
  • Port: 27017
  • Database name: development
  • Origin: production (This is used to map where the data will be pulled from, useful when dealing with multiple source and target databases.)
weaver add client -f mongodb -n development -t target -o production -u mongodb://127.0.0.1:27017

2. Find the document to clone

From the source database, find the document you want to copy to the target database and copy the stringified _id:

// Assume the following mongodb document in the `production` database, `pets` collection:
{
    _id: ObjectId("507f1f77bcf86cd799439011"),
    name: "fluffy"
}

3. Clone the document(s)

Once steps 1 & 2 are done, just run the following command:

weaver run --queries 507f1f77bcf86cd799439011

4. See the results

If all is setup correctly, you should now have a copy of the document in your target database, it will create the collection even if it didn't exist before:

> mongodb
mongodb> use development
mongodb> db.pets.find(ObjectId("507f1f77bcf86cd799439011"))

{
    _id: ObjectId("507f1f77bcf86cd799439011"),
    name: "fluffy"
}

>

Roadmap

  • ~weaver CLI (follow the project)~
    • npm i -g @skelogh/weaver :tada:
  • Add plugins for different data sources.
  • Client UI
  • API/SDK (?)
  • Connect via SSH.
  • Doc pages

See what's been worked on.

Questions? create an issue!.