@sweepbright/sbimport
v1.1.0
Published
The dataset import tool that interfaces with the Properties API and the Contacts API
Downloads
27
Readme
sbimport
Installation
This tool is built using Node.js. You can install it globally using npm
or yarn
as follows:
For npm
npm install -g @sweepbright/sbimport
For yarn
yarn global add @sweepbright/sbimport
Once the process terminates, you can start using it. Let's try
sbimport --help
Another approach is to use npx
to immediately use the tool.
npx @sweepbright/sbimport --help
Usage
Currently, sbimport
support two commands:
init
: to initialise the import folderimport
: to import the datasets in an existing import folder.
Let's see in more details what this commands do.
init
command
The init
command expect a folder name as an argument. When executed, it will ask you the API Key and the environment. Provide them and it’s done!
You can now inspect the folder the tools created. If you have tree
installed you should get a similar output.
sbimport init ImportCustomerX
tree ImportCustomerX
ImportCustomerX
├── company.config.yaml
├── contacts
├── logs
└── properties
└── files
└── README.txt
5 directories, 2 files
The folder is organised by entity that can be imported. Currently, we support import of properties (properties
subfolder) and contacts (contacts
subfolder).
We also have:
company.config.yaml
: the configuration file that contains the information specified during the init (we reserve the possibility to expand this in the future with more options).logs
: The folder where all the logs will be persisted.
import
command
The import
command expect a folder name as an argument and support a few options. To have a full overview of the available option run
sbimport import --help
The import
command will attempt to import every record it finds in the import folder provided as an argument. These behaviour can be changed by using specific options. Having the possibility to select what to import is useful when you just need to import specific records.
The sbimport
keeps track of what has been process so that consecutive execution of the same dataset will not generate unnecessary import operations. It does that by storing an hash for each record and asset in the datasets and comparing it during subsequent executions. When the comparison fail we consider the record to be imported.
Some options are:
--entities
: Allow to import just the selected entity type.--force
: By default thesbimport
remembers what was processed to avoid importing the same data. To prevent that use this option.
Examples
To import everything in the import folder test
sbimport import test
To import just properties datasets in the import folder test
sbimport import test --entities property
To import properties datasets and relative images in the import folder test
sbimport import test --entities property --entities propertyAssetTo import just a specific contacts datasets between the multiple ones in the import folder `test`
Preparing the datasets
The sbimport
tool expect data in JSONL format. In short, files using JSONL expect to have a valid JSON object per line.
To convert a normal JSON file to the JSONL version that the tool expect, you could use jq as follows:
jq -c '.[]' < dataset.json > dataset.jsonl
Once your datasets are ready, move them to the respective folders:
- Properties datasets in
properties
folder - Contacts datasets in the
contacts
folder.
Preparing properties files
To support SweepBright properties file options, we decided to organise the import folder properties files using the following hierarchy (it’s also described in the Readme file inside the files
folder)
This directory is used to store the properties files.
The structure is as follows:
{property-reference}/
├── documents/
│ ├── private/
│ └── public/
├── images
│ ├── private/
│ └── public/
└── plans
├── private/
└── public/
Replace {property-reference} with your property ID.
For example, assuming you are importing one property with ID a3155152-3cb3-4878-b1e6-39466844328c
, and this property has:
doc1.pdf
public anddoc2.pdf
privateimg1.png
public andimg2.png
private- and no plans
The files
folder should look like that
files
└── a3155152-3cb3-4878-b1e6-39466844328c
├── documents
│ ├── private
│ │ └── doc2.pdf
│ └── public
│ └── doc1.pdf
├── images
│ ├── private
│ │ └── img2.png
│ └── public
│ └── img1.png
└── plans
├── private
└── public