darp² (Distributed Automated Remote Processing Pipeline)

darp² is an experimental tool for distributed automated remote processing pipelines for point clouds to be used in conjunction with DAT. Its purpose is to allow processing of data, that is remotely stored in a distributed network without the need to create a local copy of the data first. Instead only necessary data is requested from its peers The data is streamed through a processing pipeline, that writes the result into a directory, that is immediately available as a Dat archive again.

darp² can be used as a CLI (Command Line Interface; aka. terminal) tool or as a Node.js library.

CLI usage

darp² is distributed using npm, the package manager for Node.js. Please install the latest version of Node.js from the official homepage.

To install darp² you then type the following command:

$ npm install darp2 --global

Once that succeeds the darp2 command should be available inside your terminal.

Options

$ darp2 --help

darp2 -i dat://... -g "**/*" -c "ls -lash \"\${DARP2_TEMP_FOLDER}\" > ls.out"

Options:
  --version          Show version number                                                   [boolean]
  --input, -i        Input DAT key                                               [string] [required]
  --run-script, -r   Script to run on the task.                                             [string]
  --cmd, -c          Command to run on after downloading the dat.                           [string]
  --work-folder, -w  Folder used for the output. Passed as `DARP2_WORK_FOLDER` and
                     `DARP2_UPLOAD_FOLDER` env variable to the cmd.                         [string]
  --temp-folder, -t  Folder used to download the dat. Passed as `DARP2_TEMP_FOLDER` and
                     `DARP2_DOWNLOAD_FOLDER` env variable to the cmd.                       [string]
  --nolog            Dont write the log file to the out dat.              [boolean] [default: false]
  --log-path         File to write the log into.              [string] [default: ".dat-through.log"]
  --glob, -g                                                              [string] [default: "**/*"]
  --glob-nobrace     Do not expand `{a,b}` and `{1..3}` brace sets.       [boolean] [default: false]
  --glob-noglobstar  Disable `**` matching against multiple folder names. [boolean] [default: false]
  --glob-dot         Allow patterns to match filenames starting with a period, even if
                     the pattern does not explicitly have a period in that spot.

                     Note that by default, `a/**/b` will **not** match `a/.d/b`, unless `dot`
                     is set.                                              [boolean] [default: false]
  --glob-noext       Disable "extglob" style patterns like `+(a|b)`.      [boolean] [default: false]
  --glob-nocase      Perform a case-insensitive match.                    [boolean] [default: false]
  --glob-nonull      When a match is not found by `minimatch.match`, return a list containing
                     the pattern itself if this option is set.  When not set, an empty list
                     is returned if there are no matches.                 [boolean] [default: false]
  --glob-matchBase   If set, then patterns without slashes will be matched
                     against the basename of the path if it contains slashes.  For example,
                     `a?b` would match the path `/xyz/123/acb`, but not `/xyz/acb/123`.
                                                                          [boolean] [default: false]
  --glob-nocomment   Suppress the behavior of treating `#` at the start of a pattern as a
                     comment.                                             [boolean] [default: false]
  --glob-nonegate    Suppress the behavior of treating a leading `!` character as negation.
                                                                          [boolean] [default: false]
  --glob-flipNegate  Returns from negate expressions the same as if they were not negated. (Ie, true
                     on a hit, false on a miss.)                          [boolean] [default: false]
  --concurrency, -p  Specifies how many files and folders should be max. processed in parallel.
                                                                               [number] [default: 1]
  --help             Show help                                                             [boolean]

PDAL Example

This example shows how darp² can take a dat link and then run a processing pdal processing pipeline to pointcloud files and share the output of the processing pipeline with a new dat link:

$ darp2 \
  --input dat://74117a268a95438ceabcde6541dc9f5f00f5c21f192bd14f17258e5ca9ff194b \
  --cmd "pdal pipeline $(pwd)/processing_pipeline.json --readers.las.filename=\"\${DARP2_TEMP_FOLDER}/lidar_raleigh_nc_spm_height_feet.laz\""

This example specifies the input as the dat available under the dat://74117a..b link. The dat link should start with dat:// followed by the 64-character hex string.

It runs a pdal processing pipeline that is defined by the processing_pipeline.json file (You can download it here)

The output will generate a new dat link ready for sharing.

License

MIT License