@cumulus/sync-granule

Download a given granule from a given provider to S3

Message Configuration

Config

| field name | type | default | values | description | | ----------------- | ------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | buckets | object | (required) | | Object specifying AWS S3 buckets used by this task | | downloadBucket | string | (required) | | Name of AWS S3 bucket to use when downloading files | | provider | object | (required) | | The cumulus-api provider object | | collection | object | | | The cumulus-api collection object | | duplicateHandling | string | error | error - Throws an error on duplicatesreplace - Replaces the existing fileskip - Skips the duplicate fileversion - Adds a suffix to the existing filename to avoid a clash | Specifies how duplicate filenames should be handled | | fileStagingDir | string | file-staging | | Directory used for staging location of files. Granules are further organized by stack name and collection name making the full path file-staging/<stack name>/<collection name> | | pdr | object | | | Object containing the name and path for a PDR file | | sftpFastDownload | boolean | false | | If true, sftp download is performed using parallel reads for faster throughput. Lambda ephemeral storage is used to download files before files are uploaded to s3. Please note that not all sftp servers have the concurrency support required. See https://www.npmjs.com/package/ssh2-sftp-client#orge45232c for more information. | | syncChecksumFiles | boolean | false | | If true, checksum files are also synced. | | stack | string | | | The name of the deployment stack to use. Useful as a prefix. | | useGranIdPath | boolean | true | true, false | If set to false, the granule file staging path will be '<staging_dir>/<collection_id>/`. If set to true, the granule file staging path will be '<staging_dir>/<collection_id>/<md5_granIdHash_file>. Please note that duplicate collision settings in sync granule are impacted by this as file collision will be -per granuleId- instead of -per collection- for this task | workflowStartTime | integer | | | Specifies the start time (as a timestamp) for the current workflow and will be used as the createdAt time for granules output. If the specified timestamp is in the future, then the current time will be used instead.

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

To make a contribution, please see our contributing guidelines.