cc-util-sync-collective

This module allows you to move data between Journey Apps without having to use code outside of the Delivery framework. There are two supported models, Master / Slave replication and Master / Master replication.

The purpose of this module is to make the process as generic and seamless as possible.

Code origin is from JS Task with the original owner Eugene Louw

Documentation

All code is clearly documented, please see code for more details. You can view some example code below

Required Reading

To enable replication you are expected to have a decent understanding of how certain parts of the Journey System work. Please review the links below:

1. Webhooks: http://resources.journeyapps.com/v4/webhooks
2. JourneyApps Backend API: http://resources.journeyapps.com/v4/api
3. Cloud Code : https://docs.journeyapps.com/docs/cloudcode-introduction

1. Master / Slave Replication

Setup

1. Ensure that the Model(s) exist on all Slaves.
2. On the Master app, create the webhook on the Model(s) that you want replicated to the slave apps
3. Add the CloudCode Task to the Master Journey App
4. Add the collective on the Data Broker
5. Add the endpoints to the Data Broker
6. Add the tables to sync to the Data Broker
7. Create the Model(s) in the Data Model of the Broker

1. Ensure that the Model(s) exist on all Slaves.

Copy the model(s) that you want to sync from your Master Data Model XML to all of the Slave Data Model XML.

2. Create the webhook on the Model of the Master Journey App

Open the Data Model view of the Master Journey App and add the following xml to the Model(s) that you want to sync:

  <!-- Required for Sync -->
  <webhook type="update" receiver="cloudcode" action="sync_collective" />

3. Add the CloudCode to the Master Journey App

1. Create a new CloudCode task in your Master Journey App.
2. Create an config file with the credentials required.

module.exports = {
    "collective": "aviat",
    "credentials" : {
      local: {
        "url" : "<url>",
        "api_user": "<api_user>",
        "api_pass": "<api_pass>",
      },
      remote: {
        "url" : "<url>",
        "api_user": "<api_user>",
        "api_pass": "<api_pass>",
      }
    }
}

Review the following fields and change as described:

1. collective: The name that you have given the organization, only use alpha characters. The naming convention of a collective is built from the app url as follows, https://build.journeyapps.com/kaufmann/accrete-solutions-grupo-kaufmann-service-advisor In the above example the collective will be called "kaufmann". In the case where the url might contain "-" as follow, "kauf-mann", then remove the "-" and the collective name becomes "kaufmann".

The following fields point to the Data Broker and correspond to the Testing Environment for the Data Broker and will not need changing.

Local:

1. url: The url to the Instance DB
2. api_user: The api_user to the Instance DB
3. api_pass: The api_pass to the Instance DB

Remote:

1. url: The url to the Data Broker
2. api_user: The api_user of the Data Broker
3. api_pass: The api_pass of the Data Broker

Ensure that you have checked "JourneyApps Webhooks" and named the task sync_collective.

4. Add the collective on the Data Broker

Create the collective in, https://run-testing-us.journeyapps.com/app/journey-usa-data-broker/acc/577ec96be1a2ef691feaf19a/objects#/collective/new NB. The name of the collective should match collective from the CloudCode task you created in step 2.

5. Add the endpoints to the Data Broker

Endpoints are defined as all the Journey Apps that are part of the replication process. This includes all the Slave and the Master Journey Apps.

Create all the endpoints in, https://run-testing-us.journeyapps.com/app/journey-usa-data-broker/acc/577ec96be1a2ef691feaf19a/objects#/endpoint/new

1. Endpoint.name: Name to reference the Journey App Name, the naming convention to be used comes from the Journey App url. https://build.journeyapps.com/kaufmann/accrete-solutions-grupo-kaufmann-service-advisor

The following fields can all be found in the Deploy / Manage API section of your Journey App,

2. Endpoint.api_user: The HTTP Auth Username field on the Journey App
3. Endpoint.api_pass: The HTTP Auth Password field on the Journey App
4. Endpoint.url: The Base URL field on the Journey App
5. Endpoint.collective: Select the collective you created in Step 3.

6. Add the models to sync to the Data Broker

In this step you need to create an entry for each endpoint and model that need to be synced. So if you had three endpoints and two models, you would create 3x2=6 entries in this step.

The naming convention for the Sync Model Name is

{collective_name}_{model_name}

Create an entry (Sync Model Name) for each endpoint that forms part of the replication process (master and all slaves) https://run-testing-us.journeyapps.com/app/journey-usa-data-broker/acc/577ec96be1a2ef691feaf19a/objects#/sync_table/new

7. Create the model in the Data Model of the Broker

1. Open the Data Broker Data Model XML, https://build.journeyapps.com/journey-usa/journey-usa-data-broker/schema/edit#/data_model/code

2. a) Copy the data model from your Master Journey App Data Model XML view, scroll to the bottom of the Data Broker Data Model XML if the collective has never been used before and paste it there.

OR

b) If the collective has been used before find the collective within the XML document and paste the model structure there. Rename the pasted model to follow the naming convention defined in step 6.

{collective_name}_{model_name}

3. The following fields must be added to your Data Broker Data Model XML that you are syncing, some of these fields might already exist and should not be duplicated,

<!-- Required for Sync -->
<field name="synced" label="Synced" type="boolean" />
<field name="api_user" label="ApiUser" type="text" />
<field name="updated_at" label="UpdatedAt" type="datetime" />

<!-- Required for Sync -->
<webhook type="update" receiver="cloudcode" action="sync_collective" />

Note on syncing relationships: Remove all has-many fields and convert belongs-to fields to {field_name}_id. The editor will not like this very much but it will continue to work. See this quandora for further information about the warning.

2. Master / Master Replication

Overview: Changes to a Model can come from any of the associated Journey Apps within the collective and will be synced out to all of the other Journey Apps within the collective.

Setup

1. Create the webhook on the Model of the Master Journey App
2. Add the CloudCode Task to the Master Journey App
3. Copy the model and repeat step 2 for all Master Journey Apps
4. Add the collective on the Data Broker
5. Add the endpoints to the Data Broker
6. Add the tables to sync to the Data Broker
7. Create the model in the Data Model of the Broker
8. Ensure that the table exists on all Master Apps.

1. Create the webhook on the Model of the Master Journey App

Open the Data Model view of your app and add the following xml to the Model(s) that you want to sync,

<!-- Required for Sync -->
<field name="synced" label="Synced" type="boolean" />
<webhook type="update" receiver="cloudcode" action="sync_collective" />

2. Add the CloudCode Task to the Master Journey App

1. Create a new CloudCode in your Master Journey App. https://us-be-js.journeyapps.com/admin/testing_usa/577eccd7ecc8cf1d48849d9c/577ece976cfe5403bd290f61

2. Copy and paste from below section to your json config in the cloudcode that you are creating.

module.exports = {
    "collective": "aviat",
    "credentials" : {
      local: {
        "url" : "<url>",
        "api_user": "<api_user>",
        "api_pass": "<api_pass>",
      },
      remote: {
        "url" : "<url>",
        "api_user": "<api_user>",
        "api_pass": "<api_pass>",
      }
    }
}

Review the following fields and change as described:

1. collective: The name that you have given the organization, only use alpha characters. The naming convention of a collective is build from the app url as follows, https://build.journeyapps.com/kaufmann/accrete-solutions-grupo-kaufmann-service-advisor

   In the above example the collective will be called "kaufmann". In the case where the url might contain "-" as follow, "kauf-mann", then remove the "-" and the collective name becomes "kaufmann".

The following fields point to the Data Broker and correspond to the Testing Environment for the Data Broker and will not need changing.

Local:

1. url: The url to the Instance DB
2. api_user: The api_user to the Instance DB
3. api_pass: The api_pass to the Instance DB

Remote:

1. url: The url to the Data Broker
2. api_user: The api_user of the Data Broker
3. api_pass: The api_pass of the Data Broker

Ensure that you have checked "Webhook triggering" and named the Cloud Code Task sync_collective before saving.

3. Copy the model and repeat step 2 for all Master Journey Apps

Copy the data model from step 1 to all of your Master Journey Apps and repeat step 2 for each of the apps that you copied the model to.

4. Add the collective on the Data Broker

Create the collective in, https://run-testing-us.journeyapps.com/app/journey-usa-data-broker/acc/577ec96be1a2ef691feaf19a/objects#/collective/new

NB. The name of the collective should match collective from the Cloud Code you created in step 2.

5. Add the endpoints to the Data Broker

Endpoints are defined as all the Journey Apps that are part of the replication process. This includes all the Slave and the Master Journey Apps.

Create all the endpoints in, https://run-testing-us.journeyapps.com/app/journey-usa-data-broker/acc/577ec96be1a2ef691feaf19a/objects#/endpoint/new

1. Endpoint.name: Name to reference the Journey App Name, the naming convention to be used comes from the Journey App url. https://build.journeyapps.com/kaufmann/accrete-solutions-grupo-kaufmann-service-advisor

The following fields can all be found in the Deploy / Manage API section of your Journey App, 2. Endpoint.api_user: The HTTP Auth Username field on the Journey App 3. Endpoint.api_pass: The HTTP Auth Password field on the Journey App 4. Endpoint.url: The Base URL field on the Journey App 5. Endpoint.collective: Select the collective you created in Step 4.

6. Add the models to sync to the Data Broker

In this step you need to create an entry for each endpoint and model that need to be synced. So if you had three endpoints and two models, you would create 3x2=6 entries in this step.

The naming convention for the Sync Model Name is

{collective_name}_{model_name}

Create an entry (Sync Model Name) for each endpoint that forms part of the replication process (master and all slaves)

https://run-testing-us.journeyapps.com/app/journey-usa-data-broker/acc/577ec96be1a2ef691feaf19a/objects#/sync_table/new

7. Create the model in the Data Model of the Broker

1. Open the Data Broker Data Model XML, https://build.journeyapps.com/journey-usa/journey-usa-data-broker/schema/edit#/data_model/code

2. Copy the data model from your Journey App Data Model XML view, scroll to the bottom of the Data Broker Data Model XML if the collective has never been used before and paste it there.

3. If the collective has been used before find the collective within the XML document and paste the model structure there. Rename the pasted model too follow the naming convention defined in step 6.

{collective_name}_{model_name}

4. The following fields must be added to your Data Broker Data Model XML that you are syncing, some of these fields might already exist and should not be duplicated,

<!-- Required for Sync -->
<field name="synced" label="Synced" type="boolean" />
<field name="api_user" label="ApiUser" type="text" />
<field name="updated_at" label="UpdatedAt" type="datetime" />

<!-- Required for Sync -->
<webhook type="update" receiver="cloudcode" action="sync_collective" />

Note on syncing relationships : Remove all has-many fields and convert belongs-to fields to {field_name}_id. The editor will not like this very much but it will continue to work. See this quandora for further information about the warning.

8. Ensure that the model exists on all Master Apps.

Installation

Per machine

# Once as setup
npm login
 > journeyapps-solutions-user
 > see password on 1Password
 > [email protected]

yarn add @journeyapps-solutions/cc-util-sync-collective --save

Per CloudCode app

NPM_TOKEN=... # Token for journeyapps-solutions-user from 1Password
echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ./app/cloudcode/.npmrc

Per CloudCode task

yarn add @journeyapps-solutions/cc-util-sync-collective

Deploying

yarn version

Usage

Data Model

Broker Data Model, The broker acts as the Master

<!-- ADD OBJECT TYPES HERE: -->    
<!-- Collective can be seen as the organization -->
<!-- for which we want to sync the data between -->
<model name="collective" label="Collective">
    <field name="name" label="Name" type="text" />

    <has-many model="endpoint" name="endpoints" />

    <display>{name}</display>        
</model>

<!-- Endpoint is where the data is coming from and where -->
<!-- data is going to. We will have two models here, master / slave and  -->
<!-- master / master, think of it in terms of traditional replication -->
<!-- The data here should be found on the Deploy section of your app -->
<model name="endpoint" label="Endpoint">
    <field name="name" label="Name" type="text" />

    <field name="api_user" label="ApiUser" type="text" />
    <field name="api_pass" label="ApiPass" type="text:password" />
    <field name="url" label="Url" type="text" />

    <belongs-to model="collective" />
    <has-many model="sync_table" name="sync_tables" />

    <display>{name}</display>
</model>

<!-- Endpoint might not want all the data tables synced that is associated with a collective -->
<!-- So this will give you a way to only set the tables the you require for the endpoint -->
<!-- table_name should be in the format of {collective}_{name_on_endpoint} -->
<model name="sync_table" label="SyncTable">
    <field name="table_name" label="Sync Model Name" type="text" />

    <belongs-to model="endpoint" />
    <display>{endpoint.name}: {table_name}</display>
</model>

Collective example. Broker side (Master)

<!-- Collective:aviat -->
<model name="aviat_project" label="Aviat: Project">
    <field name="number" label="Number" type="text"/> <!-- Identifier -->
    <field name="customer_name" label="Customer Name" type="text"/>
    <field name="project_name" label="Project Name" type="text"/> <!-- typically the name of the end-user e.g. Oracle Stadium, where the Aviat Customer would be Verizon installing wireless for the stadium -->
    <field name="date_created" label="Date Created" type="date"/>
    <field name="status" label="Status" type="single-choice">
        <option key="draft">Draft</option>
        <option key="compiling">Compiling</option>
        <option key="survey">Survey</option>
        <option key="review">Review</option>
        <option key="archived">Archived</option>
    </field>
    <field name="archived" label="Archived?" type="boolean" />

    <!-- Required for Sync -->
    <field name="country_id" label="Country ID" type="text" />
    <field name="created_by_account_id" label="Created By Account ID" type="text" />
    <field name="synced" label="Synced" type="boolean" />
    <field name="api_user" label="ApiUser" type="text" />
    <field name="updated_at" label="UpdatedAt" type="datetime" />

    <!-- Required for Sync -->
    <webhook type="update" receiver="cloudcode" action="sync_collective" />

    <display>{number} - {customer_name} ({status})</display>
</model>

Collective example, Slave side.

<model name="project" label="Project">
        <!-- NB: This model is synced with other apps (Path Middleman, Install & COP, Data Broker). Any changes need to be replicated there if you want them in those apps. -->
        <field name="number" label="Number" type="text"/> <!-- Identifier -->
        <field name="customer_name" label="Customer Name" type="text"/>
        <field name="project_name" label="Project Name" type="text"/> <!-- typically the name of the end-user e.g. Oracle Stadium, where the Aviat Customer would be Verizon installing wireless for the stadium -->
        <field name="system_description" label="System Description" type="text:paragraph" />
        <field name="system_diagram_photo" label="System Diagram Photo" type="photo" />
        <field name="date_created" label="Date Created" type="date"/>
        <field name="status" label="Status" type="single-choice">
            <option key="draft">Draft</option>
            <option key="compiling">Compiling</option>
            <option key="survey">Survey</option>
            <option key="review">Review</option>
            <option key="archived">Archived</option>
        </field>
        <field name="archived" label="Archived?" type="boolean" />
        <field name="previous_status" label="Previous Status" type="text" />
        <!-- Required for sync with the data broker -->
        <field name="synced" label="Synced" type="boolean" />

        <belongs-to model="country" /> <!-- Required for i18n -->
        <belongs-to model="user_account" name="created_by_account"/>
        <belongs-to model="user_account" name="project_owner" />
        <has-many model="assignment" name="assignments" />
        <has-many model="site_survey" name="site_surveys"/>
        <has-many model="path_survey" name="path_surveys"/>
        <has-many model="site_information" name="site_informations" /> <!-- Will only have one -->

        <index on="created_by_account" />
        <index on="country" />

        <!-- Required for sync with the data broker -->
        <webhook type="update" receiver="cloudcode" action="sync_collective" />

        <display>{number} - {customer_name} ({status})</display>
    </model>

Cloud Code:

index.js: - Master example

// make sure config.js has the relevant credentials
const config = require('./config');
const SyncCollective = require('@journeyapps-solutions/cc-util-sync-collective');

export async function run() {

  const SyncCollective = new SyncCollective({/*..Options..*/});

  let _object = webhook.object;
  let _operation = webhook.operation;
  let _sync_tables = DB.sync_table.where("table_name = ?", _object.type).first();
  let _endpoint = sync_table.endpoint();

  SyncCollective.syncToSlave({
    operation : SyncCollective.operation[_operation],
    objectToSync : _object,
    endpoint : _endpoint
  });


}

index.js: - Slave example

// make sure config.js has the relevant credentials
const config = require('./config');
const SyncCollective = require('@journeyapps-solutions/cc-util-sync-collective');

export async function run() {

  const SyncCollective = new SyncCollective({/*..Options..*/});

  let _object = webhook.object;

  SyncCollective.syncToMaster({
  	objectToSync : _object
  });

  //Now save the Synced Object
  _object.synced = true;
  await _object.save();

}

Testing

yarn test