The dbt project for managing data transformations for LDT's organiser-facing reporting.

Setting up the project for local development

1. Create a copy of the .env.sample file and name it .env. Delete any comments (noted by #...).

2. Set the environment variables in .env according to the commented instructions.

3. To run dbt commands locally, you'll need to setup a virtual environment:

• Check that you have Python3 installed on your Mac by running which python3. If not, install Python3 using run install-python or via the Python downloads page.
• Create your virtual environment, by running:

    yarn create-virtual-env

• Activate your virtual environment, by running:

    source dbt_venv/bin/activate

4. Install dbt (with the postgres adapter), as specified in the requirements.txt file, by running:

    yarn install-dbt

5. Setup, and install any external dbt package dependencies that this project requires, by running:

    yarn setup-dbt

6. If this outputs "All checks passed!" you're now ready to run some dbt commands! Here are some examples:

• To run all the models, run:

    yarn run

• To run one model, use the --select flag (or -s flag), followed by the name of the model:

    doppler run -- dbt run --select <MODEL NAME>

• To run a model and all of its downstream children:

    doppler run -- dbt run --select <MODEL NAME>+

• To run a model and all of its upstream parents:

    doppler run -- dbt run --select +<MODEL NAME>

• To run just the models in one subdirectory of the models folder, use the --select flag (or -s flag), followed by the name of the subdirectory:

    doppler run -- dbt run --select models.<SUBDIRECTORY NAME>

• To run all the tests, run:

    doppler run -- dbt test

• To run a macro, run:

    doppler run -- dbt run-operation <MACRO NAME>

• To run a macro with arguments, run:

    doppler run -- dbt run-operation <MACRO NAME> --args '{<ARG 1>: <VALUE 1>, <ARG 2>: <VALUE 2>}'

Creating a new dbt model

dbt models are all stored in the models subfolder in src. To create a new dbt model:

1. Create a .sql file for your model in a folder under the models directory. For operational reports, create the model in the operational subfolder. Aggregate analytic models should be created in the analytics subfolder.

2. Add a description of the model and columns in schema.yml. Here you can also define tests to run on your model. For more details about writing tests in dbt, see [this link] (https://docs.getdbt.com/docs/build/tests).

3. Run dbt run -m <MODEL NAME> to create the model locally.

4. If you have specified tests on your model, run dbt test -m <MODEL NAME> to test your model. Note that you need to build the model using dbt run before you test it.

5. Push your changes and raise a PR.

Sharing DBT Models with wider code-base

1. To re-generate the typescript schemas and view creation commands run:

    yarn gen

2. Copy package to where it needs to be used

Resources:

• Learn more about dbt in the docs
• Check out Discourse for commonly asked questions and answers
• Join the chat on Slack for live discussions and support
• Find dbt events near you
• Check out the blog for the latest news on dbt's development and best practices