BigQuery Tools

This package consists of 2 parts:

1. Light-weight CLI with minimum dependencies to sync BQ table schemas as one of the steps during deployment.
2. Small BQ client that could be used in a limited Cloudflare Workers environment.

CLI

Configuration can be provided via environment variables, command line arguments or a mix of both with command line arguments having priority over environment variables.

Credentials must be specified either via file path in GOOGLE_APPLICATION_CREDENTIALS or as an encoded value in ADC_ENCODED.

Schemas to be synced must be located in directory specified in schemas-path parameter and named in the following format: <table name>.schema.json (e.g. user.schema.json). File content must be a valid JSON that satisfies TableSchema structure. Ref: https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableSchema.

Note that prefix/suffix will be concatenated as is without adding extra separators since BigQuery supports multiple special characters that could be used as separators in tables' names. Ref: https://cloud.google.com/bigquery/docs/tables#table_naming.

Usage example:

npx nodeart-bq-cli --dataset=my_dataset --schemas-path=./schemas --table-prefix=project1_ --table-suffix=_stage

| env | arg | type | required | example | description | |:--------------------------------:|:----------------:|:------:|:--------:|:----------------------------:|:-------------------------------------------------------------------------------------------------------:| | BQ_DATASET | --dataset | string | YES | my_dataset | Dataset ID. Ref: https://cloud.google.com/bigquery/docs/datasets#dataset-naming | | BQ_SCHEMAS_PATH | --schemas-path | string | YES | ./path/to/schemas | Path to schemas directory | | GOOGLE_APPLICATION_CREDENTIALS | - | string | NO | ./path/to/credentials.json | Path to ADC file. Ref: https://cloud.google.com/docs/authentication/application-default-credentials#GAC | | ADC_ENCODED | - | string | NO | - | Base64-encoded JSON value of Application Default Credentials | | BQ_TABLE_PREFIX | --table-prefix | string | NO | some_prefix_ | Prefix that will be added to the beginning of tables' names when syncing their schemas | | BQ_TABLE_SUFFIX | --table-suffix | string | NO | _and_suffix | Suffix that will be added to the end of tables' names when syncing their schemas |