@idearium/cli

The Idearium cli, which makes working with our projects, super easy :)

Documentation

The cli is self documenting and you'll find limited information about the commands within the cli here.

Install

Via NPM:

$ npm install -DE [-g] @idearium/cli

We actually recommend installing @idearium/cli locally in your project. This allows each project to use a different version of the cli. If you do however, we also recommend adding ./node_modules/.bin to your path, so that you can run the cli via c without a global install.

Usage

Once installed, simply run c, and you'll get the help output for all commands. Running a sub-command and then --help will give you the help output for that command (i.e. c dc --help).

Requirements

The cli has been designed to work closely with infrastructure-common. Infrastructure provides MongoDB, Redis, RabbitMQ and importantly for the cli, Kubernetes ingress.

You can read more about how this all hangs together at https://github.com/idearium/developers/tree/master/stack

Commands

The following is a summary of the top level commands.

• c d is for everything Docker.
• c dc is for everything Docker Compose.
• c hosts helps with hosts management.
• c kc is for everything kubectrl.
• c mk is for everything Minikube.
• c mongo is for MongoDB connections.
• c npm is for everything NPM.
• c op manages 1Password secret values for the secrets contract.
• c project is for project management.
• c secrets manages project secrets across their stores (1Password, GSM, Kubernetes).

Be aware that kubectl uses a global configuration, but the c kc command supercedes those where possible and ensures kubectl runs in the context of the project you're within.

Concepts

The cli has two concepts to be aware of:

• Configuration
• State

The cli can be highly configured within information about your project, and work with Docker, NPM and Kubernetes with the information provided. All configuration is stored within a c.js file that lives in the root of your project. This file should be stored in Git.

The cli is designed to work with multiple environments. You put your project into a particular environment, and the commands run within the context of that environment. All state is stored in devops/state.json (this file should be added to .gitignore).

Configuration

The Idearium cli can be customised through configurations. Configurations are provided through a c.js file located in projects root folder.

The c.js should be a standard Node.js file. It will be loaded through require. The c.js should export an object, with the following keys representing configuration:

module.exports = {
    docker: {
        // ...
    },
    environments: {
        // ...
    },
    kubernetes: {
        // ...
    },
    mongo: {
        // ...
    },
    npm: {
        // ...
    },
    project: {
        // ...
    },
};

The command c project init can be used to initialise a project. It will generate a c.js file as a starting point, although, it will need to be completed.

Docker configuration

The Idearium cli supports a Docker configuration. The Docker configuration can be used to define locations of multiple Dockerfiles within your project.

An example Docker configuration:

'use strict';

const { exec } = require('shelljs');

module.exports = {
    docker: {
        locations: {
            'app': {
                buildArgs: {
                    NPM_AUTH_TOKEN: () => exec('c npm auth -n', { silent: true }).stdout,
                },
                path: './app',
                useTar: true,
            },
            'static': {
                path: './static',
            },
        },
    },
};

The Docker configuration supports the following keys.

locations

The locations key should be an object, describing all Dockerfile locations in your project. Each key should be the name of a Dockerfile location. You can use the name to reference the location when using the cli. Each location supports the following structure.

path

A path to the Dockerfile location.

buildArgs

An object represent key=value --build-arg flags to pass to docker build. Each property should present a build argument name, and the value for the build argument. The value can be either a static value (i.e. string, number) or a function.

useTar

If useTar is set to true, tar will be used to prepare the Docker context and package and send files to the Docker daemon. This allows for some additional flexibility such as using symlinks which point outside the Dockerfile context.

test

You can supply a test object to enable the c kc test <location> command. test has the following properties:

• cmd containing the command to run, inside the Docker container. It defaults to npm test.
• params containing an object keyed by docker run parameter name. For example: { '-e': ['foo="bar"', 'bar="foo"'], '-w': '/app' }. This allows you to customise the docker run command used when testing the image.

Environments configuration

The Idearium cli supports multiple environments. The environments can be whatever you need them to be as long as you define them in c.js. Here is an example environments configuration:

'use strict';

module.exports = {
    environments: {
        local: { url: 'http://domain.name.local' },
        production: { url: 'https://domain.name.com' },
    },
};

For each environment, you should provide a URL for the environment.

Kubernetes configuration

You can supply a Kubernetes configuration. The configuration allows you to define contexts and namespaces for each environment your project supports, along with Kubernetes locations which describe the Kubernetes objects to deploy. Here is an example kubernetes configuration:

'use strict';

module.exports = {
    kubernetes: {
        environments: {
            local: {
                context: 'minikube',
                locations: {
                    /* eslint-disable sort-keys */
                    'namespace': [
                        {
                            path: 'namespace',
                            templateLocals: ['environment', 'namespace'],
                            type: 'namespace',
                        },
                    ],
                    'app': [
                        {
                            path: 'app.deployment',
                            templateLocals: ['namespace', 'prefix', 'tag'],
                            type: 'deployment',
                        },
                        {
                            path: 'app.service',
                            templateLocals: ['namespace'],
                            type: 'service',
                        },
                    ],
                    'static': [
                        {
                            path: 'static.deployment',
                            templateLocals: ['namespace', 'prefix', 'tag'],
                            type: 'deployment',
                        },
                        {
                            path: 'static.service',
                            templateLocals: ['environment', 'namespace'],
                            type: 'service',
                        },
                    ],
                    /* eslint-enable sort-keys */
                },
                path: './manifests/local'
            },
            production: {
                context: 'gke_focus-booster_us-east1-b_focus-booster',
                namespace: 'override',
                region: 'us-east1',
            },
        }
    },
};

The kubernetes.environments path holds a descrete environment. Each Kubernetes environment supports the following keys:

• context, the kubectl context that should be used.
• locations, the Kubernetes objects to work with.
• namespace, if provied, will override the generated namespace.
• path, the path to the folder containing Kuberentes manifest YAML files.
• region, an optional region, mostly required for production.

Kubernetes locations

The Kuberentes locations object describes all Kubernetes objects for that Kuberentes environment. Each location should contain the following ({} used to represent meaning):

locations: {
    {location-name}: [
        {
            dockerLocation: '',
            path: '',
            templateLocals: [],
            type: '',
        }
    ]
}

The location-name should be unique, and can be used within the cli commands to target a specific Kubernetes location.

Each location should be an array of services. One Kubernetes location can contain multiple services.

Each service object should provide:

• dockerLocation (optional): The Docker location that a particular Kubernetes service object is associated with. This is usually provided with a deployment or pod object.
• path: The filename within the Kubernetes environment path. The filename should not include the extension; but only yaml and yaml.tmpl are supported.
• templateLocals: An array to provide a list of locals that should be passed to a tmpl file to create a yaml file.
• type: The type of Kubernetes object this service describes (i.e. pod, deployment, namespace).

YAML templates

The c kc apply command supports templates, which will generate yaml files which will be used to deploy Kuberntes objects.

It supports templates, because you'll often want to substitute specific information for Kubernetes during development. A good example is the tag that should be used with a particular image.

If you'd like to use a template create Kubernetes manifest file ending in .yaml.tmpl rather than tmpl. The path in the Kubernetes location service object, should not contain the extension. In the template file use {{tag}} to substitute with an actual value before being applied to Kubernetes.

You'll then need to supply all of the values that the template file requires. You provide this via the templateLocals array in the Kubernetes location service object. Here is an example:

{
    path: 'app.deployment',
    templateLocals: ['namespace', 'prefix', 'tag', () => { label: 'a', value: 'b' }],
    type: 'deployment'
}

The c kc apply will automatically provide the values for namespace, prefix and tag. If you'd like to provide something else, simply write a function that returns an object with label and value. Then use the value of label within a template placeholder (i.e. {{a}}) and it will be updated with the value (i.e. {{b}}).

Secret templates

With a secrets contract in c.js (see Secrets configuration below), secret-shaped templates are generated automatically — you don't commit them. This section applies to repositories without a contract.

Any template (.yaml.tmpl) can contain 1Password secret references (i.e. op://vault/item/section/field). The template is rendered (placeholders) first, and then any references are resolved with the 1Password cli (op inject) - only when present. Rendering first means secret values never flow through the template engine (a value containing curly braces could otherwise be mangled). As a consequence, references must be bare (i.e. op://vault/item/section/field, not wrapped in {{ }}): placeholders are gone by the time references are resolved, and a wrapped reference would be blanked as an unknown placeholder. When references are present, the cli first ensures the 1Password cli is installed and authenticated (via op whoami), halting with a friendly message if not (eval $(op signin)). Templates without secret references never invoke op.

This makes it possible to commit a template containing secret references, and have the compiled manifest (within .compiled, which should be gitignored) contain the actual secrets, ready to be deployed to Kubernetes.

For a service of type: secret, author the template with stringData instead of data:

apiVersion: v1
kind: Secret
metadata:
    name: site
    namespace: ras-rsc-local
type: Opaque
stringData:
    ALGOLIA_SEARCH_API_KEY: op://ras-rsc/site/LOCAL/ALGOLIA_SEARCH_API_KEY

When the template is compiled, each stringData value will be base64 encoded and written to the compiled manifest as data, just as Kubernetes expects. stringData values must be single-line, and shouldn't be quoted or contain inline comments.

Compiled secret manifests contain plaintext secrets, so they are treated as sensitive: they're written with 0600 permissions, and removed once they've been applied to Kubernetes. c kc start and c kc apply remove them after applying, and c skaffold dev removes them (via c kc secrets-clean) when it exits. c kc manifests intentionally leaves them in place, as its compiled output is its purpose. This only applies to the local environment; other environments are unaffected.

Secrets configuration

The secrets block of c.js is the secrets contract: the single source of truth for every secret-shaped manifest and value reference. With a contract present, the cli generates SecretProviderClasses, SecretSyncs and the secret-sync ServiceAccount for deployed environments, and local Secrets (1Password references) for the local environment, whenever manifests are compiled (c kc manifests, c kc apply, c kc start). A committed manifest file that the contract also generates is a hard error - the contract is the one true source. Without a contract, nothing changes.

secrets: {
    dev: {
        opItem: '{dev-item-id}', // {org}-{repo}/dev
        consumers: {
            'site-build': { keys: ['FONTAWESOME_AUTH_TOKEN'] },
        },
    },
    services: {
        site: {
            opItem: '{item-id}',
            keys: ['CACHE_URL', 'RAS_API_KEY'], // owned
            sharedKeys: ['IMGIX_TOKEN'], // borrowed
        },
        'prismic-published-workflow': {
            sharedKeys: ['prismic-published-algolia.BASIC_AUTH'], // dotted owner.KEY
        },
    },
    shared: {
        // keys whose GSM ids carry no service segment
        // (org-repo-env-key); opItem points at the item storing the
        // values, usually a reference - never auto-created.
        opItem: 'services.site',
        keys: ['IMGIX_TOKEN'],
    },
},

Rules (enforced at load):

• opItem values are 1Password item ids or services.<name>/dev references (one hop - each id appears exactly once). Services resolve (or create) their item lazily by title; the shared block never auto-creates.
• A bare sharedKeys name resolves to its unique owner across services and shared; dotted 'owner.KEY' disambiguates.
• GSM secret ids derive as {org}-{repo}-{env}[-{service}]-{key} - no literal ids anywhere.
• Region comes from gcloud.region; project from project.gcpProjectId.

Commands (values are only ever displayed as truncated hashes; --plaintext requires a terminal and is blocked in agent/CI contexts unless forced):

• c secrets init - draft a contract from existing declarations (requires a warm 1Password session; no-op when a contract exists). Binds shared.opItem by comparing value hashes and removes value-verified legacy duplicates.
• c secrets add <service> <env> <KEY> - add a key end to end (1Password, GSM, c.js). Never touches the cluster.
• c secrets push <service> <env> <KEY> - push the 1Password value to GSM, verified by hash readback.
• c secrets get <service> <env> <KEY> / c op get - value summaries (hash, reference, length).
• c op set <service> <env> <KEY> - upsert a value from stdin (or --generate).
• c secrets verify <env> - reconcile structure + 1Password/GSM/Kubernetes hashes, stale 1Password fields (op:EXTRA), deployed-functions drift (STALE, MISSING) and cross-env misbindings (MISBOUND).
• c secrets ls <env> - the environment's contract-derived GSM inventory.
• c secrets consumers <secretId> - everything referencing a GSM secret id, with each consuming function's name-parsed env. Gate every deletion on this.

Commands that need 1Password, gcloud or kubectl fail fast with friendly re-authentication instructions when their credentials have expired.

A note on the 1Password session. c op session stores the session token at ~/.local/state/idearium/op-session (0600), and every op invocation the cli makes passes it via --session (as does c op cmd). That argument is visible in the local process list - the same exposure any op --session or OP_SESSION_* environment carries. It is machine-local and expires, and secret values themselves never appear in argv: they move via stdin pipes only.

MongoDB configuration

The Idearium cli supports a MongoDB configuration. The MongoDB configuration can be used to access local and remote databases.

An example MongoDB configuration:

'use strict';

module.exports = {
    mongo: {
        beta: {
            host: 'dbhost.com',
            name: 'db-beta',
            password: 'password',
            port: '12345',
            ssl: true,
            user: 'user',
        },
        local: {
            host: 'db.common.idearium.local',
            name: 'db',
            port: '27017',
            ssl: false,
        },
        beta: {
            host: 'dbhost.com',
            name: 'db',
            password: 'password',
            port: '12345',
            ssl: true,
            user: 'user',
        },
    },
};

The MongoDB configuration supports the following keys.

Environment

Each top level key will be used as an env variable, setting the context for the commands. e.g. c mongo connect local.

Each environment can contain the following keys host, name, password, port, ssl, and user.

password and user are currently optional for local connections.

NPM configuration

The Idearium cli supports an NPM configuration. The configuration can be used to provide the cli without information about where NPM commands can be run. To provide this information, add an npm property in c.js like so:

'use strict';

module.exports = {
    npm: {
        locations: {
            name: './app/root/app/',
            project: './',
        },
    },
};

You should customise it, but you need to provide a locations key, containing an object with a name and folder for each NPM location in your project. You'll use the name to reference the location.

For example, the c npm proxy command can be used to run a NPM command, at a specific location or all locations:

• Execute c npm proxy all install -SE logentries to install the logentries module at all NPM locations in your project.
• Execute c npm proxy project install -DE jest to install the jest module at the project location in your project.

Project configuration

The Idearium cli supports a project configuration. This configuration is used to provide some general information about the project. This information is also used to automatically generate some strings such as the project prefix, and Kubernetes namespaces. Here is an example project configuration:

'use strict';

module.exports = {
    project: {
        name: 'www',
        organisation: 'fb'
    },
};

State

The Idearium cli, understands the concept of state. The only state it manages at present is the environment of your project. The current environment of your project will be used in certain commands, but not all commands.

State is stored in a JSON file at ./devops/state.json, and is created and written to by the c project env set and c project setup commands.

Ngrok

The Idearium cli can be used to easily expose a Kubernetes service to the world via Ngrok, and the c kc ngrok command.

To use this command, however, you need to provide a configuration file for Ngrok. Follow these steps.

1. Create a file at ./manifests/local/ngrok.configmap.yaml.tmpl.
2. Add the code from the Manifest template (below) to the file and update <authtoken> with your Ngrok authtoken (or remove that line) and update <hostname> with the hostname you'd like the tunnel to be exposed at.
3. Added the code from the Kubernetes location (below) to c.js in the kubernetes.environments.local.locations array.
4. Then run c kc ngrok start in a new terminal (after running c mk docker-env).
5. To stop, issue CTRL + C, then run c kc ngrok stop to stop ngrok.

Manifest template:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ngrok
  namespace: {{namespace}}
data:
  ngrok.yaml: |
    authtoken: <authtoken>
    console_ui: false
    log: stdout
    region: au
    tunnels:
      http:
        proto: http
        bind_tls: false
        addr: static:80
        hostname: <hostname>
    update: false

Kubernetes location:

'ngrok': [
    {
        path: 'ngrok.configmap',
        templateLocals: ['namespace'],
        type: 'configmap',
    },
],

Publishing

To publish a new version of the cli, create a new tag vX.X.X or vX.X.X-beta.X and push to GitHub. The GitHub workflow will automatically publish the new version to NPM.