github-task-manager

receive github hook, notify agent, receive task results, notify github (Unofficial)

Aims

Create an asynchronous CI agnostic mechanism for running custom test stage gates for github pull requests.

• allow team leads to configure task sequences without leaving github
• allow developers to see output from tasks without leaving github
• trigger multiple jobs in parallel and indicate pending status on pr checks
• then add results for each back to pull request check/comments as they complete
• make extensible for other github event handling
• stateless and deployable to Kubernetes at scale

Design

• Deploy two functions to lambda via serverless framework ('gtmGithubHook', 'gtmGithubResults')
• github PR open hook pointed at 'gtmGithubHook' lambda which adds event to a 'PendingQueue' SQS queue
• agent(s) watch SQS for new test execution jobs
• agent notifies 'ResultsQueue' SQS queue that a task has started (lambda updates github PR state)
• agent triggers CI PR build, deploy and tests (plugin for each CI type)
• agent formats and adds test results to 'ResultsQueue' SQS queue
• agent notifies SNS topic which triggers 'gtmGithubResults' lambda
• 'gtmGithubResults' lambda posts results to github pull request.

Install

• clone this repo or npm install --save github-task-manager
• npm install
• setup serverless aws creds per https://github.com/serverless/serverless/blob/master/docs/providers/aws/guide/credentials.md
• setup a .env file in the repo root (copy from .envExample and modify)
• create and AWS KMS key, and capture the id for var GTM_AWS_KMS_KEY_ID

| Environment variable | description | | -------------------- | ----------- | |GTM_AWS_KMS_KEY_ID | aws kms key id | |GTM_CRYPT_GITHUB_TOKEN | encrypted access token for accessing github | |GTM_CRYPT_GITHUB_WEBHOOK_SECRET | encrypted shared secret from github webook config | |GTM_CRYPT_AWS_ACCESS_KEY_ID | encrypted aws key id - for agent only | |GTM_CRYPT_AWS_SECRET_ACCESS_KEY | encrypted aws secret - for agent only | |GTM_CRYPT_AGENT_AWS_SECRET_ACCESS_KEY|secret key for agent| |GTM_CRYPT_AGENT_AWS_ACCESS_KEY_ID|access key for agent| |GTM_CRYPT_JENKINS_TOKEN| encrypted token | |GTM_CRYPT_TEAMCITY_PASSCODE| encrypted teamcity executor passcode| |GTM_CRYPT_SONAR_LOGIN| encrypted sonar access token | |GTM_CRYPT_SONAR_GITHUB_OAUTH| encrypted github token for sonar to post comments and status | |GTM_CRYPT_DOCKER_REG_PASSWORD| encrypted docker private registry password| |GTM_AWS_REGION | awsregion to create resources in | |GTM_SQS_PENDING_QUEUE | name of SQS queue for new event | |GTM_SQS_RESULTS_QUEUE | name of SQS queue for results | |GTM_SNS_RESULTS_TOPIC | name of SNS topic for result ping | |GTM_GITHUB_HOST | api hostname can be updated for github enterprise | |GTM_GITHUB_DEBUG | debug mode for api calls | |GTM_GITHUB_TIMEOUT | github api timeout | |GTM_GITHUB_PATH_PREFIX | path prefix for github enterprise | |GTM_GITHUB_PROXY | github api client proxy | |GTM_TASK_CONFIG_FILENAME | filename in repo to look for for task config - default is .githubTaskManager | |AWS_PROXY|URL of proxy to use for network requests. Optional| |GTM_AGENT_PORT| defaults to 9091 | |GTM_JENKINS_USER|login for jenkins executor| |GTM_JENKINS_URL|url executor uses to talk to jenkins| |GTM_JENKINS_CSRF| is csrf enabled? true or false| |GTM_TEAMCITY_USER|teamcity executor user| |GTM_TEAMCITY_URL|teamcity api url| |GTM_DOCKER_IMAGE_WHITELIST| comma separated list of regex of allows docker images eg. alpine:*,bash:latest| |GTM_DOCKER_IMAGE_WHITELIST_FILE|use an optional docker whitelist file .dockerImageWhitelistExample| |GTM_DOCKER_COMMANDS_ALLOWED| default is false, set to true to enable docker executor| |GTM_DOCKER_ALLOW_PULL| allow agent to pull from registry | |GTM_DOCKER_DEFAULT_WORKER_IMAGE| for running ci tasks, default is zotoio/gtm-worker:latest | |GTM_DOCKER_REG_USERNAME| username for docker private registry| |GTM_DOCKER_REG_SERVER| hostname for docker private registry| |IAM_ENABLED|agent host uses IAM ?| |LAUNCHDARKLY_API_TOKEN|token for launchdarkly sass executor| |GTM_LOGSTASH_HOST|optional logstash host for elasticsearch analysis| |GTM_LOGSTASH_PORT|optional logstash port| |GTM_SONAR_HOST_URL| sonar host url to connect to | |GTM_SONAR_PROJECTNAME_PREFIX| prefix if reporting to sonarqube | |GTM_SONAR_ANALYSIS_MODE| mode for sonar runner, default preview for PRs | |GTM_SONAR_SOURCES| default source dir is src| |GTM_SONAR_JAVA_BINARIES| default is target| |GTM_SONAR_MODULES| comma separated modules| |GTM_SONAR_GITHUB_ENDPOINT| optional enterprise github api url| |GTM_TASK_CONFIG_DEFAULT_URL| url to default sample config used when repo is missing .githubTaskManager.json| |GTM_TASK_CONFIG_DEFAULT_MESSAGE_PATH| path to markdown comment file added to PRs when repo is missing .githubTaskManager.json| |GTM_DYNAMO_TABLE_EVENTS| DynamoDB table to store event summaries | |GTM_DYNAMO_TABLE_AGENTS| DynamoDB table to store agent summaries | |GTM_AWS_VPC_ID| vpc id - only required for ddb endpoints | |GTM_BASE_URL| Base url used to render links to agent ui - eg elb cname | |GTM_WELCOME_MESSAGE_ENABLED| If not 'false', send a warning message on unconfigured repository pull requests | |GTM_S3_DEPENDENCY_BUCKET| aws s3 storage of build dependencies| |GTM_AWS_S3_PROXY| https_proxy for aws s3 | |GTM_REPO_BLACKLIST| comma separated list of regex to blackist repo names from triggering events | |GTM_SLS_EXECUTOR_AWS_STAGE| stage override from default calculation of dev/test| |GTM_SLS_EXECUTOR_AWS_REGION| aws region for lambdas default ap-southeast-2| |GTM_SLS_EXECUTOR_AWS_EXECUTION_ROLE| docker serverless lambda execution role | |GTM_SLS_EXECUTOR_DEPLOY_MODE| deploy multiple lambdas 'parallel' (default) or 'sequential'| |GTM_SLS_EXECUTOR_SNS_ERROR_TOPIC_ARN| sns topic to notify on serverless error| |GTM_SLS_EXECUTOR_APIGW_ENDPOINT_TYPE|EDGE or REGIONAL or PRIVATE| |GTM_SLS_EXECUTOR_APIGW_DOMAIN_SUFFIX|reverse proxy domain name that will have apiId added as subdomain. eg. lambda.mysuffix.com will result in [apiId].lambda.mysuffix.com| |GTM_SLS_EXECUTOR_HTTP_PROXY|proxy passed to serverless executor| |GTM_SLS_EXECUTOR_NO_PROXY|no_proxy passed to serverless executor| |GTM_SLS_EXECUTOR_VPC_ID|vpc id for private apigw endpoints| |GTM_SLS_EXECUTOR_VPC_SECURITY_GROUP_ID|vpc security group id| |GTM_SLS_EXECUTOR_VPC_SUBNET_A| vpc az subnet | |GTM_SLS_EXECUTOR_VPC_SUBNET_B| vpc az subnet | |GTM_SLS_EXECUTOR_VPC_SUBNET_C| vpc az subnet | |GTM_SLS_EXECUTOR_AWS_KMS_KEY_ID| kms key id for sls env var encryption | |GTM_SLS_EXECUTOR_CONFIG_TYPE|'ssm' (aws parameter store) or 'spring' or 'dotenv'| |GTM_SLS_EXECUTOR_SPRING_CONFIG_ENDPOINT|endpoint url of spring config server eg. http://spring:8888 if GTM_SLS_EXECUTOR_CONFIG_TYPE is 'spring'| |GTM_WORKER_SCRIPTS_CLONE| for docker executors using https://github.com/zotoio/gtm-worker based image - url of git repo to overlay on workspace eg. https://github.com/zotoio/gtm-worker-scripts.git | |GTM_WORKER_SCRIPTS_PATH| directory within scripts clone repo to overlay|

important: values of env vars prefixed with GTM_CRYPT_* must be created via npm run sls-encrypt [name] [value]

Configure and deploy

• run: npm run sls-deploy - note that this will create aws re$ources..
• capture the hook url output in console and add to github repo pull request conf
• run: npm run sls-logs-hook or npm run sls-logs-results to tail the logs
• create a .githubTaskManager.json in your repo per https://github.com/zotoio/github-task-manager/wiki/Creating-a-Task-Configuration
• start an agent locally using npm run build && npm start agent (or use docker/k8s)
• create a pull request and confirm the hook is being hit and agent processes event

Docker and Kubernetes agents

You can run the latest image from docker hub: https://hub.docker.com/r/zotoio/github-task-manager

npm run docker-hub-run

..or run using the local checkout and tail logs:

npm run docker-local-bounce

..or if you have a k8s cluster and kubectl configured:

npm run k8s-apply
npm run k8s-delete

note that these k8s npm script inject vars from .env into the manifest

a starting point k8s manifest is in ./k8s/k8s-gtm-agent.yml

Agent Configuration

Running the Agent

The agent uses environment variables to configure itself as well as any executors running within it. The below environment variables are required for basic operation of the GTM Agent.

TODO: Provide Variables

Agent Homepage

The GTM Agent provides an information page summarising the ongoing operation of the agent. The page is available on port 9091 by default.

Plugins

Task executors for Jenkins, Teamcity, Travis, Http, Docker are in progress. Custom task executors can be added by adding this project as a dependency, and registering new Executors and EventHandlers. Please see https://github.com/zotoio/gtm-agent for an example that you can fork and modify as required while still using this project as the core.

• Executors contain the logic to run tasks against a given system type, and format the results.
• EventHandlers are used to map Github events to specific functionality such as pull requests.

Contributing

Fork this repository and work on your enhancements, then send a pull request. If you build custom plugins that may be useful to others in your forked gtm-agent, please let us know and may be able to assist wwith backporting to this project.

Use commitizen for conventional commit messages via git cz instead of git commit.
To setup if not already installed:

npm install -g commitizen
npm install -g cz-conventional-changelog
echo '{ "path": "cz-conventional-changelog" }' > ~/.czrc

...or you can just use npm run commit which will use local commitizen install.