benchctl

benchctl is a CLI tool designed to generate benchmark references for ISO 5055 Security-related CWEs using GitHub CodeQL. It automates the process of setting up CodeQL, preparing query suites (including gap queries for missing coverage), fetching CodeQL databases, and analyzing repositories to produce structured findings.

Installation

npm install -g @sneezry/benchctl

Prerequisites

• Node.js: >= 18
• CodeQL CLI:
  • macOS/Linux: benchctl can automatically download and install the CodeQL CLI.
  • Windows: You may need to install the CodeQL CLI manually if it's not found in your path or CODEQL_BINARY environment variable.

Usage

The typical workflow involves four steps:

1. Setup

Ensure the CodeQL CLI is installed and available.

benchctl setup

2. Prepare

Generate the ISO 5055 query suites and gap queries. This step scans existing CodeQL packs to determine coverage and generates "gap queries" for CWEs that are applicable but not covered by standard queries.

benchctl prepare

• Use --no-pull to skip updating CodeQL packs if you are offline or want to save time.

3. Fetch Database

Download a CodeQL database for a specific GitHub repository.

benchctl fetch-db <owner>/<repo>

Example:

benchctl fetch-db expressjs/express

4. Analyze

Run the analysis using the generated ISO 5055 suites.

benchctl analyze <owner>/<repo> [options]

Options:

• --kind <gap|real|all>: Filter findings by evidence type.
  • real: Only show findings from actual CodeQL queries.
  • gap: Only show findings from generated gap queries (indicating missing coverage).
  • all: Show all findings (default).
• --output <file>: Write the JSON results to a specific file.

Example:

benchctl analyze expressjs/express --kind all --output results.json

Global Options

• --no-interactive: Disable the interactive UI (spinners, progress bars) and use plain console logging. This is automatically enabled in CI environments.

License

MIT