renovate-graph, the Renovate package data exporter

renovate-graph is a tool, powered by Renovate, that allows exporting the dependency tree of a repository as a JSON blob.

It's named like so because it is a lightweight wrapper on top of the Renovate dependency update tool, extending Renovate with the ability to collect additional package data against repositories, and exporting the data in a form that can be used for other purposes.

renovate-graph leverages the dozens of supported package ecosystems that Renovate does, as well as the ability to define your own.

However, as renovate-graph is purely used to extract package metadata, it ignores the way that Renovate is configured for different workflows, allowing you to extract all package metadata, even if a repository disables it i.e. via enabledManagers.

FAQ

Do I already need to use Renovate?

No, you don't need to be onboarded to Renovate to use renovate-graph, and comes with a reasonable set of defaults.

Do I have to use renovate-graph if we're already using Renovate?

If you are using Renovate already, you'll get better data out of renovate-graph.

It's recommended you run renovate-graph in addition to Renovate, given renovate-graph can ignore any disabling of dependency scanning (i.e. via enabledManagers or packageRules), which allows it to collect more data than may be found by Renovate itself.

Repository configuration will be followed, but disabling of dependency scanning will be ignored.

Why would I use renovate-graph over Renovate's inbuilt data export functionality?

Aha, you know your Renovate configuration well 👀

You're likely referring to the mechanisms described in GitHub Discussion: create dependency list / tree or GitHub Discussion: Add support for outputting result of extracted package tree and pending updates.

If so, you referring to one of the following:

Renovate debug logs

When Renovate is configured to output debug logs i.e. via the LOG_LEVEL=debug environment variable, you will see log lines such as:

{
  "baseBranch": "main",
  "config": {
    "github-actions": [
      {
        "deps": [
          { ... }
        ],
        "packageFile": ".github/workflows/rebuild.yml"
      }
    ]
  },
  "hostname": "...",
  "level": 20,
  "logContext": "...",
  "msg": "packageFiles with updates",
  "name": "renovate",
  "pid": 1,
  "repository": "jamietanna/jamietanna",
  "time": "...",
  "v": 0
}

This can be parsed to receive similar data to what renovate-graph exports.

NOTE: When using an existing Renovate setup, this is susceptible to missing data due to the way that Renovate is configured, for instance to require configuration, explicitly opting out a repository or limiting the enabledManagers, all of which will report different data to that of renovate-graph, which ignores any repo-specific configuration. This is suboptimal data collection, but is still a lot better than other tools on the market, and is better than not using Renovate data.

If you are happy with this, then you can consume these via Dependency Management Data (aka DMD).

Renovate Reports

When Renovate is configured to perform Report exports, you will receive a file export such as:

{
  "problems": [
  ],
  "repositories": {
    "jamietanna/jamietanna": {
      "branches": [

      ],
      "packageFiles": {
        "github-actions": [
          { ... }
        ],
        "packageFile": ".github/workflows/rebuild.yml"
      }
    },
    "oapi-codegen/oapi-codegen": {
      "branches": [

      ],
      "packageFiles": {
        "gomod": [
          { ... }
        ]
      }
    }
  }
}

This can be parsed to receive similar data to what renovate-graph exports.

NOTE: When using an existing Renovate setup, this is susceptible to missing data due to the way that Renovate is configured, for instance to require configuration, explicitly opting out a repository or limiting the enabledManagers, all of which will report different data to that of renovate-graph, which ignores any repo-specific configuration. This is suboptimal data collection, but is still a lot better than other tools on the market, and is better than not using Renovate data.

If you are happy with this, then you can consume these via Dependency Management Data (aka DMD).

How can I use the data?

renovate-graph produces an unminified, but human readable, JSON blob that can be used by other tools for any other purposes.

However, there are a few tools that are best used with renovate-graph, under the Dependency Management Data (aka DMD) project:

• Importing the data via the dmd import renovate command, detailed more under the Getting Started guide
  • This then produces an SQLite database which can be further queried
• renovate-to-sbom, to convert a data dump into a (low fidelity) Software Bill of Materials (SBOM)
• renovate-packagedata-diff, to perform a semantic diff between two data exports

You're also very free to do what you want with it, as it's all open data 🚀

What does the data look like?

You can find a number of example renovate-graph data exports in the dependency-management-data-example project, from a range of public projects.

Why is it called renovate-graph if it doesn't provide a full dependency graph?

You're right, this is a bit of a misnomer 🫣

renovate-graph was chosen as a name at the project's creation, as it sounded like it worked for the project.

In retrospect, this wasn't the best name, but it's where we are right now!

It may be that at some point in the future, we'll start producing a full graph of dependency data, or maybe this project will be renamed. Let's see!

Usage

renovate-graph is available as an executable package, and can be installed with:

# to invoke with `npm exec renovate-graph`
npm i @jamietanna/renovate-graph
# to invoke with `renovate-graph`
npm i -g @jamietanna/renovate-graph
# to invoke with `renovate-graph`
npx @jamietanna/renovate-graph@latest

A single repository

To run renovate-graph against a single repository, and then output a JSON blob, you can run:

renovate-graph --token $GITHUB_COM_TOKEN jamietanna/jamietanna

This will create the file out/github-jamietanna-jamietanna.json.

Autodiscovery (with a filter)

You can also leverage Renovate's autodiscover functionality to automagically discover repositories.

NOTE: using autodiscover requires Write-access to the underlying repository. This due to how renovate-graph leverages Renovate's underlying autodiscovery mechanism, which requires write access to create branches.

You can run the following to find any repositories under github.com/jamietanna/ that the specified GITHUB_COM_TOKEN can access:

renovate-graph --token $GITHUB_COM_TOKEN --autodiscover --autodiscover-filter 'jamietanna/*'

This will create the files out/github-jamietanna-jamietanna, out/github-jamietanna-disARM, etc.

Using the local platform

Since Renovate 35.76.0, it's been possible to use Renovate's local platform to run Renovate against a local directory, without requiring a full Git checkout.

From a directory - possibly without a .git folder - you can then run:

env RG_LOCAL_PLATFORM=gitlab RG_LOCAL_ORGANISATION=jamietanna RG_LOCAL_REPO=jamietanna renovate-graph --platform local

This requires the following environment variables to ensure the metadata is correct:

• RG_LOCAL_PLATFORM: The platform that this local directory is for
• RG_LOCAL_ORGANISATION: The organisation that this local directory is for. Can include a / if a nested organisation
• RG_LOCAL_REPO: The repository name that this local directory is for

Resolving local> presets

If you point Renovate to a repository with the local> preset, such as the below, then Renovate will end up erroring:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "local>oapi-codegen/renovate-config"
  ]
}

This is because Renovate doesn't understand how to resolve this.

However, renovate-graph approaches this differently.

Instead, because renovate-graph knows that i.e. RG_LOCAL_PLATFORM=github, it will then resolve that configuration from:

 {
   "$schema": "https://docs.renovatebot.com/renovate-schema.json",
   "extends": [
-    "local>oapi-codegen/renovate-config"
+    "github>oapi-codegen/renovate-config"
   ]
 }

This will allow you to resolve these presets, while still processing in the local platform.

Specifying a directory for the data

If you wish to export your data to a separate location, you can specify the OUT_DIR environment variable to either a relative or absolute path, such as:

env OUT_DIR=../data renovate-graph --token $GITHUB_COM_TOKEN jamietanna/jamietanna

This will then create the file ../data/github-jamietanna-jamietanna.json.

Running as a GitHub App

To simplify the means to authenticate, as well as more easily discovering repositories that you wish to retrieve data from, this has support for running as a GitHub App.

The App needs to be created with the following permissions:

• Contents: Read only
• Issues: Read only
• Metadata: Read only

Then, when running this, you will need to set the following environment variables:

export RG_GITHUB_APP_ID='...'
export RG_GITHUB_APP_KEY='-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEA4XmSD...'
export RENOVATE_USERNAME='renovate[bot]'
export RENOVATE_GIT_AUTHOR='Renovate Bot <29139614+renovate[bot]@users.noreply.github.com>'

# may only be required if you're using this as a library, not an executable
export RG_GITHUB_APP_INSTALLATION_ID='...'

Additional configuration

As renovate-graph is built on top of Renovate, any of the self-hosted or repository configuration options that apply to Renovate, also apply for renovate-graph.

As noted above, there are some settings that will be ignored by renovate-graph, for instance enabledManagers or packageRules that disable a package.

There are also the following configuration options for further operability:

RENOVATE_REQUIRE_CONFIG=optional / --require-config=optional

The Renovate configuration, RENOVATE_REQUIRE_CONFIG defaults to optional with renovate-graph, but can be overridden.

If any Renovate configuration is set in a repository, renovate-graph will prevent the disabling of managers, which could limit the amount of data that can be retrieved by renovate-graph, while allowing the use of custom regexManagers or otherwise, we explicitly ignore enabledManagers and packageRules in a given repo.

RENOVATE_DRY_RUN=extract / --dry-run=extract

By default, renovate-graph will run under RENOVATE_DRY_RUN=extract, which only determines what dependencies are currently in the repository being scanned.

This means renovate-graph will not detect if there are any pending package updates.

More importantly, this also can lead to not knowing the current version a dependency is resolved at. For instance, if a dependency is pinned to ^35.x, renovate-graph cannot determine that this would resolve to i.e. 35.159.7 until running in RENOVATE_DRY_RUN=lookup, or RG_INCLUDE_UPDATES=true.

This can be modified by overriding this variable to RENOVATE_DRY_RUN=lookup, or setting RG_INCLUDE_UPDATES=true.

If RENOVATE_DRY_RUN is not set, renovate-graph will run under its default, RENOVATE_DRY_RUN=extract.

Using RENOVATE_DRY_RUN=full is treated as RENOVATE_DRY_RUN=lookup.

RG_INCLUDE_UPDATES=false

By default, renovate-graph will run under RENOVATE_DRY_RUN=extract, which only determines what dependencies are currently in the repository being scanned.

This means renovate-graph will not detect if there are any pending package updates.

More importantly, this also can lead to not knowing the current version a dependency is resolved at. For instance, if a dependency is pinned to ^35.x, renovate-graph cannot determine that this would resolve to i.e. 35.159.7 until running in RENOVATE_DRY_RUN=lookup, or RG_INCLUDE_UPDATES=true.

This can be modified by overriding RENOVATE_DRY_RUN=lookup, or setting RG_INCLUDE_UPDATES=true.

As well as parsing the dependency tree, also receive the list of updates that Renovate can see across your dependencies. This is opt-in as it can lead to considerably slower executions of renovate-graph due to needing to hit the network much higher for dependency updates.

This is synonymous with RENOVATE_DRY_RUN=lookup.

RG_DELETE_CLONED_REPOS=false

After checking the dependency data for each repo, immediately delete it. This will slow execution of renovate-graph, but will allow running when using large repositories or against a large number of repositories and organisations that could lead to exhaustion of disk space.

This generally isn't necessary, unless you regularly hit disk space issues.

RG_INCLUDE_LOCKFILES=false

Parse any lockfiles that Renovate discovers for any dependencies that may be referenced.

This is off by default as these aren't necessarily dependencies being used, and can significantly bloat the dependency import.

RG_DO_NOT_FETCH_NON_LOCAL_PRESETS=false

When running in --platform local mode, unlike renovate, renovate-graph will attempt to discover the presets that may be defined on a platform, i.e. github>... or using the local> syntax. However, this may cause issues i.e. if you require specifying your self-hosted instance's --endpoint, and so you can disable this functionality by setting RG_DO_NOT_FETCH_NON_LOCAL_PRESETS=true.

RG_ALLOW_UNPATCHED_DEPENDENCIES=false

renovate-graph patches the underlying Renovate dependency using patch-package, applying a number of changes that tunes certain functionality better for renovate-graph.

If renovate-graph detects that the patching process hasn't successfully worked - for instance, if you're running pnpx @jamietanna/renovate-graph (which is not currently supported) - then renovate-graph will fail to continue.

This prevents the execution of renovate-graph without important patches, which could lead to renovate-graph not processing repositories correctly.

However, it's possible to override this with RG_ALLOW_UNPATCHED_DEPENDENCIES=true to allow this error to be ignored. This isn't recommended, and should only be considered if you're aware of the functionality that is missing if you were to not have these patches applied.

Using renovate-graph as a library

👷🏼 👷🏼 👷🏼 👷🏼

This section is under construction!

👷🏼 👷🏼 👷🏼 👷🏼

License

As this is heavily modified code from Renovate itself, this project is licensed in the same was as Renovate - AGPL-3.0.