Tricoteuses-Senat

Retrieve, clean up & handle French Sénat's open data

Requirements

• Node >= 22

Installation

git clone https://git.tricoteuses.fr/logiciels/tricoteuses-senat
cd tricoteuses-senat/

Create a .env file to set PostgreSQL database informations and other configuration variables (you can use example.env as a template). Then

npm install

Database creation (not needed if downloading with Docker image)

Using Docker

docker run --name local-postgres -d -p 5432:5432 -e POSTGRES_PASSWORD=$YOUR_CUSTOM_DB_PASSWORD postgres

Download data

Basic usage

Create a folder where the data will be downloaded and run the following command to download the data and convert it into JSON files.

mkdir ../senat-data/

npm run data:download ../senat-data

Available Commands

• npm run data:download <dir>: Download, convert data to JSON
• npm run data:retrieve_documents <dir>: Retrieval of textes and rapports from Sénat's website
• npm run data:retrieve_agenda <dir>: Retrieval of agenda from Sénat's website
• npm run data:retrieve_cr_seance <dir>: Retrieval of comptes-rendus de séance from Sénat's data
• npm run data:retrieve_cr_commission <dir>: Retrieval of comptes-rendus de commissions from Sénat's website
• npm run data:retrieve_senateurs_photos <dir>: Retrieval of sénateurs' pictures from Sénat's website

Filtering Options

Downloading all the data is long and takes up a lot of disk space. It is possible to choose the type of data that you want to retrieve to reduce the load.

Examples:

# Only download amendments
npm run data:download ../senat-data -- -k Ameli

# Only process data from session 2023 onwards
npm run data:download ../senat-data -- --fromSession 2023

Common Options

• --categories or -k <name>: Filter by dataset categories (Available options: All, Ameli, Debats, DosLeg, Questions, Sens)
• --fromSession <year>: Specify the session year to retrieve data from (default: 2022)
• --dataDir <path> (Mandatory): Path to the working directory where all data is stored (required)
• --silent or -s: Disable logging
• --verbose or -v: Enable verbose logging
• --commit or -c: Automatically commit converted data
• --pull or -p: Pull repositories before starting
• --clone or -C <url>: Clone Git repositories from a remote group or organization
• --remote or -r <name>: Push commits to specified Git remote(s)
• --keepDir: Keep directories when cleaning data
• --only-recent <days>: Retrieve only documents created within the last N days

Options for Retrieving Documents

• --formats <format>: Specify document formats to retrieve (options: xml, html, pdf)
• --types <type>: Specify document types to retrieve (options: textes, rapports)
• --parseDocuments: Parse documents after retrieval
• --parseAgenda: Parse agenda after retrieval
• --parseDebats: Parse comptes-rendus after retrieval

Examples

# Retrieval of textes and rapports in specific formats
npm run data:retrieve_documents ../senat-data -- --fromSession 2022 --formats xml pdf --types textes

# Retrieval & parsing (textes in xml format only for now)
npm run data:retrieve_documents ../senat-data -- --fromSession 2022 --parseDocuments

# Retrieval & parsing of agenda
npm run data:retrieve_agenda ../senat-data -- --fromSession 2022 --parseAgenda

# Retrieval & parsing of comptes-rendus de séance
npm run data:retrieve_cr_seance ../senat-data -- --parseDebats --keepDir

# Retrieval & parsing of comptes-rendus de commissions
npm run data:retrieve_cr_commission ../senat-data -- --parseDebats --keepDir

Data download using Docker

A Docker image that downloads and converts the data all at once is available. Build it locally or run it from the container registry. Use the environment variables FROM_SESSION and CATEGORIES if needed.

docker run --pull always --name tricoteuses-senat -v ../senat-data:/app/senat-data -d git.tricoteuses.fr/logiciels/tricoteuses-senat:latest

Use the environment variable CATEGORIES and FROM_SESSION if needed.

Using the data

Once the data is downloaded, you can use loaders to retrieve it. To use loaders in your project, you can install the @tricoteuses/senat package, and import the iterator functions that you need.

npm install @tricoteuses/senat

import { iterLoadSenatQuestions } from "@tricoteuses/senat/loaders"

// Pass data directory and legislature as arguments
for (const { item: question } of iterLoadSenatQuestions("../senat-data", 17)) {
  console.log(question.id)
}

Generation of raw types from SQL schema (for contributors only)

npm run data:generate_schemas ../senat-data

PostgreSQL import modes

retrieve_open_data.ts supports two import modes:

• default mode: direct import in a single database
• --incremental: separate staging database plus postgres_fdw bridge into the target database

The default mode is the simplest one and is the one used by the Docker image. The incremental mode is intended for replicated environments where heavy transient work should stay outside the target database.

Incremental PostgreSQL import architecture

When --incremental is enabled, the open-data SQL import is designed to keep heavy transient work out of the replicated target database.

                       +-----------------------------------+
                       |  Source Open Data Senat           |
                       |  ZIP -> SQL dumps                 |
                       +----------------+------------------+
                                        |
                                        v
                       +-----------------------------------+
                       |  PostgreSQL staging instance      |
                       |  non-replicated                   |
                       |  database: senat_staging          |
                       |                                   |
                       |  schemas:                         |
                       |  - ameli_staging                  |
                       |  - debats_staging                 |
                       |  - dosleg_staging                 |
                       |  - questions_staging              |
                       |  - sens_staging                   |
                       |                                   |
                       |  heavy operations:                |
                       |  - raw dump import                |
                       |  - table renaming/prefixing       |
                       |  - staging indexes                |
                       +----------------+------------------+
                                        |
                             postgres_fdw (read-only bridge)
                                        |
                                        v
+----------------------------------------------------------------------------+
| PostgreSQL target instance, replicated                                     |
| database: canutes                                                           |
|                                                                            |
| preserved schemas: assemblee, legifrance, ...                              |
| updated schema: senat                                                      |
|                                                                            |
| lightweight temporary FDW schemas:                                         |
| - ameli_staging                                                            |
| - debats_staging                                                           |
| - dosleg_staging                                                           |
| - questions_staging                                                        |
| - sens_staging                                                             |
|                                                                            |
| final operations only:                                                     |
| - read staging data through postgres_fdw                                   |
| - incremental merge into canutes.senat                                     |
| - optional schema alignment when source structure changes                  |
+----------------------------------------------------------------------------+

At the end of an incremental import:

• the target database is canutes
• the target schema is senat
• other schemas in canutes are left untouched
• bulk transient data stays in the staging database

Direct single-database mode

Without --incremental, the script works directly in the configured target database:

• dumps are imported into temporary *_staging schemas in that same database
• the final tables are merged into schema senat
• the temporary staging schemas are dropped at the end

This mode is appropriate for:

• local development
• disposable databases
• the Docker image workflow
• environments where replication cost is not a concern

Preparing postgres_fdw and database rights

This section only applies when using --incremental.

The target database must be able to connect to the separate staging database with postgres_fdw.

Environment variables

Set the target connection in .env. Add the staging connection only if you use --incremental:

# Target database
DB_HOST="localhost"
DB_PORT=5432
DB_USER="postgres"
DB_PASSWORD="PASSWORD"
DB_NAME="canutes"

# Separate non-replicated staging database or instance, only for --incremental
STAGING_DB_HOST="localhost"
STAGING_DB_PORT=5433
STAGING_DB_USER="postgres"
STAGING_DB_PASSWORD="PASSWORD"
STAGING_DB_NAME="senat_staging"

Typical commands

Default direct mode:

npm run data:retrieve_open_data -- ../senat-data --all

Incremental mode with separate staging database:

npm run data:retrieve_open_data -- ../senat-data --all --incremental

Target database prerequisites

On canutes, the import role must be able to:

• connect to the database
• create the postgres_fdw extension, or reuse it if already installed
• create and drop FDW servers and user mappings
• create and drop temporary *_staging schemas used for foreign tables
• create, alter and drop objects inside schema senat

Typical one-time setup on the target database:

CREATE EXTENSION IF NOT EXISTS postgres_fdw;

Staging database prerequisites

The staging PostgreSQL instance should ideally be outside the replicated cluster.

The staging role must be able to:

• recreate the senat_staging database
• create schemas and import dumps there
• create the technical staging indexes

The target instance must also be allowed to connect to the staging instance over the network. In practice:

• open the staging host and port from the target server
• allow the staging user in pg_hba.conf
• ensure the staging credentials used in .env are valid

Operational note

The replicated target still sees a small amount of temporary DDL for FDW objects, but the large dump imports and staging tables remain outside the replicated database. Most replicated volume should therefore come from the real incremental changes applied to canutes.senat.

Validation of prefixed SQL imports

After importing datasets with prefixed tables in schema senat, you can verify that the expected renamed tables exist and still match the generated definitions:

npm run data:validate_prefixed_tables -- --categories All

Publishing

To publish a new version of this package onto npm, bump the package version and publish.

# Increment version and create a new Git tag automatically
npm version patch   # +0.0.1 → small fixes
npm version minor   # +0.1.0 → new features
npm version major   # +1.0.0 → breaking changes
npx tsc
npm publish

The Docker image will be automatically built during a CI Workflow if you push the tag to the remote repository.

git push --tags