Confluence Dump Explorer

A tiny program to manipulate a XML dump from Confluence.

Motivations

ICIJ used to build knowledge bases on Confluence for our investigations. Since a few month we migrated to an internal tool that fits better with our needs. Yet the knowledge and documentations accumulated in our Confluence instance are still valuable and we need a way to turn the (huge) dump file from Confluence into a-more-human-readable format (JSON).

JSON files are much more easy to manipulate than a single dump file. With a collection of these tiny files (one for each page), it's quiet easy to build an interface to explore documents or use a tool like Datashare to index and search into the full corpus.

Usage

npm install -g @icij/confluence-dump-explorer

The cde program is now available:

Usage: cde [options] [command]

Options:
  -h, --help                          display help for command

Commands:
  xml [options] <path> [dist]         Analyses and manipulates a Confluence XML dump.
  pages [options] <path> [dist]       Analyses JSON files into `dist/raw`.
  contentbody [options] <dist> <api>  Manipulate existing Content Bodies in JSON files from
                                      `dist/pages`.
  help [command]                      display help for command

These commands are intended to be used in coordination with each other:

1. cde xml --collect ./dump/entities.xml ./dist
2. cde pages --build ./dump/entities.xml ./dist
3. cde contentbody --convert ./dist "http://localhost:8090/rest/api/"

For this example, we use an ephemeral instance of Confluence running inside a docker container:

docker run --name="confluence" --rm -d -p 8090:8090 -p 8091:8091 atlassian/confluence-server

Develop

This tiny utility uses libxmljs to parse XML files. Under the hood, libxmljs uses libxml which is known to have very good performances scores. In our case, we expect these XML dumps to be huge (we tested it with a 500 Mb file) so it makes sense to rely on a C parser, even if the installation if heavier.

To install node dependencies, we recommend using Yarn:

yarn