PhET Simulations scraper

This scraper creates offline versions in ZIM format of PhET science simulations for Science and Math.

Requirements

It requires Node.js version 16 or higher.

Quick Start

npm i && phet2zim

The above will eventually output a ZIM file to output/

Command line arguments

See phet2zim --help for details.

phet2zim --output generates ZIM files in a specific folder.

phet2zim --output myFolder

--withoutLanguageVariants uses to exclude languages with Country variant. For example en_CA will not be present in zim with this argument.

--subjects is used to pass specific subjects to download. Pass values as csv. Sample of valid subjects :

physics, biology, earth-science, motion, sound-and-waves, work-energy-and-power, heat-and-thermodynamics, quantum-phenomena

Available only on GET step:

--withoutLanguageVariants ...

Available on GET and EXPORT steps only:

--includeLanguages 'lang_1,lang_2,lang_3' ...
--excludeLanguages 'lang_1,lang_2,lang_3' ...
--subjects 'math,physics' ...

Available on EXPORT step only:

# Skip ZIM files for individual languages
--mulOnly

# Create a ZIM file with all languages
--createMul

Example:

phet2zim --includeLanguages en,ru,fr

Config

Another way to configure behaviour is through environment variables. Sample .env file (with default values):

# request per second, affects GET step only
PHET_RPS=8
# async workers on TRANSFORM step (keep it equal to number of CPU cores)
PHET_WORKERS=10
# number of retries on GET step (delay grow with exponential backoff)
PHET_RETRIES=5
# display verbose errors
PHET_VERBOSE_ERRORS=false

About

This project achieves multiple things:

• Download PhET content
• Generate an Index for said content
• Generate ZIM file(s) containing content and index

Things this project does not yet do, but should:

• Generate Android APK

Usage

The functionality is split into 5 npm scripts:

• npm run setup - deletes state from previous runs
• npm run get - downloads PhET simulations in specified languages
• npm run transform - prepare the content and media files
• npm run export - generates ZIM file(s)
• npm start - runs all of the above in sequence

The steps get, transform and export have their own output directories:

• get outputs HTML and PNG files to state/get
• transform outputs intermediate files to state/transform
• export outputs HTML and PNG files to state/export AND a ZIM file(s) to output/ (by default, unless customized with --output)

License

Apache or later, see LICENSE for more details.