FOP - Filter Orderer and Preener (Rust Edition)

A Rust port of the EasyList FOP tool for sorting and cleaning ad-blocking filter lists.

Features

• Filter sorting: Alphabetically sorts blocking rules and element hiding rules
• Domain combining: Merges rules with identical selectors/patterns but different domains
• Option normalization: Converts uBO-specific options to standard ABP format (can be disabled)
• Wildcard cleanup: Removes unnecessary wildcards from filters
• Validation: Removes invalid/overly-broad rules (TLD-only, too short, etc.)
• Git integration: Commit changes directly to repositories (can be disabled)
• easylist_adservers.txt validation: Ensures rules start with | or /
• :has-text() merging: Combines rules with same base selector into single regex
• Parallel processing: Processes files in parallel via Rayon, using up to 8 worker threads. Past that, extra workers mostly buy memory rather than speed; set --threads=N (or RAYON_NUM_THREADS) to override

Extended Syntax Support

FOP preserves extended filter syntax from various adblockers:

uBlock Origin

• Scriptlet injection: ##+js(...), ##^script:has-text(...)
• Procedural cosmetics: :has-text(), :has(), :upward(), :remove(), :style(), :matches-css(), :xpath(), :remove-attr(), :remove-class(), :watch-attr(), :min-text-length(), :others()
• Network options: redirect=, redirect-rule=, removeparam=, denyallow=, replace=, header=, permissions=, to=, from=, method=
• Regex domain rules: /regex/##+js(...)

AdGuard

• Scriptlet injection: #%#//scriptlet(...)
• CSS injection: #$#body { ... }
• Exceptions: #@%#, #@$#

Adblock Plus

• Extended selectors: :-abp-has(), :-abp-contains(), :-abp-properties()
• Action syntax: {remove: true;}, {height:...}, {display:...}
• Snippets: #$#hide-if-contains, #$#simulate-mouse-event

Speed Comparison (FOP.py vs FOP Rust)

Installation

npm install -g fop-cli

Building from source (optional)

Requires Rust 1.80+ (https://rustup.rs)

cd fop-rs
cargo build --release
cp target/release/fop /usr/local/bin/  # or add to PATH

Usage

# Sort filters in current directory (with commit prompt)
fop

# Sort filters in a specific directory
fop /path/to/easylist

# Sort without commit prompt (like FOP-nocommit.py)
fop --no-commit /path/to/easylist
fop -n .

# Sort without converting uBO options to ABP format
fop --no-ubo-convert /path/to/ublock-filters

# Sort multiple directories
fop -n ~/easylist ~/easyprivacy ~/fanboy-addon

Command Line Options

| Option | Description | |--------|-------------| | -n, --no-commit | Just sort files, skip Git commit prompts | | --just-sort | Alias for --no-commit | | --no-ubo-convert | Skip uBO to ABP option conversion (keep xhr, 3p, 1p, etc.) | | --no-msg-check | Skip commit message format validation (M:/A:/P:) | | --disable-ignored | Disable hardcoded ignored files and folders for testing | | --no-sort | Don't sort rules, just combine | | --alt-sort | More correct sorting method | | --localhost | Sort hosts file entries (0.0.0.0/127.0.0.1 domain) | | --localhost-files= | Specific files to sort as localhost format (comma-separated) | | --parse-adguard | Globally Parse AdGuard extended CSS (#$?#, #@$?#, $$, $@$) | | --parse-adguard=FILE | Files to parse as AdGuard extended CSS (comma-separated) | | --no-color | Disable colored output | | --no-large-warning | Disable large change warning prompt | | --backup | Create .backup files before modifying | | --keep-empty-lines | Keep empty lines in output | | --ignore-dot-domains | Don't mention rules whose domain has no dot (\|\|cfd^, \|\|countly-). Such rules are kept either way | | --ignorefiles= | Additional files to ignore (comma-separated, partial names) | | --ignoredirs= | Additional directories to ignore (comma-separated, partial names) | | --ignore-all-but= | Only process these files, ignore all others (comma-separated) | | --file-extensions= | File extensions to process (default: .txt) | | --comments= | Comment line prefixes (default: !) | | --warning-output= | Output warnings to file instead of stderr | | --git-message= | Git commit message (skip interactive prompt) | | --history= | Predefined commit messages for arrow key selection (comma-separated) | | --create-pr[=TITLE] | Create PR branch instead of committing to current branch | | --git-pr-branch=NAME | Base branch for PR (default: auto-detect main/master) | | --pr-show-changes | Include rule changes (combines, merges, typos) in PR body | | --fix-typos | Fix cosmetic rule typos in all files during sort | | --fix-typos-on-add | Check cosmetic rule typos in git additions before commit | | --check-rules-on-add | Check git additions for rules that cannot work (empty selector, unbalanced brackets, malformed domain list, unknown or empty option, bare hostname, missing || anchor), and prompt before committing; with --ci it audits the committed diff and exits 1 on a defect | | --remove-bad-rules | Delete defective lines instead of prompting, then commit what is left. Advice (a bare hostname, an unanchored host) is reported but kept, since it is legal syntax (implies --check-rules-on-add) | | --ignore-line-minimum | Keep rules shorter than 3 characters instead of dropping them as malformed | | --auto-fix | Auto-fix typos without prompting (use with --fix-typos-on-add) | | --only-sort-changed | Only process files changed according to git | | --check-banned-list=FILE | Check for banned domains in git additions | | --auto-banned-remove | Auto-remove banned domains and commit | | --ci | CI mode - exit with error code on failures (banned domains) | | --rebase-on-fail | Auto git pull --rebase --autostash and retry when a push fails, up to 5 times when another push keeps landing first. Stops at a merge conflict with instructions to finish it. A commit that could not be published makes FOP exit 1. On by default | | --threads=N | Worker threads for the parallel pool. Defaults to the core count capped at 8; overrides RAYON_NUM_THREADS. | | --no-rebase-on-fail | Don't auto-rebase; print the suggested git pull --rebase command and stop (exit 1) | | --commit-mask=N | Mask URLs in commit messages (1=[.], 2=(.), 3=space, 4=preserve subdomain dot, 5=Unicode lookalike ․) | | --no-commit-mask | Disable URL masking even when .fopconfig sets commit-mask (--commit-mask=0 cannot: 0 falls through to level 1) | | --commit-mask-users=u1,u2 | Restrict --commit-mask to these git config user.name values (lowercased) | | --commit-mask-bare | Also mask bare hostnames without http(s)://. Risks false positives on filenames | | --commit-mask-exempt-hosts=h1,h2 | Additional apex hosts exempt from masking (e.g. self-hosted Gitea/Forgejo/GitLab) | | --commit-url-template=TMPL | Override the Commit successful: URL template. Placeholders: {base}, {sha}. Auto-detects Bitbucket (/commits/); everything else defaults to /commit/ | | --ignore-config | Ignore .fopconfig file, use only CLI args | | --output | Output changed files with --changed suffix (no overwrite) | | --check-file=FILE | Process a single file | | --output-diff=FILE | Output changes as diff (no files modified) | | --quiet | Limit console output, less verbose | | --limited-quiet | Suppress directory listing only | | --add-timestamp=FILES | Update timestamp for specific files only (comma-separated) | | --validate-checksum=FILES | Validate checksum for specific files (exit 1 on failure) | | --validate-checksum-and-fix=FILES | Validate and fix invalid checksums | | --add-checksum=FILES | Add/update checksum for specific files (comma-separated) | | --add-timestamp | Update timestamp in file header (Last Modified/Last Updated) | | --config-file= | Custom config file path | | --show-config | Show applied configuration and exit | | --git-binary=<path> | Path to git binary (default: git in PATH) | | --benchmark[=N] | Time the sort: one untimed warm-up, then N runs (default 5), reporting the median, min and max. Files are left untouched: no diff is built, and timestamps and checksums are not written | | --abp-convert | Convert ABP extended selectors to uBO format | | --adguard-convert | Promote a :has-text() rule's separator to AdGuard's spelling (## → #?#, #@# → #@?#) | | --convert-trusted | Convert trusted scriptlets to non-trusted when value is safe | | -h, --help | Show help message | | -V, --version | Show version number |

Configuration File

Create .fopconfig in your working directory or home directory:

# Skip commit prompt
no-commit = false

# Skip uBO to ABP option conversion
no-ubo-convert = false

# Skip commit message format validation
no-msg-check = false

# Skip sorting (only tidy and combine rules)
no-sort = false

# Alternative sorting method
alt-sort = false

# Sort hosts file entries
localhost = false

# Disable colored output
no-color = false

# Disable large change warning prompt
no-large-warning = false

# Create .backup files before modifying
backup = false

# Keep empty lines in output
keep-empty-lines = false

# Don't mention rules whose domain has no dot
ignore-dot-domains = false

# Comment line prefixes
comments = !

# Output warnings to file
warning-output =

# Additional files to ignore
ignorefiles = .json,.backup,.bak,.swp,.gz

# Additional directories to ignore
ignoredirs =

# File extensions to process
file-extensions = txt

# Create PR branch instead of committing
create-pr =

# Include rule changes in PR body
pr-show-changes = false

# Base branch for PR (default: auto-detect)
git-pr-branch =

# Convert trusted scriptlets to non-trusted when value is safe
convert-trusted = false

# Fix cosmetic typos during sort
fix-typos = false

# Check typos in git additions
fix-typos-on-add = false

# Auto-fix without prompting
auto-fix = false

# Only sort git-changed files (skip unchanged)
only-sort-changed = false

# Path to banned domain list file
check-banned-list =

# Auto-remove banned domains and commit
auto-banned-remove = false

# CI mode - exit with error code on failures
ci = false

# Auto rebase and retry if push fails (default: true).
# On failure fop runs 'git pull --rebase --autostash' and pushes again, which
# reapplies your stashed changes and can leave conflicts to resolve. Set false
# to be told the command to run instead of having it run for you.
rebase-on-fail = true
threads = 8                 # Worker threads; omit to size from the core count

# Mask URLs in commit messages: 1=[.], 2=(.), 3=space, 4=preserve subdomain dot, 5=Unicode lookalike
# github.com, gitlab.com, and codeberg.org (and subdomains) are always exempt so PR/issue links stay clickable.
commit-mask =

# If non-empty, --commit-mask only applies when git user.name matches one of these (case-insensitive).
commit-mask-users =

# Also mask bare hostnames (no http/https). Risks FP on filenames like config.toml.
commit-mask-bare = false

# Additional apex hosts (and subdomains) exempt from masking. Adds to the
# built-in github.com / gitlab.com / codeberg.org list. Use for self-hosted
# Gitea, Forgejo, or private GitLab instances.
commit-mask-exempt-hosts =

# Override the 'Commit successful:' URL template (placeholders: {base}, {sha}).
# Default is auto-detected per host: bitbucket.org gets {base}/commits/{sha},
# everything else gets {base}/commit/{sha}. Override here if your platform
# uses something different (e.g. cgit, trac, custom redirector).
commit-url-template =

# Suppress most output (for CI)
quiet = false

# Users allowed to push directly when create-pr is enabled (comma-separated, case-insensitive)
direct-push-users =

# Predefined commit messages for arrow key selection (comma-separated)
# Use up/down arrows at commit prompt to cycle through these
history = A: ,P: ,M: Update,M: Cleanup,M: Sort,M: Adjust

A .fopconfig in the working directory travels with the repository, so it may not be yours: a pull request can add one. From there FOP ignores git-binary, ignores a warning-output outside the directory, and refuses an output-diff outside it. Set those on the command line, in ~/.fopconfig, or with --config-file. Every other option applies as normal.

Command line arguments override config file settings.

Per-File Overrides

Add [filename] sections to override settings for specific files:

# Global settings apply to all files
fix-typos = true

# Override settings for specific files
[adguard_base.txt]
parse-adguard = true

[hosts.txt]
localhost = true

[easylist.txt]
add-checksum = true
add-timestamp = true

Supported per-file options: no-sort, alt-sort, parse-adguard, localhost, add-checksum, add-timestamp, no-ubo-convert, abp-convert, adguard-convert, convert-trusted, keep-empty-lines, ignore-dot-domains, fix-typos, ignore-line-minimum.

Commit Message URL Masking

--commit-mask=N defangs URLs in commit messages so they're stored in git history in a non-clickable form. Useful for filter-list maintainers who don't want third-party scrapers, search engines, or auto-linkers to follow URLs cited in commit messages. Masking happens after commit-message validation (so A:/P: prefixes still validate against the real URL) but before git commit -m, so the masked form is what lands in git.

Levels

Using https://www.example.com/foo/bar.html and https://www.example.co.nz/foo as inputs:

| Level | Style | www.example.com/... | www.example.co.nz/foo | |---|---|---|---| | 1 | [.] | https://www[.]example[.]com/foo/bar.html | https://www[.]example[.]co[.]nz/foo | | 2 | (.) | https://www(.)example(.)com/foo/bar.html | https://www(.)example(.)co(.)nz/foo | | 3 | space | https://www example com/foo/bar.html | https://www example co nz/foo | | 4 | preserve subdomain dot | https://www.example[.]com/foo/bar.html | https://www.example[.]co[.]nz/foo | | 5 | Unicode ․ (U+2024) | https://www․example․com/foo/bar.html | https://www․example․co․nz/foo |

Notes:

• Path/query/fragment dots are never masked (only host dots), so file extensions like bar.html and query values like ?t=1.2 stay readable.
• Level 4 masks only the registrable domain plus eTLD ("eTLD+1"). Compound TLDs (co.uk, co.nz, com.au, co.jp, com.ng, com.hk, co.il, com.vn, etc. — ~25 country codes) are recognised so example.co.uk is treated as a 2-label TLD. Apex-only inputs (example.com, no subdomain) still defang the only dot.
• Level 5 is the strongest defang: U+2024 is visually nearly identical to . but is not in IDNA UTS #46 normalization tables, so URLs containing it won't resolve in browsers, curl, or most parsers — even after copy/paste. Reviewers won't realise the link is dead until they try to click it.
• Unknown levels (0, 6, 99…) silently fall through to level 1.
• Exempt hosts: github.com, gitlab.com, codeberg.org, and any of their subdomains (gist.github.com, docs.gitlab.com, etc.) are never masked, so PR/issue links remain clickable on the hosting platform. Lookalike hosts like notgitlab.com or evil.gitlab.com.attacker.com still get masked. For self-hosted Gitea/Forgejo/GitLab, add your host with --commit-mask-exempt-hosts=git.company.internal,....

Choosing a level

| When to use | Level | |---|---| | Maximum visibility ("clearly defanged") | 1, 2, or 3 | | Want the host structure to remain readable, only break TLD | 4 | | Want the URL to look real but stay broken (anti-scraper) | 5 |

Restricting masking by user

--commit-mask-users=name1,name2,... (or commit-mask-users = ... in config) gates masking on the current git config user.name. If the list is empty, masking applies to whoever runs fop. If the list is non-empty, only matching users (case-insensitive) get masking; everyone else commits with the original URL.

commit-mask = 4
commit-mask-users = fanboynz, ryanbr

Bare hostnames (--commit-mask-bare)

Off by default. When enabled, also matches hostnames without an http(s):// prefix (e.g. forums.lanik.us or www.example.com.ng). The matcher requires at least one dot and a 2+ letter final label, so version numbers like 1.2.3 are not matched. Filenames like config.toml will match — that's the documented false-positive cost of opting in. Scheme URLs still take priority where both forms appear, and the host-exempt list (github.com, gitlab.com) applies to bare matches too.

Display label

After a successful commit, fop prints Commit message: followed by what landed in git. If masking actually changed the message, the label becomes Commit message (masked): instead, so it's obvious the displayed text is the masked form.

Commit message (masked):  A: https://www.example[.]com/foo
Commit successful:        https://github.com/easylist/easylist/commit/abc1234

If commit-mask is unset, or every URL was github/gitlab (a no-op mask), the plain Commit message: label is shown.

Platform Support

Pre-built Binaries

| Platform | Binary | Optimization | Compatible Devices | |----------|--------|--------------|-------------------| | Linux | | | | | x86_64 | linux-x86_64 | Baseline + PGO | All 64-bit Intel/AMD | | x86_64 | linux-x86_64-v3 | AVX2 + PGO | Intel Haswell+ / AMD Excavator+ (~2015+) | | x86 | linux-x86_32 | Baseline | 32-bit systems, older hardware | | ARM64 | linux-arm64 | Baseline | Raspberry Pi 3/4/5, Orange Pi 3/4/5, all ARM64 | | ARM64 | linux-arm64-n1 | Neoverse N1 | Pi 5, Orange Pi 5, AWS Graviton2+, Ampere Altra | | RISC-V | linux-riscv64 | Baseline | SiFive, StarFive VisionFive 2, Milk-V | | macOS | | | | | Intel | macos-x86_64 | Baseline | All Intel Macs | | Apple Silicon | macos-arm64 | Apple M1 | M1, M2, M3, M4, M5 Macs | | Windows | | | | | x86_64 | windows-x86_64.exe | Baseline | All 64-bit Windows | | x86_64 | windows-x86_64-v3.exe | AVX2 | Intel Haswell+ / AMD Excavator+ (~2015+) | | x86_64 | windows-x86_64-msvc.exe | MSVC + PGO | All 64-bit Windows. A trial build, see below | | x86 | windows-x86_32.exe | Baseline | 32-bit Windows | | ARM64 | windows-arm64-v2.exe | Cortex-A78 | Surface Pro X, Snapdragon laptops |

Which binary should I use?

Linux x86_64:

• Use the baseline. It is built with profile-guided optimisation, trained by sorting test-lists/ so the profile reflects real filter lists, and it runs on every x86_64 CPU: PGO reorders and inlines, it adds no instructions.
• -v3 adds AVX2 targeting on top of the same PGO, worth under 1% here and within measurement noise. The hot path is byte scanning, which memchr already dispatches to AVX2 at runtime whatever the build targets, so the flag only vectorises colder code. Take it if you want to measure it on your own lists.

Windows x86_64:

• Use the baseline. -v3 adds AVX2 targeting, for the same negligible gain as on Linux.
• Neither carries PGO, and not for want of trying: rustup ships no profiler runtime for x86_64-pc-windows-gnu, the mingw target these are built with, so it cannot be instrumented at all. Only x86_64-pc-windows-msvc can.
• -msvc is that build: MSVC rather than mingw, with PGO, and a static CRT so it still needs no VCRUNTIME140.dll. It is published for comparison while the two are measured against each other; npm install continues to fetch the mingw build. Try it if you want the speedup on Windows today.

Linux ARM64 (Raspberry Pi / Orange Pi):

• Use linux-arm64 for Pi 3, Pi 4, Orange Pi 3/4, or if unsure
• Use linux-arm64-n1 for Pi 5, Orange Pi 5, AWS Graviton2+ (~10-20% faster)

npm install downloads baseline binaries for maximum compatibility. Optimized versions are available from GitHub Releases.

Build Details

| Binary | Target | RUSTFLAGS | |--------|--------|-----------| | linux-x86_64 | Native | -Cprofile-use (PGO) | | linux-x86_64-v3 | Native | -C target-cpu=x86-64-v3 -Cprofile-use (PGO) | | linux-x86_32 | i686-unknown-linux-gnu | - | | linux-arm64 | aarch64-unknown-linux-gnu | - | | linux-arm64-n1 | aarch64-unknown-linux-gnu | -C target-cpu=neoverse-n1 | | linux-riscv64 | riscv64gc-unknown-linux-gnu | - | | macos-x86_64 | x86_64-apple-darwin | - | | macos-arm64 | aarch64-apple-darwin | -C target-cpu=apple-m1 | | windows-x86_64.exe | x86_64-pc-windows-gnu | - | | windows-x86_64-v3.exe | x86_64-pc-windows-gnu | -C target-cpu=x86-64-v3 | | windows-x86_64-msvc.exe | x86_64-pc-windows-msvc | -C target-feature=+crt-static -Cprofile-use (PGO) | | windows-x86_32.exe | i686-pc-windows-gnu | - | | windows-arm64-v2.exe | aarch64-pc-windows-msvc | -C target-cpu=cortex-a78 |

Unsupported platform? FOP.rs builds from source on any platform with Rust 1.80+:

cargo build --release

Migrating from Python FOP

This Rust version is a drop-in replacement for both FOP.py and FOP-nocommit.py:

| Python | Rust Equivalent | |--------|-----------------| | python3 FOP.py | fop | | python3 FOP-nocommit.py | fop --no-commit or fop -n |

Performance

The Rust version is significantly faster than Python FOP due to:

• Compiled native code
• Parallel file processing with Rayon
• Optimized regex handling
• Efficient memory management

Windows PowerShell Note

If you see UnauthorizedAccess or PSSecurityException when running fop in PowerShell, run this once to fix permanently:

Set-ExecutionPolicy RemoteSigned -Scope CurrentUser

Alternatively, use Git Bash, MINGW64, or WSL.

Security Warnings

Pre-built binaries aren't code-signed, so your OS may show a warning on first run.

macOS

If blocked by Gatekeeper ("cannot be opened because the developer cannot be verified"):

xattr -d com.apple.quarantine ./fop-mac

Windows

If SmartScreen blocks the file:

1. Click "More info"
2. Click "Run anyway"

Alternative: Install via npm (npm install -g fop-cli) to avoid these warnings entirely.

License

GPL-3.0 (same as original Python FOP)

Credits

• Original Python FOP by Michael (EasyList project)
• Rust port maintains feature parity with Python version 3.9