csv-grep-rinse

A rules-driven text/CSV transformer: point it at a JSON list of regex rules (plus a few built-in case-conversion modes) and it streams an input file — or a whole glob of them — through those rules, one logical record at a time, into pipe-delimited output — or any other delimiter, via --delimiter.

It grew out of reformatting large CSV exports for downstream import tools: anonymizing columns, reformatting dates, normalizing case on organization names, and re-delimiting comma-separated data with | — all driven by a config file instead of a one-off script, and able to handle files with tens of thousands of rows (including quoted fields with embedded commas and hard line breaks) without mangling records.

node index.js -r rules.json -i input.csv -o output.txt
node index.js -r rules.json -i "data/*.csv" -S -x txt   # batch, one output per input
node index.js --help

Install

npm install -g csv-grep-rinse
csv-grep-rinse -r rules.json -i input.csv -o output.txt

Or run it without installing:

npx csv-grep-rinse -r rules.json -i input.csv -o output.txt

Node 18+ required (uses util.parseArgs).

What it's good for

Any pipeline step where you need to reshape rows of text/CSV using a declarative set of find-and-replace or case rules instead of hand-editing a one-off script each time the source data changes: scrubbing PII, normalizing inconsistent casing from an upstream export, converting date formats, re-delimiting a comma file to pipes for a downstream importer, or injecting synthetic timestamps into records that need one.

It reads each input file whole and walks it quote-aware — a newline inside a quoted CSV field becomes a space (keeping the record intact) and a newline outside quotes ends the record — so multi-line description/notes fields don't get split into mangled rows the way a plain line-by-line reader would.

The tool itself never splits or joins fields — re-delimiting a record (turning commas into pipes, say) is still your rules file’s job, e.g. a rule whose pattern turns unquoted commas into pipes using a lookahead that skips commas inside quoted fields. What the tool does guarantee, unconditionally and before your rules ever run, is narrower and non-negotiable: any literal occurrence of whatever delimiter you’re targeting is neutralized in the raw record first, so it can never be mistaken for a real field boundary downstream — see Field notes below.

By default that target delimiter is |, and a literal | already present in the data (e.g. "ACME | DBA Foo") is neutralized to /. Pass --delimiter <char-or-string> to target a different delimiter instead — ,, ;, and \t (a real tab) are the common cases, but any string is accepted:

node index.js -r rules.json -i input.csv -o output.txt --delimiter '\t'
node index.js -r rules.json -i input.csv -o output.txt --delimiter ';'

Each delimiter has a sensible default neutralization replacement (see Delimiters below); override it with --neutralize-with <string> if the default doesn’t fit your data.

Usage

Usage: node index.js [options]

Options:
  -r, --rules <path>      Path to rules JSON file (default: rules.json)
  -i, --input <pattern>   Input file pattern (e.g., "data/*.csv")
  -o, --output <path>     Path to output text file (Ignored if --separate is used)
  -S, --separate          Output to separate files instead of merging.
      --suffix <string>   Suffix for separate files (default: "_processed")
  -x, --extension <ext>   New extension for output files (e.g., "txt").
  -e, --encoding <type>   File encoding: 'utf8' (default) or 'latin1'.
      --delimiter <str>   Output delimiter to protect (default: "|"). Common
                          values: ",", ";", "\t", "|" -- any string accepted.
                          Escapes \t \n \r \\ are expanded.
      --neutralize-with <str>
                          Replacement for a literal delimiter found in source
                          data (default depends on --delimiter -- see README).
  -d, --date <string>     Start date/time (e.g., "2025-12-14 12:00:00").
  -k, --keep-date         Prevent calendar date rollover.
  -h, --help              Show this help message

• -i/--input accepts a glob, so it can process one file or a whole directory ("data/*.csv") in a single run.
• By default all matched input files are concatenated into a single -o/--output file. Pass -S/--separate to instead write one output file per input, named <original-name><suffix><ext> next to the source file (--suffix default _processed; -x/--extension overrides the output extension, e.g. turning .csv into .txt).
• -e/--encoding controls how the input is read (utf8 or latin1); output is always written as utf8.
• --delimiter and --neutralize-with control delimiter protection — see Delimiters below.
• -d/--date seeds a counter used to fill in a {{TIMESTAMP}} placeholder (see below) — each record gets the seed time plus one second. -k pins the seeded date's year/month/day so the counter only rolls the hour/minute/second forward instead of drifting into the next calendar day over a long run.
• A rule that reduces a line to nothing (e.g. a "delete this row" pattern) causes that line to be skipped in the output rather than emitting a blank row.

Delimiters

--delimiter <char-or-string> (default |) tells the tool which delimiter it’s protecting. Before any rule runs, every literal occurrence of that delimiter already present in the raw record — anywhere, including inside quoted fields — is replaced with a neutralization string, so it can never later be mistaken for a real field boundary. This is the same mechanism the tool has always used for |, generalized to any delimiter you name.

Escapes are expanded so you can pass a real tab from the shell:

--delimiter '\t'    # a real tab character
--delimiter ','      # comma
--delimiter ';'      # semicolon
--delimiter '|'      # pipe (the default)

\t, \n, \r, and \\ are recognized; anything else passes through literally, so multi-character delimiters (--delimiter '::') work too.

Neutralization defaults

Each delimiter neutralizes to a different replacement by default, because / (the original pipe substitute) doesn’t read sensibly for every delimiter:

| Delimiter | Default replacement | Why | |---|---|---| | \| | / | Original, production-proven default — an org name legitimately contains ACME \| DBA Foo far more often than a slash used as a real separator, and / reads naturally in place of a pipe. | | \t, ,, ; | (space) | The literal character showing up in data is usually incidental (a stray tab from a pasted spreadsheet cell, a decimal comma, a semicolon in prose) rather than structural, and a slash would read oddly in its place. | | anything else | (space), or / if the delimiter itself is a space | Reasonable general-purpose fallback. |

Override with --neutralize-with <string> if none of these fit your data (it also accepts the same \t/\n/\r/\\ escapes). The tool refuses to start if --neutralize-with is set to the same value as --delimiter — that would defeat neutralization entirely.

Comma as the output delimiter

Choosing --delimiter ',' runs headlong into the classic CSV problem: your input is very likely already comma-structured, with quoted fields that legitimately contain commas. This tool does not implement RFC-4180 quoting on output — it never has, even for the default pipe delimiter — so the decision here is the same one it’s always made for |, applied honestly to ,: every literal comma in the record is neutralized (replaced with a space by default), including commas that were legitimate CSV field separators or inside quotes. That means choosing a comma output delimiter only makes sense once your rules have already re-delimited the record into its final comma-separated shape — the neutralization step exists to protect that output structure from stray commas still lingering in the data, not to preserve the input CSV’s original comma structure through to the output.

If you need real quote-aware comma output (i.e., commas inside quoted fields preserved as data, only unquoted commas treated as delimiters), that is proper CSV quoting and this tool deliberately doesn’t do it — reach for a real CSV library instead.

The rules file

-r/--rules points at a JSON array of rule objects, applied to every record in order. Each rule matches a pattern against the record and either replaces it with a literal replacement, or applies a built-in case transform to whatever the pattern matched:

[
  { "pattern": "apple", "replacement": "orange", "flags": "gi" },
  { "pattern": "(\\d{4})-(\\d{2})-(\\d{2})", "replacement": "$2/$3/$1", "flags": "g" },
  { "pattern": "^User: .*", "replacement": "User: ANONYMOUS", "flags": "gm" }
]

| Field | Meaning | |---|---| | pattern | A JS regex source string (no slashes), passed to new RegExp(). | | replacement | Replacement string, supports $1/$2 capture groups — used when none of the case flags below are set. | | flags | Regex flags. g is always forced on even if omitted. m is stripped, since each record is matched as a single line regardless of embedded line breaks. | | lowercase | true → the matched text is lowercased instead of using replacement. | | uppercase | true → the matched text is uppercased instead of using replacement. | | titlecase | true → the matched text is smart title-cased (see below). |

Rules with a regex error are silently skipped so one bad pattern doesn't crash a batch run.

Smart title case

titlecase isn't a naive "capitalize every word" — it's tuned for messy organization-name and place-name data, and it decides case per word by shape, not by looking at whether the rest of the matched string is also all-caps.

That per-word-by-shape design replaced an earlier version that decided case per word based on two global signals: whether the whole matched string was "screaming" (100% caps), and a bare word.length > 4 threshold. Both turned out to be unsafe signals in production data (FVD-26-1361 WFMF casing audit, 2026-07): when the source organization name was itself entered fully caps (e.g. PS 81), the whole-string screaming check forced every word — including short ones — to lowercase first, mangling PS 81 → Ps 81 and P.S. 108 → P.s. 108; independently, any all-caps word over 4 characters got lowercased even inside an otherwise normal mixed-case string, mangling ... STEAM Academy → ... Steam Academy regardless of context. A "the whole match/record is screaming" flag is not a safe per-word casing signal — always decide per-word by shape (digits present? dotted-initialism shape? already intentionally mixed case? on an allowlist?), never by what the rest of the string looks like.

Current rules, applied per word (see titleCaseWord in index.js):

• A token with a digit in it (PS81, R-IV, USD446) is treated as a code and never re-cased.

• A dotted-initialism shape (P.S., C.U.S.D., S.T.E.M.) is detected and uppercased regardless of its current case — this is what lets an already-broken p.s./P.s. self-heal back to P.S. on a re-run, and it's checked before the mixed-case guard below so it isn't mistaken for a proper noun.

• A token that already has both upper and lower letters (McGlone, DeKalb, O'Brien, Wilkes-Barre) is left completely untouched. This is what protects legitimate mixed-case proper nouns — it also means the engine can't invent internal caps it doesn't already see; see CASE_EXCEPTIONS below.

• A small list of connector words (and, the, of, in, for, a, an, to, at, by, on) stays lowercase unless it's the first word.

• ACRONYM_ALLOWLIST (near the top of index.js) is kept uppercase regardless of length — INC/INC. specifically becomes Inc. (without double-perioding a source INC. that already had its own trailing period). Extend this array for your dataset's 4+ letter acronyms (it shipped with KIPP, STEAM, YMCA, USD, NAACP, etc. from the WFMF dataset — trim or extend as needed).

• CASE_EXCEPTIONS is a small hand-maintained map for proper nouns whose internal capitalization can't be derived by any general rule (e.g. LAGRANGE → LaGrange, DEKALB → DeKalb). Mc- is auto-repaired inline (MCALLEN → McAllen) because no common English word starts with Mc, but Mac-/La-/Le-/De-/Di-/Van-/Von- are deliberately not auto-repaired — Macon, Machias, Lafayette, Lansing, Denver, Detroit, Decatur are ordinary names/words with no internal cap, so a blanket prefix rule would produce MacOn, LaFayette, DeTroit. Add exceptions to the map by hand as you find them instead of guessing with a regex.

• Two modes, because the right default for a short (≤3 letter) ALL-CAPS token depends entirely on what field you're matching — there is no context-free default that's correct for both:

  • "titlecase": true → title mode (org/school names). Short ALL-CAPS tokens default to staying uppercase, since in this domain they're overwhelmingly genuine initialisms (PS, IS, MS, HS, ES, SD, DC, USD, ISD...) that would be impractical to enumerate exhaustively.
  • "titlecase": "city" → city mode, for a Loc:/city field. Short ALL-CAPS tokens default to normal Title Case instead, because in a city name they're almost always ordinary words that happen to be short (NEW York, SAN Francisco, LOS Angeles, LAS Vegas, DES Moines, ST Louis, LA Jolla, EL Paso, MT Pleasant, FT Lauderdale) rather than abbreviations. Only CITY_ABBREV_ALLOWLIST (currently just LIC, OKC — genuinely spoken-as-letters city initialisms) stays uppercase. Using title mode's default here by mistake is what produces bugs like NEW YORK → NEW York instead of New York — easy to get backwards once; verify against every distinct short token actually present in your data before shipping a city-casing rule, the way the WFMF audit did.

  Example rule pair for a pipe-delimited record with an embedded Loc: <city>__<ST> field:

  [
    { "pattern": "^[^|]+", "titlecase": true, "flags": "g" },
    { "pattern": "(?<=Loc: )[A-Za-z][A-Za-z .,'-]*(?=__)", "titlecase": "city", "flags": "g" }
  ]

Known-bad city values (mode "city" only)

Two guards run before the normal per-word title-case pass, because a truncated/abbreviated/wrong-column city value isn't something shape-based per-word casing can fix or should silently paper over:

• CITY_EXPANSIONS — a hand-maintained, append-as-found map (uppercased-key → correct spelling) of USPS postal abbreviations and other truncations that have actually turned up mis-encoded as a city name in production data — e.g. SN LUIS OBISP → San Luis Obispo, SN BERNRDNO → San Bernardino. These are matched whole-phrase and replaced outright, bypassing the per-word pass entirely, because per-word shape rules can't recover San from SN — there's no general rule for reversing postal truncation, only a growing list of specific cases as they're found. Seeded from the FVD-26-1361 WFMF casing audit (2026-07); extend the map by hand as new mis-encodings surface.
• Implausible-value warning — if the (trimmed, uppercased) city value is US/USA/U.S./U.S.A. or a bare 2-letter US state code, that's almost certainly the wrong column landing in the city field (a country or state code, not an abbreviated city), not something title-casing can repair — the real city is simply missing from the record. Rather than silently producing a plausible-looking-but-wrong city (US → Us), the value is left untouched and a console.warn is emitted flagging it for manual review.

Recommended longer-term fix — this class of bug (abbreviations, truncations, and wrong-column values like US) is fundamentally a missing-cross-check problem: the city field was never validated against anything. The systematic fix is to derive/validate the city from the record's zip code using Dan's offline-csv-geocoder (zip → city lookup) — that would catch all three failure modes at once, instead of accumulating them one CITY_EXPANSIONS entry at a time. Not implemented here; this is a recommendation for the next round of work on this tool, not something CITY_EXPANSIONS/the warning are meant to replace.

Timestamp injection

Include the literal placeholder {{TIMESTAMP}} anywhere in a rule's replacement and, when -d/--date is passed, it's swapped for a YYYY-MM-DD HH:MM:SS string that increments by one second per output record — handy for backfilling a fake but monotonically increasing created_at column for a bulk import.

Field notes

Lessons learned running this tool's production ancestor, processFile.js, against real, messy data — most recently a 14,752-record grant-database job (FVD-26-1361, 2026-07). None of this is client data; the specifics have been genericized into rules.example.json.

• A literal | in your source data is fatal to a pipe-delimited intermediate format, and it will show up. An org name like "ACME | DBA Foo" is exactly the kind of thing real data contains. The production fix was to neutralize it upstream — map | → / — before any rule runs, which is why the tool does that unconditionally rather than leaving it to a rule (see Delimiters above). If your downstream delimiter isn't |, pass --delimiter and the same protection applies to that character instead — don't discover the corruption after the fact in row 9,000-something.
• Curly typographic punctuation (’ ‘ " ") survives downstream importers fine — don't flatten it to straight quotes "for safety." It's tempting to strip punctuation down to plain ASCII on the theory that some importer somewhere might choke on a smart quote; in practice the opposite problem is far more common (straight quotes left over from a bad copy/paste read as unstyled and cheap) and no importer encountered in production choked on curly punctuation. Curl it deliberately instead — see the typography rules appended to rules.example.json.
• Rules run in order, so repair rules must precede formatting rules. A mojibake repair (fixing a caf� → café-style encoding casualty) has to run before any titlecase/case-conversion rule touches that field — once the U+FFFD replacement character has been case-converted or re-matched by a later rule, there's no way to tell it used to be an accented letter. rules.example.json orders its header-stripping and mojibake-repair rules first for exactly this reason; the punctuation-curl rules run last, after the content they're curling already has its final shape.

Worked example

examples/input.txt and examples/output.txt show rules.example.json applied end to end — a plain find/replace, a date reformat, and a line-anonymizing rule. (rules.example.json also carries a few illustrative header-stripping, mojibake-repair, and punctuation-curling rules from the Field notes above that don't happen to match anything in this particular input file, but show the shape of those fixes and — critically — where they sit in the rule order.)

$ node index.js -r rules.example.json -i examples/input.txt -o /tmp/out.txt
Found 1 file(s) to process.
[1/1] Appending examples/input.txt...

✅ Done! Processed 6 lines.

examples/input.txt:

I have one Apple and two apples.
The date is 2023-10-27.
User: John Doe
Another line with an apple.
Date of birth: 1990-05-15
User: Jane Smith

examples/output.txt:

I have one orange and two oranges.
The date is 10/27/2023.
User: ANONYMOUS
Another line with an orange.
Date of birth: 05/15/1990
User: ANONYMOUS

history/

history/ keeps the earlier iterations (processFile_v1.js through _v4.js) this tool evolved through — from a one-shot whole-file replace() to the streaming, quote-aware, case-rule version now in index.js. Kept for reference; not part of the published tool.

License

MIT.