near-dupe-pick

Groups near-duplicate photos — bursts, one moment shot by several cameras at different sizes and crops, resaves and rescans — and keeps the best frame of each group, for a named person when a manifest says where their face is. A manifest records what each keeper beat. It also finds the best still in each shot of a video, extracted at full resolution, and the same footage across re-encodes and trims. macOS only (Apple Vision). Originals are only ever read: the output is copies plus a manifest.

It runs on its own on plain folders of photos and videos: grouping, keepers by overall quality, video stills and duplicate footage all work with nothing else installed. It was written as one step of a small pipeline of photo tools (photo-face-find for who is in what, photo-date-fix for when), and reads their manifests when it is given one; see Keeping by one person’s face for the one feature that needs such a manifest.

Why it exists

Two things decide whether a near-duplicate picker leaves a clean set with no repeats, and both are what existing tools get wrong (see Compared with bppicker):

• Grouping that holds across cameras and crops. Apple Vision feature prints (VNGenerateImageFeaturePrintRequest) see the same moment from three cameras, or a 900 px cropped resave of a 2048 px original, as close; perceptual hashes do not.

• Complete linkage, not single linkage. Single linkage joins two groups when any one pair is close, so a session shot frame after frame chains into one cluster: in testing, a set shot by three cameras chained into one cluster of 17. Complete linkage joins two groups only when every cross pair is within the threshold (0.40), so each distinct moment keeps its own frame.

Install

npm install -g near-dupe-pick

Needs macOS, Node 20+, and the Xcode command-line tools (xcode-select --install): the Vision helper (lib/swift/ndp-vision.swift) is compiled with swiftc -O on first run into the work dir, cached by its source hash and CPU architecture, so an Intel and an Apple Silicon Mac each build their own. The video commands need ffmpeg and ffprobe (brew install ffmpeg). Two npm dependencies: exifr and file-type.

Hardened: 2026-10-31 · eff67b1 · gate CLOSED WITH FOLLOW-UPS

Usage

# A folder (or several): group, look, then copy the keepers out.
near-dupe-pick group ~/Pictures/shoot
near-dupe-pick pick  ~/Pictures/shoot --out ~/Desktop/keepers          # dry run
near-dupe-pick pick  ~/Pictures/shoot --out ~/Desktop/keepers --go     # copies + manifest.jsonl

# A manifest with face boxes: keep the frame where one person’s face is best.
near-dupe-pick pick people.jsonl --person jane-doe --review review.json --out picks --go

# Analyse only (fills the cache; resumable on big runs).
near-dupe-pick prints ~/Pictures/archive

# Video: the best still of each shot, or of one in/out range.
near-dupe-pick video-stills ~/Movies/interview.mov --out ~/Desktop/stills --go
near-dupe-pick video-stills ~/Movies/interview.mov --from 00:01:12 --to 00:01:30 --out ~/Desktop/stills --go
near-dupe-pick video-stills ranges.jsonl --person jane-doe --out stills --go   # range rows with face boxes

# Video: the same footage saved several times.
near-dupe-pick video-dupes ~/Movies/exports

Run it from outside the photo and video folders, or pass --work <dir>: the work dir (a cache) defaults to ./.near-dupe-pick-work in the current folder, and a default that would land inside a folder of originals is refused. --out goes outside them too, never beside or below an input file.

An input is a folder or an image file, walked by the stills frame source. It keeps images of at least 4096 bytes (--min-bytes), skips stock libraries and app caches (iPhoto Thumbnails, Previews, Database, node_modules, …), folds byte-identical copies into one item (the manifest keeps every path), and sets zero-filled husks from data recovery aside as skips. A folder it cannot read is a warning. Or the input is a manifest (.jsonl), whose blocks pass through untouched.

Every command speaks four output modes from one item list: human (the default), --brief (one line per item), --json (a stable { tool, version, contract, command, effect, exit, items, summary, next } object) and --quiet (the summary line). Each ends with a next: line of advice. pick and video-stills are dry runs unless --go, and --out may not lie inside a folder of originals: an input folder, the folder of an input file, or the folder of any file a manifest names, symlinks resolved. Exit codes: 0 ok, 1 a finding (an image that cannot be decoded, a folder that cannot be read, a source missing, a name already taken in --out, a review-file merge that did more than it named), 2 usage (checked before any work starts, so nothing is created or hashed), 3 the run could not complete (no swiftc, no ffmpeg, an internal error). --help is the full reference.

Driving it from a script or an agent: read --brief or --json, never the human output. Any line whose verdict is not ok is a finding; a run is clean only when the summary reads warn=0 refuse=0 fail=0. Exit 0 does not mean go: a clean dry run exits 0 and its next: line still says ask, a person’s call before --go.

Keeping by one person’s face

--person <slug> needs a manifest whose still rows say where that person’s face is. The tool that writes those, photo-face-find, is not published, so outside its author’s pipeline you write the manifest yourself (or leave --person out: everything else works on plain folders). One JSON object per line:

{"id":"3f9c0a1b2c3d4e5f","kind":"still","paths":["/photos/IMG_0412.JPG"],"face":{"people":{"jane-doe":{"box":[0.41,0.18,0.12,0.16],"facePx":211,"score":0.62}}}}

• id: the first 16 hex digits of the file’s sha256 (byte-identical copies share it; a row with another id, or an id given twice, is a warning).
• face.people.<slug>.box: the face as [x, y, w, h], normalised to 0–1, top-left origin, on the image as displayed (rotation applied).
• facePx (the face’s size in pixels) and score (how sure the match is) are optional; facePx breaks ties between frames.

A frame with no box for that person ranks below every frame that has one. For video-stills --person, the manifest instead carries kind: "range" rows (fileId, person, in and out in seconds, and bestFrame, a frame row’s id), one per range to take a still from, plus kind: "frame" rows (id <fileId>@<seconds>, fileId, t, paths naming the video) whose face.people.<slug>.box gives the face to track.

How it decides

Grouping. The distance between two photos is the Euclidean distance between their Vision feature prints. It equals VNFeaturePrintObservation.computeDistance, which the suite checks, so the clustering runs in Node from cached vectors. Groups form by complete linkage at --threshold (0.40; at most 0.8, past which pairs are never compared).

The burst rule. Frames captured within 60 s of each other count 0.07 closer, but only when both carry a date block from photo-date-fix with confidence: "high". A camera whose clock was never set, or one hours off at the same event, would otherwise hand slack to frames that were never a burst. Plain folders get no burst slack.

The keeper. A scorer is an adapter in lib/rank.mjs; adding one is one entry in SCORERS.

• person-face (the default with --person): Vision face-capture quality on the named person’s face box from the manifest, plus a small weight for resolution and face size. A frame where the person has no face box ranks below every frame where they do.

• overall (the default otherwise): Vision’s aesthetics score, plus sharpness and resolution relative to the rest of the group. Sharpness is measured at a fixed 512 px scale, so a small resave and its original compare like with like.

The review file (JSON, by id) outranks every scorer.

• merge: [[keeperId, ...ids]] joins the named photos into one group and keeps the first id. Merges are keeper-led: a named photo brings its whole group along only when it is that group’s current keeper (a reviewer names one keeper per group, and its near-duplicates come along); a named photo that is not its group’s keeper moves alone, and the keeper it leaves behind stays a keeper. A merge that takes in photos it did not name, or leaves a keeper it did not name (or a promoted one, or an earlier merge’s) no longer a keeper, is a warning that lists them.

• promote: {id: note} prefers that frame as keeper.

• demote: {id: note} keeps it only when nothing else is left.

• drop: [ids] leaves the photo out entirely.

• accept, reject and any other key belong to other tools and are ignored. Ids the file names that the run never saw are reported as one skip item. A key this tool reads in the wrong shape (a drop that is not a list of ids, say) refuses the file.

A review file belongs to the run it reviewed. Replayed against a re-run whose distances moved near the threshold, a named keeper can bring different companions with it; the merge warnings list exactly those.

Video. Each video is sampled densely (--fps, default 8: the first frame of each bucket by its own timestamp, so variable frame rate keeps true times). A cut opens a new shot where consecutive samples are more than 0.6 apart. Each shot keeps one frame:

• For a person (a range row): their face, tracked as the detected face that best overlaps the range’s face box, ranked by face-capture quality, then sharpness (motion blur), then exposure.

• Otherwise: aesthetics, then sharpness, then exposure.

The shot keepers of each video are then grouped once more, by complete linkage at --threshold, and each group keeps its best: a take that the cut detector split (a flash, a passer-by, a pan away and back) yields one still, not several near-duplicates.

The still is extracted from the source with ffmpeg -ss at that frame’s presentation time, computed from its integer pts and the stream time base. It comes out at full resolution, with rotation applied, and the suite checks it is that exact frame by pixel hash, at 29.97 fps too. The timecode (from the source’s own timecode when it has one) goes in the file name and the manifest row; the frame id is <fileId>@<seconds>.

Duplicate footage. Each file is sampled at --fps (default 2) into a sequence of feature prints. Two files match when, at some start offset, the median distance over their overlap is under 0.45 and the overlap covers at least 80% of the shorter file. The best master is the highest resolution, then bitrate, then duration. Matches join transitively, and a copy that matched another copy but not the master directly says so (via in --json).

The 0.45 was measured on a 27 s 4K HEVC phone clip. Its copies, matched directly: a 720p re-encode 0.264, a 540p copy trimmed 3 s in 0.322. Unrelated clips shot in the same place: 0.588 at the closest, then 0.61, 0.638, and 0.695 and up. A 360p re-encode sits at 0.583 from its master, inside the unrelated range, so it joins only through the 540p copy (0.43).

The manifest

pick --go writes --out/manifest.jsonl: every input row in order, with this tool’s group block added. It replaces any older group block and passes every other block (face, date, anything unknown) through untouched.

{"id":"3f9c0a1b2c3d4e5f","kind":"still","paths":["…/IMG_0412.JPG"],"W":2272,"H":1704,"face":{…},"date":{…},
 "group":{"groupId":"g0042","keeper":true,"beat":["a1b2…","c3d4…"],"reason":"best of 3 by person-face: face-capture quality 0.62 for jane-doe","scorer":"person-face","score":0.71,"copy":"g0042_2009-11-16_IMG_0412.JPG"}}
{"id":"a1b2…","kind":"still",…,"group":{"groupId":"g0042","keeper":false,"keeperId":"3f9c0a1b2c3d4e5f","reason":"beaten by 3f9c…: …","score":0.44,"distance":0.29}}

Keepers are copied byte for byte as <groupId>_<date label>_<name>. A name already taken by a different file in --out refuses that item; the same file already there is reported, not copied again. A manifest that would come out byte for byte the same is left alone, and one from a run that found nothing, or whose every copy was refused, is never written. When some input wasn’t read or analysed (a locked folder, a corrupt video), neither pick nor video-stills replaces an earlier manifest in --out, because the new one would drop those inputs’ rows: fix or remove the inputs the run reports and run again, or move the old manifest out of --out to accept a manifest without them. With no earlier manifest, the run writes one as usual.

video-stills writes kind: "frame" rows with fileId, t, timecode and the same group block, whose still says how the frame was chosen (how, t, shot, score, and for a tracked person their face box and faceQuality). It never writes a face block. Given a manifest, its rows pass through first, in order and untouched, then the stills it had no row for. A still is extracted once into the work dir and copied into --out, so a repeated --go finds the same still from the same frame already there. A run that found no still (every video unreadable, or a range past the end) leaves an earlier manifest as it was.

How it was tested

The regression suite’s fixtures are synthetic: they are derived at test time from a handful of public-domain photographs and from ffmpeg test patterns. No personal photo is in the package or its tests. Each case was tuned to sit clear of the threshold:

• Chaining. Nine crops sliding across one wide frame: neighbours 0.21–0.33 apart, the ends 0.72. Single linkage chains all nine; complete linkage keeps at least three groups, and none of them holds a pair past 0.40.

• One moment, three cameras. Three crops of one frame at 1600, 1200 and 800 px, with a colour shift, a gamma change and a slight rotation (pairwise 0.29–0.35): one group.

• A cropped resave. A 900 px crop of a 2048 px frame (0.26): one group, and the original is kept.

• Two poses that must stay apart. Two crops of one scene at 0.45 stay two groups. The same pair is the burst-rule test: with trusted capture times 30 s apart the 0.07 slack joins them, while medium confidence, five minutes apart, or no dates leave them apart.

• A person keeper. One portrait twice: a large frame with the face blurred, and a small one with the face sharp. With --person the small one is kept (face-capture quality 0.50 against 0.10); without it, the large one.

• A merge replayed on a re-run. The keeper-led cases, on exact distances: a merge naming a photo whose group another photo keeps moves that photo alone; one naming a promoted keeper’s companion leaves the promoted keeper alone; overlapping merges each keep what they named; a later merge that takes in an earlier merge’s keeper replaces it and says so. Through the CLI, naming a burst’s keeper brings its companions along with a warning that lists them.

Also covered: a burst, byte-identical copies, a zero-filled husk, promotes and drops by id, beat[] and every other block passed through untouched, the dry run writing nothing, usage errors before any work, --out inside a folder of originals (an input file’s, a manifest’s, or one reached through a symlink), the default work dir refused inside an input, an unreadable folder, and the video cases. These are a 5 s shot sharpest at 2.5 s, a cut between two shots, a take split by a flash (one still), an in/out range, a face tracked through a range (with the manifest’s rows passed through), frame-exact extraction, a clip whose timestamps start ten hours in (every sampled time checked against its frame, the still against the sampled frame), a repeated --go, an unreadable video and a range past the end (neither replaces a manifest), and a 1080p master with its 480p re-encode and a copy trimmed 2 s in.

Beyond the suite, it was tested read-only on several thousand personal photos (several cameras, scans and resaves), on phone video with its re-encodes, and against a hand-picked set. Nothing failed, and complete linkage kept apart what single linkage at the same threshold chained together. Against the hand-picked set, the --person keeper matched the human pick in all but a few borderline face-quality calls.

Compared with bppicker

The closest existing tool is bppicker (MIT, Python 3.11, v0.1). It was compared from its source and by running both tools on the same labelled set. The verdict was to build, for two reasons:

• Its grouping cannot catch one moment shot by different cameras. It clusters greedily against each cluster’s first member. It compares dHash/aHash inside a 30 s capture-time window or, when it has CLIP embeddings (only in its web library), CLIP cosine: ≥ 0.92 inside the window and ≥ 0.95 outside it. Hashes break under different crops, and the time gate breaks when two cameras’ clocks disagree. On the labelled multi-camera set, its CLIP pass at the default joined one of the three cross-camera groups that near-dupe-pick joins, and its hash pass none; no CLIP threshold from 0.88 to 0.94 joined the cross-camera moments without merging two different moments.

• Its “boost face” adds to a photo’s overall score during top-K selection: +0.05 per named person in the photo, at most +0.15. It does not choose a keeper by one person’s face quality, and the person must first be named in its web UI.

It also writes no manifest of the photos a keeper beat, and takes manual merges only in its UI, not from a file. czkawka and imagededup find near-duplicates but leave the keeper to you and know nothing of faces.

What bppicker has that this lacks: a review UI, face clustering, per-day diversity caps and a signed app. Review pages here can come from contact-sheet-cli pages (1.3.0 or later): --sort date --keepers shows the keepers of a pick manifest in date order.

Known limits

• --person needs a manifest with face boxes, and the tool that writes one, photo-face-find, is not published. Write the manifest by hand in the shape above, or run without --person.

• Same moment, same subjects. Complete linkage on whole-frame feature prints joins frames where a static scene dominates and the subject is small: a series from a fixed camera in which a different person steps into the same spot each time, two crowd shots of different people, several framings of one view. Neither tiled face detection, tile or saliency prints, a people-region crop nor capture time separated these from real cross-camera moments without splitting real moments elsewhere. The likely way forward is a face-identity signal for people, plus a decision on whether different framings of one view are one moment.

• Feature prints match textures and styles, not only content, and the 0.40 threshold was tuned on photographs. Two different synthetic fractal images can come out closer than a real resize; on graphics, scans of text or generated images, check the groups before trusting them.

• Moments within a shot: a long shot that pans across several subjects yields one still. Blur moves feature prints too far (a σ = 2 blur of a burst frame lands 0.62 away) for distance alone to split moments from soft frames.

• Scale: grouping compares all pairs (a few hundred photos take seconds end to end, hashing and Vision included). A library of hundreds of thousands of photos would need blocking first, by folder, by date, or by an approximate nearest-neighbour index.

• Same-camera bursts without photo-date-fix: two frames from one camera body seconds apart are trustworthy relative to each other even when its clock is wrong. Not used yet, so plain folders get no burst slack.

• Drop-frame timecode is counted as non-drop.

• An empty or all-filtered input passes clean for group and prints (exit 0, nothing found), where pick warns not written: nothing found.

• A husk read back from this tool’s own manifest is a warning (cannot analyse), where the same file from a folder is a skip.

• Files under the size floor and non-images are left out without an item saying so.

• Counts read “1 rows” in the manifest line when there is one row.

• Video errors in --brief start with ffmpeg’s or ffprobe’s command line, so the 160-character cut can hide the cause; --json has the full reason.

• No review command: for review pages, run contact-sheet-cli pages <manifest> --sort date --keepers --go directly.

• Not tested on Node 20 itself (the code avoids newer APIs) or on an Intel Mac.

Development

npm test        # node --test: pure grouping rules, Vision + ffmpeg end to end, the CLI contract

The suite builds its fixtures in a temp folder and takes about 1–2 minutes on an Apple Silicon MacBook Pro, most of it the video tests. lib/frame-source/ and lib/util.mjs are vendored from photo-face-find and must not be edited here (lib/VENDORED.md). The runtime block at the bottom of near-dupe-pick.mjs is mjs-tool-skeleton’s, byte for byte.

License

MIT © Danniel T. Gaidula