ericr.org

SmartCopy: copying files without copying history

Years of backups of backups left me with the same folders copied into five different places. SmartCopy compares by content, skips what's already there, and writes down every decision.

2 min read

Painted illustration of a hall of mirrors repeating the same stack of folders, with a glowing magnifying glass in front.

Every long-lived home network has a folder like this somewhere:

Backup/
  OldLaptop/Projects/...
  OldLaptop-2/Projects/...
  From D/Projects/...
  Sep2026/FromD/Projects/...
  Projects (copy)/...

Each copy made sense on the day it was made. Put together, they're a mess: mostly the same trees, a few genuinely different files, and timestamps that are useless because every copy tool has rewritten them at least once.

I wanted to consolidate all of it onto one big drive without making things worse. The usual tools don't quite do it. cp and robocopy happily copy the fifth identical tree. Sync tools compare against one destination path, not “does this already exist anywhere over there?” So I wrote SmartCopy.

smartcopy <source> <destination> [--ssd] [--skipdevbin]

Content, not timestamps

The rules are small and strict:

  • A whole directory tree is compared by relative paths, file lengths, and file contents. Timestamps and attributes are ignored, because I've learned not to trust them.
  • If a complete matching tree already exists anywhere in the destination, even under a different parent, it's skipped.
  • If a tree has changed at all, it's copied normally. Individual files inside it are never skipped just because a matching file exists somewhere else. I decided to deduplicate at the directory level so the result still looks like the folders I remember.
  • Large archives (anything over 100 MiB with a known extension like .zip, .7z, .iso, or .zst) are the exception. Those are compared globally, because five copies of the same 4 GB installer ISO are the whole reason the drive is full.

Copying keeps the source's top-level directory name, so smartcopy ~/Restore/Projects /mnt/archive lands in /mnt/archive/Projects instead of spraying files into the root.

Asking instead of guessing

When a different file is already sitting at the destination path, SmartCopy stops and shows both sides (name, size, last-write time) and lets me skip, overwrite, create a numbered conflict name, or cancel. The choice can apply to everything after it. If input is redirected and nobody's there to answer, an unresolved conflict is an error rather than a silent guess.

A paper trail

The destination root gets three files that make the whole run auditable:

File What it records
manifest.json Cached hashes, so re-runs don't re-read everything
duplicates.txt Every directory and archive that was skipped, and what it matched
merge_conflicts.txt Every conflict decision that was made

That last point matters more than it sounds. The scary part of deduplication isn't the copying, it's wondering later whether something got skipped that shouldn't have. With the paper trail I can check afterward instead of worrying.

Two small switches

--ssd turns on bounded four-way parallel hashing and copying. On spinning disks it's slower (the heads just thrash), so it's opt-in. --skipdevbin skips the bin and obj folders next to recognized .NET project files, because nobody needs build output from 2014.

The same flag, down to the -sdb shorthand, is in my backup tool PatternBuffer.NET. Small utilities and big ones keep borrowing ideas from each other around here.