How fast is it, really

Measured against tar, rsync, scp, croc, rclone and Syncthing — including the corpus where MeshHold loses and the mistakes the benchmark made.

Every number here comes from one two-container stand, run against tar, rsync, scp, croc, rclone and Syncthing. The method, the losses and the mistakes are all below, because a benchmark that reports only the axis you won on is an advertisement.

Where MeshHold wins: one large file over a real WAN, and propagating a change to a folder both machines already hold.

Where it loses: thousands of small files against a plain tar | ssh push, wire efficiency against Syncthing and rsync, and CPU against everything.

The stand

Two containers on one host, same image, same kernel, same CPU quota, so a difference in the numbers is a difference in the protocol. Traffic is shaped with netem inside each container — applied on both sides, because a one-sided delay is not a network.

The corpus is incompressible, so nobody wins on zlib. Two shapes: one 128 MB file, and 5000 files of about 20 KB over a two-level tree. A run is finished when the destination directory holds the expected file count and byte total — never when a tool's own API says so, because those APIs mean different things.

Five runs per arm, both arms inside the same invocation, arm order alternated between runs. Medians below; ranges where they matter.

Versions: Syncthing 2.1.3, croc 11.1.0, rclone 1.75.0, rsync 3.2.7, OpenSSH 9.2.

One 128 MB file

Against tar | ssh — one stream, one cipher, no per-file exchange, and the fastest honest way to push bytes over ssh — MeshHold is 1.28x faster on the WAN. That is the case the architecture was built for, and the one that started this work at 13.8–49.2 s.

On wan-loss the field spreads out and the ranges get wide: 8.1–134 s for MeshHold, 10.9–115 s for tar | ssh. Loss recovery is high-variance. Read that column as an ordering, not as precision.

Five thousand small files

MeshHold loses to tar | ssh here, by about 1.15x, and the reason has not moved across three attempts to fix it: a block fetch is a round trip, and on one-block files the round trip is the transfer. tar | ssh pays none because it never asks — it pushes. Closing that needs the sender to push unbidden, which is a different protocol rather than a better batch.

scp negotiates per file, so five thousand files cost it round trips times file count: 18 s on a LAN, and over 120 ms three runs out of three gave up at the 30-minute mark having moved about 78 % of the data.

croc inverts on file count — respectable on one file, 232 s on five thousand over a gigabit LAN. It is slow there too, so this is not the network: it runs its setup per file, and its discovery beacon fires about fourteen times per file to prove it.

Most of Syncthing's number is fsync

Syncthing calls fsync on every file it finishes. MeshHold did not, until this benchmark made the difference visible. Turning Syncthing's off — which can only make a competitor faster — on the same corpus:

3.7x, and receiver CPU moves by 1.5 s: the other thirty-one seconds were spent waiting, not working. So most of the gap was durability, not protocol. The honest ladder on five thousand small files:

• 7.8x ahead with both tools at their defaults
• 2.5x ahead with neither calling fsync
• 1.14x ahead with both durable

The protocols are close. The defaults were what differed. MeshHold now has the setting too — fsync_writes, per vault, off by default. It costs nothing on a big file (2.09 s to 2.08 s) and 7.7–9.1x on five thousand small ones: one fsync on 128 MB is free, five thousand of them are the entire transfer.

Changing files that both machines already hold

Everything above measures a cold first copy — the only thing a copy tool does, and the worst case for a system that has to build a catalog before it can move anything. What a sync tool actually does from its second day onward is the next sync: 5000 files in place, 50 of them edited.

tar | ssh moves 120x the changed bytes, on both profiles, because it has no notion of change: the whole 147 MB corpus for 1.4 MB of edit. On a gigabit LAN that costs it almost nothing. Over 120 ms it costs twelve seconds.

rclone is last at 28 s — the same transport as rsync and the same delta rule, but it discovers the delta by walking the tree over SFTP, and every stat is a round trip.

rsync wins the LAN and loses the WAN, the same mechanism from the other side: a whole-tree walk is nearly free at 0.3 ms and expensive at 120.

Against MeshHold: our wire cost on a LAN is the worst in the table. x2.2 against Syncthing's x1.06 and rsync's x1.14. It was x5.2 until this benchmark found a catalog sync falling back to a full pull it did not need. About a third of the excess is still there, in a holder refresh that walks every file to read the claims attached to them — linear in catalog size, and not fixed.

What it costs

Three times the CPU on a big file and about eight on small ones. That is not overhead waiting to be optimised away — it is convergent encryption, a content hash per block and a catalog row per file, which is what buys deduplication, resumability, and a receiver that can verify what it got. tar | ssh offers none of those and should be cheaper.

Syncthing uses half our CPU on a big file (2.2 s against 4.4 on a LAN) and beats us on wire efficiency for small files.

Things this benchmark got wrong

Kept, because a method is only as trustworthy as its corrections.

• Syncthing is not single-connection. It opens three TCP connections per peer at its defaults; croc opens six. An early draft would have claimed a multi-stream advantage that does not exist.
• A three-run median is not a measurement. On wan-loss, three runs put tar | ssh at 25.4 s and second-to-last; seven runs put it at 14.4 s and third. Same binaries, same corpus.
• The wan-loss column does not yet test what it was built to test. At 128 MB the transfer is over in about 13 s, too short for a single TCP flow to meet enough loss events to collapse: tar | ssh holds 9.6 MB/s where the Mathis limit predicts 1.25. A longer transfer is needed before this column means what it claims.
• A "regression" that was a busy machine. Small files on a LAN measured 27 % slower after six days of unrelated development. Re-running with Syncthing as a control — an unchanged binary — put both back where they started.

Reproducing it

The stand lives in the repository under bench/, with FINDINGS.md as the full working log: every measurement, every wrong turn, and the reasoning behind each method rule. bench/COMPARISON-PLAN.md states the fairness rules these runs were held to, including the one that matters most — every contender runs at its own defaults, and any tuning is applied along the same axis for everybody, because tuning ours and leaving theirs alone is the easiest dishonest table available.