This week had one question behind it: how fast is MeshHold, really? Not "does it feel quick," but a number, measured against the tools people already trust. So I built a benchmark stand, raced it against the classics, and spent the rest of the week fixing what the stopwatch exposed — including a fair few theories I was certain about and turned out to be wrong.
A stand for measuring the truth
The stand is a set of Docker hosts that run MeshHold and the usual suspects — rsync, scp, Samba, Syncthing, croc, rclone — moving the same data over the same paths, including the relayed path most real deployments use rather than the ideal direct one. The hard part of a benchmark isn't running it, it's making it honest: giving every contender the same durability guarantees, stopping each one from quietly phoning home mid-run, and shouting when a container is somehow running yesterday's binary. Half the value this week was catching my own stand lying to me before I trusted a single number.
One delivery layer
The measuring pointed straight at the architecture. MeshHold had grown three overlapping ways to move a block, and the seams between them cost real time. So they collapsed into one: a single delivery layer with a connection pool underneath, blocks riding QUIC streams, and one admission budget measured in bytes instead of three different limits arguing with each other. The best result of that cleanup — a download's speed stopped being a function of round-trip time, which is exactly the property you want on a relayed, high-latency path.
Along the way I seriously considered forking QUIC itself; the loss numbers looked damning. A couple of days of measurement later, the "losses" turned out to be our own block-sizing bug, so we stayed on upstream. A fork you decide not to take is still a decision — I'd just rather make it with data.
The bugs a plot finds that an average hides
This was the fun part, and the humbling one: a wall of small, specific bottlenecks that only showed up once I plotted a run instead of averaging it.
- The decide-what-to-fetch phase was 83% a single blocking network call.
- A catalog page sized for the fattest row was being used for the thinnest, so a catching-up node re-read far more than it needed.
- The receiver hashed every block twice, and a file-mode vault re-derived each block's nonce on every serve.
- An announcement of new data woke nothing, so the receiver waited out a whole cycle it didn't need to.
And one that wasn't about speed at all: a peer's newer version of a file could get republished as a local edit, forever — a genuine correctness bug the benchmark flushed out by accident. Every one of these got measured, fixed, and measured again.
Two more that mattered
Not everything was transfer. A nasty one got fixed: reboot a node while none of its peers are online and the full-tunnel VPN could capture the very routes it needed to reach those peers — eating its own carrier and going dark. Now capture waits for a live exit, and the UI says out loud when routing is paused instead of silently dropping everything. And durability is a choice now: a per-vault fsync_writes setting, so you decide whether a vault trades a little speed for the guarantee that a write survived the power going out, instead of me deciding for you.
Where it landed
Honestly: faster, and far better understood. The obvious gaps are closed and I can now point at where every millisecond of a transfer goes. There's one case left — an incremental sync of a folder that barely changed — where I'm slower than I'd like and can't fully explain why yet. That's the next thread to pull. Still pointed at 0.8.
Since last week
- A transfer benchmark stand: MeshHold raced against rsync, scp, Samba, Syncthing, croc and rclone over identical paths (including the relayed one), with the durability, phone-home and stale-binary traps closed so the numbers mean something
- One delivery layer: three overlapping transfer paths collapsed into a single link layer with a connection pool; blocks over QUIC streams; one byte-based admission budget; a download's speed no longer scales with round-trip time
- Stayed on upstream QUIC: investigated and measured a quic-go fork, found the apparent packet loss was our own block-sizing bug, and declined the fork
- Bottlenecks found by measuring: a decide phase that was 83% one blocking call; a mis-sized catalog page; double-hashing on the receiver and per-serve nonce re-derivation in file-mode vaults; an announcement that woke no cycle; and announcement rates the mesh can actually absorb
- A correctness bug: a peer's newer file version could be republished as a local edit indefinitely — found and fixed
- VPN self-capture fix: full-tunnel capture waits for a live exit, so a node rebooting with no peers online can't eat its own carrier; paused routing is shown, not silent
- Durability is a choice: a per-vault
fsync_writessetting; the scanner flushes once per pass, not once per file - Legal: third-party attributions shipped, including the
wintun.dlland vendored-fork notices their licenses require