Day 02: BitTorrent & Hypercore

Two ways to move data with no server: a static swarm and a signed, append-only log.

⏱ ~90 min · theory 50 / lab 40

Yesterday's DHT tells peers how to find each other. Today is about what they do once connected:
move data efficiently and verifiably. BitTorrent solved this for static files in 2001 with a beautifully
pragmatic incentive design. Hypercore — the core of the Holepunch stack — solved a harder version: a single-writer,
append-only log that any peer can replicate and cryptographically verify block-by-block. Understanding the contrast
between them is the fastest way to sound fluent about P2P data transfer.

2.1 BitTorrent: the swarm model

A torrent splits a file into fixed-size pieces (typically 256 KB–1 MB), and the
.torrent metadata (or magnet link) carries a SHA-1/SHA-256 hash of every piece. That hash list is the
trust anchor: a peer can verify any piece it receives against the expected hash and reject corrupt or malicious data
without trusting the sender. Peers swap pieces until everyone has the whole file.

Piece selection: rarest-first

If every peer downloaded pieces in order, the first pieces would be everywhere and the last pieces would be
scarce — and if the only seed left before sharing a rare piece, the file would be unrecoverable. BitTorrent instead
prefers the rarest piece first: each peer tracks how many of its neighbors have each piece and
fetches the least-replicated ones. This spreads scarcity risk and keeps the swarm healthy as seeds come and go. A
small exception ("random first piece") bootstraps a brand-new peer quickly so it has something to trade.

Choking: tit-for-tat incentives

BitTorrent's real innovation is social, not technical. Each peer chokes (refuses to upload to)
most of its neighbors and unchokes only a few — preferentially those who upload to it the
fastest. This tit-for-tat reciprocation rewards contributors and starves freeloaders. Periodically a peer does an
optimistic unchoke of a random neighbor, to discover better partners and to give newcomers (who have
nothing to offer yet) a foot in the door.

The bitfield message — sent on connection — is how a peer advertises which pieces it already has,
as a compact bit array. have messages then incrementally announce new pieces. The BitTorrent
DHT (a Kademlia network keyed by infohash) and PEX (peer exchange) let swarms operate with
no tracker at all — this is exactly the DHT material from Day 1 applied in production.

2.2 Hypercore: the append-only log

BitTorrent distributes a fixed, known-in-advance file. But many applications need a growing
data feed — a chat history, a database changelog, a file that gets revised. Hypercore is the answer: a
secure, append-only log owned by a single keypair. Only the holder of the secret key can append;
anyone with the public key (which is the feed's address) can read and verify.

The Merkle tree over blocks

Each appended block is hashed (BLAKE2b), and those hashes form a Merkle tree. The signed root
commits to the entire log. The payoff: a peer can request block i alone and receive it plus a small set
of sibling hashes (a Merkle proof) that verify it against the signed root — *without downloading the rest of the
log*. This is what makes random access and partial replication trustworthy. It is the same idea as BitTorrent's
piece hashes, generalized into a tree so proofs stay O(log n) and the structure can grow.

Why a tree, not a flat list
A flat hash list (BitTorrent) is fine when the piece count is fixed and known. A Merkle tree lets the dataset
grow without bound while keeping verification proofs logarithmic and letting the writer publish a single
signed root that commits to all of history.

Single-writer, many-reader

The keypair is the heart of the model. The public key is the discovery key / address; the secret
key authorizes appends. Replication is a duplex stream: two peers exchange what blocks they have and want, request
ranges, and verify every block against the Merkle root as it arrives. Higher layers in the Holepunch stack
(hyperbee for key–value/B-tree, hyperdrive for filesystems, autobase for
multi-writer) are all built on this one primitive.

2.3 Lab — append-only log with Merkle verification

Goal

Build a tiny append-only log that produces a Merkle root and lets a "reader" verify a single block with a proof —
the conceptual core of Hypercore — using only Node's built-in crypto. Then run real Hypercore replication
between two instances.

Part A — Merkle log from scratch (no dependencies)

// merkle-log.js
const crypto = require('crypto');
const H = (buf) => crypto.createHash('sha256').update(buf).digest();

class MerkleLog {
  constructor() { this.blocks = []; this.leaves = []; }

  append(data) {
    const buf = Buffer.isBuffer(data) ? data : Buffer.from(data);
    this.blocks.push(buf);
    this.leaves.push(H(buf));
    return this.blocks.length - 1; // index
  }

  // build the tree bottom-up; return the layered hash arrays
  tree() {
    let layer = this.leaves.slice();
    const layers = [layer];
    while (layer.length > 1) {
      const next = [];
      for (let i = 0; i < layer.length; i += 2) {
        const left = layer[i];
        const right = layer[i + 1] || left; // duplicate last if odd
        next.push(H(Buffer.concat([left, right])));
      }
      layers.push(next);
      layer = next;
    }
    return layers;
  }

  root() { const l = this.tree(); return l[l.length - 1][0]; }

  // proof = sibling hash at each level needed to recompute the root
  proof(index) {
    const layers = this.tree();
    const path = [];
    let idx = index;
    for (let lvl = 0; lvl < layers.length - 1; lvl++) {
      const isRight = idx % 2 === 1;
      const sibling = isRight ? idx - 1 : idx + 1;
      const layer = layers[lvl];
      path.push({ hash: (layer[sibling] || layer[idx]), isRight });
      idx = idx >> 1;
    }
    return path;
  }
}

// a reader who has ONLY the trusted root verifies one block
function verify(block, index, proof, trustedRoot) {
  let h = H(block);
  for (const step of proof) {
    h = step.isRight
      ? H(Buffer.concat([step.hash, h]))   // our node was on the right
      : H(Buffer.concat([h, step.hash]));
  }
  return h.equals(trustedRoot);
}

// --- demo ---
const log = new MerkleLog();
['genesis', 'block-1', 'block-2', 'block-3', 'block-4'].forEach(d => log.append(d));
const root = log.root();

const i = 3;
const ok = verify(log.blocks[i], i, log.proof(i), root);
console.log('root      :', root.toString('hex').slice(0, 16), '…');
console.log('verify #3 :', ok);

// tamper test: flip the block, proof must fail
const bad = Buffer.from('block-3-EVIL');
console.log('tampered  :', verify(bad, i, log.proof(i), root));

Expected output

node merkle-log.js
// root      : 2f6a... …
// verify #3 : true
// tampered  : false   <- the Merkle proof rejects altered data

Part B — the real thing: Hypercore replication

Now see it in production form. Modern Hypercore (v11+) is managed through a Corestore backed by a
storage path; we replicate a log from a writer store to a key-only reader store over piped streams (the same API
works over a real socket).

npm init -y
npm install hypercore corestore

// hc.js
const Corestore = require('corestore');
const os = require('os'), path = require('path'), fs = require('fs');

async function main() {
  // two independent stores on disk (Hypercore 11+ requires a storage dir)
  const dirA = fs.mkdtempSync(path.join(os.tmpdir(), 'hc-a-'));
  const dirB = fs.mkdtempSync(path.join(os.tmpdir(), 'hc-b-'));

  // writer core
  const store = new Corestore(dirA);
  const writer = store.get({ name: 'log' });
  await writer.ready();
  await writer.append([Buffer.from('hello'), Buffer.from('p2p'), Buffer.from('world')]);
  console.log('writer length:', writer.length,
              'key:', writer.key.toString('hex').slice(0, 12));

  // reader store: opens the core by PUBLIC KEY only — it has no data yet
  const store2 = new Corestore(dirB);
  const reader = store2.get({ key: writer.key });
  await reader.ready();

  // pipe the two replication streams together (stand-in for a socket)
  const s1 = store.replicate(true);
  const s2 = store2.replicate(false);
  s1.pipe(s2).pipe(s1);

  // reader fetches block 2 — verified against the signed root automatically
  const block = await reader.get(2);
  console.log('reader got block 2:', block.toString()); // "world"
  process.exit(0);
}
main();

Experiments

• In Part A, append 1000 blocks and confirm proof length grows like log₂(n), not n. That logarithmic proof size is the whole point.
• In Part B, call reader.get(0) before the writer appends (start the reader first), then append — observe that get resolves once the block arrives. Hypercore get means "wait until available."
• Inspect reader.length vs reader.contiguousLength to see the difference between "known to exist" and "actually downloaded."

2.4 Self-check & interview drill

• I can explain rarest-first and tit-for-tat choking and what each prevents.
• I can describe how a Merkle proof verifies one block against a signed root.
• My from-scratch Merkle log verifies good blocks and rejects tampered ones.
• I replicated a real Hypercore between a writer and a key-only reader.