Day 04: Binary Protocols & Serialization

Why every serious P2P system hand-rolls its bytes — framing, varints, and zero-copy formats.

⏱ ~90 min · theory 45 / lab 45

JSON is wonderful for humans and terrible for hot networking paths. It's bulky, slow to parse, ambiguous about
number types, and forces you to allocate strings for everything. P2P protocols move millions of small messages, so
they encode data as tight binary. Today you'll learn how messages are framed on a stream,
how integers are packed with varints, and how the major serialization formats trade off — then
hand-write an encoder/decoder, which is exactly the kind of thing you may be asked to live-code.

4.1 Why binary beats JSON on the wire

The self-describing nature of JSON is its biggest cost at scale: every message re-sends the field names. Binary
protocols agree on a schema once, then send only values in a known order.

4.2 Framing: where does a message end?

TCP and UDX give you a stream of bytes, not messages. If you write two messages, the reader may receive
them glued together, or split across two reads. You must impose framing. Two common approaches:

• Delimiter-based: end each message with a sentinel byte (e.g. newline). Simple, but you must escape the delimiter if it can appear in the payload — awkward for binary data.
• Length-prefix: write the message length first, then the bytes. The reader reads the length, then knows exactly how many bytes to consume. This is the standard for binary protocols and what you'll build.

The framing bug everyone hits once
Forgetting that a single 'data' event can contain part of a message, multiple
messages, or a message plus part of the next. A correct length-prefix reader buffers incoming bytes and
emits messages only when a full frame has arrived. Get this wrong and your protocol works in tests and corrupts
under load.

4.3 Varints: integers that earn their bytes

Most integers in real protocols are small (lengths, counts, indices). Spending 4 or 8 fixed bytes on a value of
5 is wasteful. A varint (LEB128, as used by Protobuf, WebAssembly, and Holepunch's
compact-encoding) encodes an integer in a variable number of bytes: 7 bits of value per byte, with the
high bit as a "more bytes follow" continuation flag.

• Values 0–127 → 1 byte
• 128–16383 → 2 bytes
• …7 more bits per additional byte

So small numbers cost one byte, and you only pay for big numbers when you actually use them. This is the single
most common trick in compact wire formats.

4.4 The serialization landscape

The key conceptual split: parse-based formats (Protobuf, MessagePack) decode bytes into new
in-memory objects, while zero-copy formats (Cap'n Proto, FlatBuffers) let you read fields directly
out of the received buffer with no allocation or decode pass. Zero-copy wins when messages are large or you only
read a few fields; parse-based is simpler and fine for small messages. compact-encoding sits in the
parse-based camp but is deliberately minimal and gives you total control — which is why it suits a P2P stack where
every byte and allocation counts.

4.5 Lab — hand-write a length-prefixed varint protocol

Goal

Implement varint encode/decode, then a length-prefixed framer that correctly reassembles messages from an
arbitrarily-chunked byte stream. Finish by encoding a structured message by hand. Then compare against real
compact-encoding.

Step 1 — varints

// varint.js
function encodeVarint(value) {
  const bytes = [];
  while (value > 0x7f) {            // while more than 7 bits remain
    bytes.push((value & 0x7f) | 0x80); // low 7 bits + continuation flag
    value = Math.floor(value / 128);  // >>> 7 but safe past 32 bits
  }
  bytes.push(value & 0x7f);            // final byte, no continuation
  return Buffer.from(bytes);
}

function decodeVarint(buf, offset = 0) {
  let result = 0, shift = 0, pos = offset;
  while (true) {
    const byte = buf[pos++];
    result += (byte & 0x7f) * Math.pow(2, shift);
    if ((byte & 0x80) === 0) break;  // continuation bit clear -> done
    shift += 7;
  }
  return { value: result, bytesRead: pos - offset };
}

// sanity
for (const v of [0, 1, 127, 128, 300, 1000000]) {
  const enc = encodeVarint(v);
  const dec = decodeVarint(enc).value;
  console.log(v, '->', [...enc], `(${enc.length} bytes) ->`, dec);
}
module.exports = { encodeVarint, decodeVarint };

node varint.js
// 0 -> [0] (1 bytes) -> 0
// 127 -> [127] (1 bytes) -> 127
// 128 -> [128,1] (2 bytes) -> 128
// 300 -> [172,2] (2 bytes) -> 300
// 1000000 -> [192,132,61] (3 bytes) -> 1000000   <- vs 15 bytes as JSON

Step 2 — a correct length-prefixed framer

This is the part interviewers love, because the naive version is subtly broken. The framer buffers raw bytes and
only emits a message when a complete frame is present — handling split and coalesced reads.

// framer.js
const { encodeVarint, decodeVarint } = require('./varint');
const EventEmitter = require('events');

function frame(payload) {
  const len = encodeVarint(payload.length);
  return Buffer.concat([len, payload]); // [varint length][payload bytes]
}

class Decoder extends EventEmitter {
  constructor() { super(); this.buf = Buffer.alloc(0); }
  push(chunk) {
    this.buf = Buffer.concat([this.buf, chunk]); // accumulate
    while (this.buf.length > 0) {
      let header;
      try { header = decodeVarint(this.buf, 0); }
      catch { return; }                       // length not fully arrived
      const total = header.bytesRead + header.value;
      if (this.buf.length < total) return;     // full frame not here yet — wait
      const payload = this.buf.subarray(header.bytesRead, total);
      this.emit('message', Buffer.from(payload));
      this.buf = this.buf.subarray(total);   // keep the remainder
    }
  }
}
module.exports = { frame, Decoder };

Step 3 — prove it survives evil chunking

// test-framer.js
const { frame, Decoder } = require('./framer');

const messages = ['hello', 'a', 'a much longer message here', 'last']
  .map(s => Buffer.from(s));
const stream = Buffer.concat(messages.map(frame)); // all frames glued together

const dec = new Decoder();
const out = [];
dec.on('message', m => out.push(m.toString()));

// feed the stream in deliberately awful 3-byte chunks
for (let i = 0; i < stream.length; i += 3)
  dec.push(stream.subarray(i, i + 3));

console.log(out);
// [ 'hello', 'a', 'a much longer message here', 'last' ]  ✅ reassembled perfectly

Step 4 — encode a structured message by hand

A "peer announce" message: a 1-byte type tag, a varint topic length + topic bytes, and a fixed 6-byte address
(4-byte IPv4 + 2-byte port).

// announce.js
const { encodeVarint, decodeVarint } = require('./varint');
const TYPE_ANNOUNCE = 0x01;

function encodeAnnounce({ topic, ip, port }) {
  const topicBuf = Buffer.from(topic);
  const addr = Buffer.alloc(6);
  ip.split('.').forEach((o, i) => addr[i] = Number(o));
  addr.writeUInt16BE(port, 4); // big-endian = network byte order
  return Buffer.concat([
    Buffer.from([TYPE_ANNOUNCE]),
    encodeVarint(topicBuf.length), topicBuf,
    addr
  ]);
}

function decodeAnnounce(buf) {
  let off = 0;
  const type = buf[off++];
  const { value: tlen, bytesRead } = decodeVarint(buf, off); off += bytesRead;
  const topic = buf.subarray(off, off + tlen).toString(); off += tlen;
  const ip = [...buf.subarray(off, off + 4)].join('.'); off += 4;
  const port = buf.readUInt16BE(off);
  return { type, topic, ip, port };
}

const wire = encodeAnnounce({ topic: 'chat-room-42', ip: '192.168.1.50', port: 9000 });
console.log('bytes :', wire.length, [...wire]);
console.log('decoded:', decodeAnnounce(wire));

Step 5 — the real thing: compact-encoding

npm install compact-encoding

// ce.js
const c = require('compact-encoding');

// compose a struct codec from primitives
const announce = {
  preencode(state, m) {     // pass 1: measure
    c.string.preencode(state, m.topic);
    c.uint16.preencode(state, m.port);
  },
  encode(state, m) {        // pass 2: write
    c.string.encode(state, m.topic);
    c.uint16.encode(state, m.port);
  },
  decode(state) {
    return { topic: c.string.decode(state), port: c.uint16.decode(state) };
  }
};

const buf = c.encode(announce, { topic: 'chat-room-42', port: 9000 });
console.log('compact bytes:', buf.length, [...buf]);
console.log('decoded      :', c.decode(announce, buf));

Notice the two-pass design: preencode computes the exact size so encode
can write into a single pre-allocated buffer with no resizing — an allocation optimization you'll revisit on Day 6.

Experiments

• Change the test framer chunk size to 1 byte, then to 1000. Both must produce identical output — that's the definition of correct framing.
• Compare your hand-rolled announce size against the equivalent JSON string. Measure the ratio.
• Add a malformed frame (a length prefix larger than the data that ever arrives) and confirm the decoder simply waits rather than crashing or emitting garbage.

4.6 Self-check & interview drill

• I can encode and decode a varint and explain its byte-size curve.
• I can implement a length-prefix framer that survives split and coalesced reads.
• I can hand-encode a structured message with mixed fixed and variable fields.
• I understand compact-encoding's two-pass preencode/encode design.