122: Buffers, Binary Data, Encodings, Streams, Pipelines, and Backpressure

Learning objectives

You will learn to:

explain what a byte is and why Node needs Buffer;
distinguish text encoding from binary data;
create and inspect Buffers safely;
understand UTF-8 boundaries and byte length;
understand readable, writable, duplex, and transform streams;
consume streams using events, async iteration, and pipeline;
understand backpressure;
build transform pipelines;
avoid buffering large payloads unnecessarily;
handle stream errors and cancellation;
know when Web Streams and Node streams interact.

Why binary data matters

JavaScript strings are text abstractions.

Networks, files, compressed data, images, and cryptographic material are ultimately bytes.

Node's Buffer is a byte-oriented data type built on typed-array concepts.

js
const buffer = Buffer.from('hello', 'utf8');

console.log(buffer);
console.log(buffer.length);

buffer.length is bytes, not JavaScript string characters.

UTF-8 byte length

js
console.log('A'.length); // 1
console.log(Buffer.byteLength('A', 'utf8')); // 1

console.log('₹'.length);
console.log(Buffer.byteLength('₹', 'utf8'));

JavaScript string .length counts UTF-16 code units.

Network/body limits usually care about bytes.

Do not enforce upload/request limits using only string character counts.

Create Buffers

From text:

js
const a = Buffer.from('hello', 'utf8');

Allocate initialized memory:

js
const b = Buffer.alloc(1024);

Allocate uninitialized memory:

js
const c = Buffer.allocUnsafe(1024);

allocUnsafe can be faster but memory must be overwritten before reading/exposing it.

Do not send uninitialized Buffer contents to users.

Encoding and decoding

js
const bytes = Buffer.from('hello', 'utf8');

console.log(bytes.toString('hex'));
console.log(bytes.toString('base64'));
console.log(bytes.toString('utf8'));

Base64 is an encoding, not encryption.

Do not treat:

text
base64(secret)

as secure.

Hex and binary identifiers

Cryptographic hashes commonly display as hex:

js
const digest = Buffer.from([0xde, 0xad, 0xbe, 0xef]);

console.log(digest.toString('hex'));
// deadbeef

For random tokens, use cryptographically secure random APIs rather than Math.random.

Buffer slicing/views

Buffers inherit typed-array behaviors.

Be careful: some slice/subarray operations can share underlying memory.

js
const original = Buffer.from([1, 2, 3, 4]);

const view = original.subarray(1, 3);

view[0] = 99;

console.log(original);

If you require an independent copy:

js
const copy = Buffer.from(view);

Know whether your code is passing a view or an owned copy when handling mutable binary data.

Binary parsing

Never read fields beyond bounds.

js
function readUInt32(buffer, offset) {
  if (offset + 4 > buffer.length) {
    throw new RangeError('Not enough bytes');
  }

  return buffer.readUInt32BE(offset);
}

Untrusted binary parsers need strict length validation.

Streams

A stream processes data over time instead of requiring the complete value in memory.

Core categories:

Readable

Source of data:

text
file read
HTTP request body
process.stdin

Writable

Destination:

text
file write
HTTP response
process.stdout

Duplex

Readable and writable:

text
TCP socket

Transform

Duplex stream where output is transformed from input:

text
gzip
cipher
line parser

Why streams matter

Bad for huge file:

js
const data = await readFile('./huge.log');

await upload(data);

The entire file exists in memory.

Streaming:

text
disk → chunks → network

keeps memory bounded.

Readable stream events

js
import { createReadStream } from 'node:fs';

const stream = createReadStream('./notes.txt', {
  encoding: 'utf8',
});

stream.on('data', (chunk) => {
  console.log(chunk);
});

stream.on('end', () => {
  console.log('done');
});

stream.on('error', (error) => {
  console.error(error);
});

Event mode is useful to understand, but modern async iteration can be clearer.

Async iteration

js
const stream = createReadStream('./notes.txt', {
  encoding: 'utf8',
});

for await (const chunk of stream) {
  console.log(chunk);
}

This works naturally with async functions and helps express consumption flow.

Writable streams

js
process.stdout.write('hello\n');

For a writable:

js
const canContinue = writable.write(chunk);

If it returns false, the internal buffer reached its high-water threshold.

Wait for drain before producing more.

This is backpressure.

Backpressure

Mental model:

text
producer faster than consumer
→ buffer grows
→ memory grows
→ latency grows
→ process can fail

Backpressure lets consumer signal:

text
slow down

Streams build this into their flow-control model.

Manual backpressure example

js
import { once } from 'node:events';

async function writeAll(writable, chunks) {
  for (const chunk of chunks) {
    if (!writable.write(chunk)) {
      await once(writable, 'drain');
    }
  }

  writable.end();
}

Most application code should use pipeline instead of manually wiring everything.

`pipeline`

js
import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

await pipeline(
  createReadStream('./access.log'),
  createGzip(),
  createWriteStream('./access.log.gz'),
);

pipeline:

connects streams;
propagates errors;
coordinates closure;
respects backpressure.

This is the preferred high-level composition primitive for many stream pipelines.

Transform stream

js
import { Transform } from 'node:stream';

const upper = new Transform({
  decodeStrings: false,

  transform(chunk, encoding, callback) {
    callback(null, chunk.toUpperCase());
  },
});

Then:

js
await pipeline(
  process.stdin,
  upper,
  process.stdout,
);

For line-oriented text, remember chunks are arbitrary boundaries. One chunk is not guaranteed to equal one line.

Chunk boundaries

Network/file streams can split data anywhere.

Input:

text
hello\nworld\n

may arrive as:

text
"hel"
"lo\nwo"
"rld\n"

Do not parse line/protocol messages assuming each data chunk is one message.

Maintain framing buffer or use a parser.

StringDecoder

When decoding streaming multi-byte text manually, a UTF-8 character can be split across chunks.

Node's string-decoder utilities/stream encoding support can preserve incomplete character bytes correctly.

Prefer setting stream encoding or using proper text-decoding abstractions rather than chunk.toString() independently on arbitrary split bytes when correctness matters.

Object mode

Streams can process JavaScript objects:

js
new Transform({
  objectMode: true,

  transform(task, encoding, callback) {
    callback(null, {
      ...task,
      title: task.title.trim(),
    });
  },
});

Object-mode highWaterMark represents object counts rather than byte sizes.

Do not use streams merely because you have an array of 20 objects. Use where incremental flow matters.

Stream errors

Every stream pipeline can fail:

text
source read error
transform parse error
destination closed
client disconnect
disk full

pipeline() rejects.

js
try {
  await pipeline(...);
} catch (error) {
  logger.error({ error }, 'pipeline failed');
}

Do not continue sending HTTP success after the pipeline failed.

Cancellation

Modern stream/pipeline APIs can work with AbortSignal in supported forms.

Example concept:

js
const controller = new AbortController();

await pipeline(source, transform, destination, {
  signal: controller.signal,
});

Verify exact signatures for your supported Node version.

Cancellation should tear down all involved resources.

HTTP request bodies are streams

Native Node:

js
for await (const chunk of request) {
  ...
}

Never accumulate unbounded request bodies:

js
let body = '';

for await (const chunk of req) {
  body += chunk;
}

without a maximum byte limit.

Attackers can exhaust memory.

Framework JSON parsers should also have explicit body limits.

Streaming HTTP responses

Large export:

js
await pipeline(
  createReportStream(),
  response,
);

This can begin response earlier and avoid buffering full report.

But after headers/body start, error mapping becomes harder because HTTP status may already be committed.

Design stream failure behavior.

File upload

A multipart parser can stream:

text
HTTP body
→ multipart parser
→ size/type policy
→ virus scanner / object storage

Avoid:

text
HTTP body
→ entire Buffer in RAM
→ then upload

for large inputs.

Web Streams

Modern Node includes web-compatible:

text
ReadableStream
WritableStream
TransformStream

and interoperability utilities with Node streams.

fetch() response bodies are Web Streams.

Example:

js
const response = await fetch(url);

for await (const chunk of response.body) {
  ...
}

Node provides bridging APIs where you need to connect Web and Node stream ecosystems.

Know which stream type a library expects.

High water mark

Streams buffer up to configured thresholds.

Changing highWaterMark affects:

memory;
throughput;
syscall frequency;
latency.

Do not tune it without measurement.

Defaults are usually a good starting point.

`stream.finished`

Use to await stream completion when not using full pipeline.

Prefer promises helper where appropriate.

Compression

js
import { createGzip } from 'node:zlib';

Compression is CPU work and some zlib operations use the libuv thread pool.

At high throughput, compression can become CPU/pool pressure.

Reverse proxy/CDN may be a better compression owner.

Security

Compression bombs

A tiny compressed file can expand massively.

Set limits on decompressed size where processing untrusted archives/content.

Zip Slip

Archive extraction paths can contain traversal entries. Use secure extraction libraries/policies.

Binary parsers

Validate lengths and formats.

Secret buffers

Sensitive Buffer contents can remain in process memory. Avoid unnecessary copies/logging.

Failure clinic

`Buffer.allocUnsafe` exposed

Data leak.

Treating chunk as message

Protocol corruption.

Ignoring writable `false`

Memory growth.

`readFile` on huge upload

OOM.

No body size limit

DoS risk.

Pipe without error strategy

Partially written output and uncaught failures.

Worked project: streaming log compressor

js
import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

async function compress(input, output) {
  await pipeline(
    createReadStream(input),
    createGzip(),
    createWriteStream(output),
  );
}

try {
  await compress(
    process.argv[2],
    process.argv[3],
  );
} catch (error) {
  console.error(error.message);
  process.exitCode = 1;
}

Extend it with:

validation;
progress;
cancellation;
temp-file + rename;
tests.

Exercises

Compare string length and UTF-8 byte length.
Encode/decode hex and base64.
Demonstrate a shared Buffer view.
Read a 1 GB file with stream and measure memory.
Build stdin uppercase Transform.
Write manual backpressure handling.
Replace it with pipeline.
Parse newline-delimited JSON correctly across chunk boundaries.
Add AbortSignal cancellation.
Design safe streaming upload limits.

Mastery checklist

Explain:

Buffer;
encoding;
byte length;
stream categories;
chunks;
backpressure;
highWaterMark;
pipeline;
object mode;
cancellation;
Node versus Web Streams;
streaming security.