131: Node Debugging, Profiling, Memory Leaks, `--inspect`, `perf_hooks`, and APM

Learning objectives

You will learn to:

use Node's inspector;
debug with breakpoints;
understand heap versus RSS at a practical level;
identify common memory leaks;
take heap snapshots carefully;
use CPU profiles;
measure event-loop delay/utilization;
use perf_hooks;
understand diagnostic reports;
understand logging, metrics, traces, and APM;
debug slow Node services systematically.

Debugging starts with a hypothesis

Avoid random changes.

Ask:

text
Is it CPU?
event-loop blocking?
memory?
database?
upstream network?
too many requests?
connection pool?
GC?
stream buffering?
worker queue?

Collect evidence.

`--inspect`

Run:

bash
node --inspect src/server.js

Then connect Chrome DevTools/IDE.

Break on first line:

bash
node --inspect-brk src/server.js

Do not expose inspector port publicly.

Inspector can execute code/control process.

Bind securely in production diagnostics, ideally not enabled publicly.

Breakpoints

Use source breakpoints.

Inspect:

local variables;
closure;
call stack;
async stack;
request data.

Do not leave debugger statements:

js
debugger;

in production hot paths accidentally.

Stack trace

Understand:

text
sync call stack
async boundaries
source maps
cause chain

Enable/source-map support appropriately for transpiled code.

Process memory

js
console.log(process.memoryUsage());

Common:

`heapUsed`

V8-managed JS heap currently used.

`heapTotal`

Allocated V8 heap.

`rss`

Resident process memory including:

V8;
native code;
stacks;
Buffers;
mapped memory.

`external`

Memory associated with C++/external objects, including Buffers in many cases.

A growing RSS with stable heapUsed may point to native/Buffer behavior rather than ordinary JS object retention.

Garbage collection

V8 automatically collects unreachable objects.

Memory leak means objects/resources remain reachable or native resources remain allocated longer than intended.

Examples:

global Map never deletes;
EventEmitter listener leak;
unbounded cache;
timers;
request closures retained;
buffers in queue;
sockets not closed;
worker/job references;
database cursor/session not ended.

Classic leak

js
const cache = new Map();

app.get('/user/:id', async (req, res) => {
  const user = await loadUser(req.params.id);

  cache.set(req.params.id, user);

  res.json(user);
});

If IDs unbounded, cache grows forever.

Fix with:

no cache;
bounded LRU;
TTL;
external cache;
correct invalidation.

A cache is controlled memory retention by design. It still needs bounds.

Listener leak

js
app.get('/events', (req, res) => {
  bus.on('update', handler);
});

Never removed after client disconnect.

Add cleanup:

js
req.on('close', () => {
  bus.off('update', handler);
});

or use abortable listener APIs.

Heap snapshots

Take snapshots through inspector/diagnostic tooling.

Snapshot can:

pause process;
use significant memory;
contain sensitive strings/tokens/user data.

Do not casually capture production heaps.

Use replica/staging or controlled procedures.

Compare snapshots:

text
before load
after repeated load
after GC/idle

Inspect retaining paths.

CPU profiling

If CPU high:

sample profile;
inspect hot functions;
measure algorithm;
identify JSON/regex/compression/crypto;
compare native vs JS.

Do not optimize functions absent from profile.

Event-loop delay

js
import {
  monitorEventLoopDelay,
} from 'node:perf_hooks';

const histogram = monitorEventLoopDelay({
  resolution: 20,
});

histogram.enable();

setInterval(() => {
  console.log({
    meanMs: histogram.mean / 1e6,
    maxMs: histogram.max / 1e6,
  });

  histogram.reset();
}, 10_000);

High delay means event loop cannot service callbacks promptly.

Causes include:

CPU blocking;
synchronous I/O;
huge GC pauses;
overload.

Event-loop utilization

performance.eventLoopUtilization() can help estimate how busy event loop is.

Use trends, not one sample.

Interpret with CPU and latency.

Performance marks/measures

js
import { performance } from 'node:perf_hooks';

performance.mark('start');

await doWork();

performance.mark('end');

performance.measure('work', 'start', 'end');

console.log(
  performance.getEntriesByName('work'),
);

Useful for local application stages.

For distributed systems, tracing is stronger.

Histograms

Node perf APIs can build histograms for durations.

Production metrics systems can record:

text
request duration
DB duration
upstream duration
queue wait
worker duration

Use percentiles:

text
p50
p95
p99

Average alone hides tail latency.

Diagnostic reports

Node can produce diagnostic reports containing runtime/process information useful for crashes/performance incidents.

Reports may include sensitive environment/process details.

Protect access.

Use current Node docs for flags/signals/API.

Core dumps

Advanced crash analysis can use core dumps/native tooling.

This is specialized operational debugging.

Do not enable without security/storage planning.

Structured logs

Recommended fields:

json
{
  "level": "error",
  "time": "...",
  "requestId": "...",
  "route": "PATCH /tasks/:id",
  "durationMs": 123,
  "errorCode": "DB_TIMEOUT"
}

Do not log whole request body by default.

Metrics

Examples:

text
request rate
error rate
latency
event-loop delay
CPU
RSS/heap
GC
DB pool
upstream failures
worker queue depth
active connections

Use labels carefully.

Do not use high-cardinality labels such as raw userId/requestId in metrics systems.

Tracing

Distributed trace:

text
browser/API request
→ Node route
→ service
→ Mongo
→ external API

Spans capture timing/context.

OpenTelemetry is a common standard.

Do not manually invent incompatible trace IDs if ecosystem tooling can propagate standard context.

AsyncLocalStorage and traces

Tracing/logging systems often use async context to propagate current span/request ID.

Avoid mutating shared global “current request” variable.

Async execution would mix users.

APM

Application Performance Monitoring tools can provide:

automatic HTTP traces;
database timing;
error aggregation;
service maps;
continuous profiling.

Understand overhead, privacy, and sampling.

Do not depend on APM without understanding basic runtime metrics.

Slow-request investigation

Suppose p95 jumps 100ms → 3s.

Check:

request rate;
CPU;
event-loop delay;
Mongo/upstream spans;
pool saturation;
worker queue;
GC/memory;
deployment/version change.

If event-loop delay is low but DB span is 2.8s, use Mongo query diagnostics—not use worker_threads.

Load testing

Use a controlled load generator.

Measure:

throughput;
latency percentiles;
errors;
saturation.

Do not load-test production irresponsibly.

Warm-up, realistic payloads, keep-alive, auth, and think time affect results.

Memory load test

Run sustained workload, not 30 seconds only.

Leaks may need hours.

Track heap after GC/steady periods.

A sawtooth heap is normal GC behavior if baseline remains bounded.

Logging performance

Synchronous logging or huge JSON serialization can block.

Use appropriate structured logger and output strategy.

Container stdout can still become bottleneck if logging enormous volumes.

Source maps and error monitoring

Deploy maps to monitoring service/private storage.

Do not necessarily serve .map publicly.

Ensure release version identifies correct map.

Failure clinic

inspector open to internet;
average latency only;
heap snapshot taken on tiny-memory production pod;
cache without limit;
high-cardinality metrics;
logging secrets;
CPU optimization when DB is bottleneck;
profiling dev mode only;
no baseline before change.

Exercises

Debug a route with --inspect.
Create an intentional Map leak and inspect memory trend.
Create EventEmitter listener leak and fix.
Measure event-loop delay during CPU loop.
Add performance marks around service and repository.
Build request-duration histogram.
Add structured request ID logging.
Sketch OpenTelemetry spans.
Run load test and identify first saturation point.
Write incident checklist for high p99 latency.

Mastery checklist

Explain:

inspector;
heap/rss/external;
GC/leaks;
heap snapshot;
CPU profile;
event-loop delay/utilization;
perf hooks;
diagnostic reports;
logs/metrics/traces;
APM;
evidence-driven debugging.