132: Production Node Operations — Configuration, Graceful Shutdown, Health, Logging, Process Managers, Containers, and Reliability

Learning objectives

You will learn to:

design production startup/shutdown lifecycle;
validate config before listening;
order dependencies during startup;
handle SIGTERM/SIGINT;
stop accepting new traffic;
close HTTP/database/worker resources;
use readiness and liveness health checks;
understand process supervisors such as systemd/PM2/container orchestrators;
understand stateless horizontal scaling;
design timeouts/retries/circuit-breaker boundaries;
understand logging and deployment releases;
avoid graceful-shutdown and retry-storm mistakes.

Production lifecycle

A service has phases:

text
boot
↓
validate config
↓
connect dependencies
↓
listen
↓
serve
↓
draining
↓
close dependencies
↓
exit

Make phases explicit.

Startup

Bad:

js
app.listen(3000);

connectDatabase().catch(console.error);

Server accepts traffic before database ready.

Better:

js
async function start() {
  const config = loadConfig();

  const database = await connectDatabase(config.databaseUrl);

  const app = createApp({
    database,
    config,
  });

  const server = app.listen(config.port);

  return {
    server,
    database,
  };
}

try {
  const runtime = await start();
  installShutdown(runtime);
} catch (error) {
  logger.fatal({ error }, 'startup failed');
  process.exitCode = 1;
}

Fail fast before serving.

Dependency readiness

If optional dependency fails:

should service fail startup?
degrade feature?
retry?

Define explicitly.

For primary database required by every route, accepting traffic without it often causes noise.

Graceful shutdown

When SIGTERM arrives:

text
mark not ready
↓
stop accepting new requests
↓
allow bounded in-flight completion
↓
stop background polling/jobs
↓
close database/pools
↓
flush critical telemetry/logs
↓
exit

Use deadline.

Never wait forever.

Re-entrant shutdown

Signals/errors can happen twice.

js
let shuttingDown = false;

async function shutdown(reason) {
  if (shuttingDown) return;

  shuttingDown = true;

  ...
}

Protect lifecycle.

Native server close

js
server.close((error) => {
  ...
});

Modern Node includes additional connection-closing helpers in supported versions.

Understand:

new connections;
idle keep-alive;
active requests.

Do not assume close() instantly ends every socket.

Shutdown deadline

js
const shutdownTimer = setTimeout(() => {
  logger.fatal('forced shutdown deadline exceeded');
  process.exit(1);
}, 30_000);

shutdownTimer.unref();

Normal cleanup should clear it.

unref() prevents timer alone keeping event loop alive.

Use force-exit as last resort.

Abort background work

Global shutdown signal:

js
const lifecycle = new AbortController();

process.once('SIGTERM', () => {
  lifecycle.abort(new Error('shutdown'));
});

Pass signal to:

polling loops;
fetch;
jobs;
stream pipelines;
worker pools.

Polling loop

js
async function runPoller(signal) {
  while (!signal.aborted) {
    await pollOnce({ signal });

    await setTimeout(1000, undefined, {
      signal,
    });
  }
}

Use abort-aware timers where supported.

Do not create unbreakable while(true) workers.

Liveness versus readiness

Liveness

Is process alive enough that supervisor should not restart it?

Usually simple.

Readiness

Should this instance receive traffic now?

Can become false during:

startup;
shutdown/drain;
critical dependency unavailable if app cannot serve.

Do not make liveness depend on temporary database outage and cause restart storms.

Health endpoints

Example:

text
/live
/ready

Protect detail.

Public health should not expose:

text
database password
internal hosts
stack traces
deployment secrets

Stateless scaling

Multiple instances:

text
load balancer
→ Node A
→ Node B
→ Node C

Application state requiring sharing cannot live only in:

js
new Map()

Examples:

sessions;
rate limits;
job status;
shared cache.

Use external state or sticky architecture where justified.

Process supervisors

systemd

OS service manager.

PM2

Node-oriented process manager.

Docker/Kubernetes

Container/orchestrator controls lifecycle/restart/scaling.

Choose clear ownership.

Avoid:

text
PM2 cluster inside Docker pod
+
Kubernetes scaling
+
another supervisor

without a reason.

One process per container is common, but not absolute.

Restart policy

A crashed process should usually restart via supervisor.

But constant crash loops need:

backoff;
alert;
health status.

Do not hide fatal bugs by restarting thousands of times per minute.

`NODE_ENV`

Some libraries use:

text
NODE_ENV=production

Do not use NODE_ENV as full deployment configuration.

Use explicit:

text
APP_ENV
REGION
LOG_LEVEL
FEATURE flags

as needed.

Do not make business behavior depend on dozens of implicit environment conditions.

Secrets

Production secrets should come from:

orchestrator secret;
cloud secret manager;
vault;
mounted secret file;
protected environment injection.

Rotate.

Do not bake into container image.

Logging

Use JSON/structured stdout in containerized environments.

Collector handles transport.

Include:

text
timestamp
level
service
release
requestId
message
error code

Avoid multiline unstructured logs for machine processing.

Log levels

text
debug
info
warn
error
fatal

Production level is configuration.

Do not disable all useful logs for performance.

Do not log every request body.

Release metadata

Expose internally:

text
service version
git SHA
build ID

for diagnostics.

Health/status endpoint may include non-sensitive version.

This makes:

text
"error started after deploy abc123"

traceable.

Timeouts

Every network hop should have deadlines.

text
client → Node
Node → Mongo
Node → upstream

Without timeout, dead dependency consumes resources indefinitely.

Set timeout based on SLA and operation.

Retry policy

Retry only when:

error likely transient;
operation safe/idempotent;
deadline remains;
bounded attempts.

Use exponential backoff + jitter.

No retry loop:

js
while (true) {
  await fetch(...)
}

at full speed.

Retry storm

When dependency fails, every instance retries aggressively:

text
outage
→ more retries
→ dependency overloaded
→ recovery delayed

Use backoff, jitter, circuit breaking, concurrency limits.

Circuit breaker concept

After repeated failure:

text
open
→ fail fast temporarily
→ allow probes
→ close on recovery

Can protect system.

Do not add a circuit-breaker library blindly; timeouts and bounded retries come first.

Bulkhead

Separate limited pools/queues so one downstream cannot consume every resource.

Examples:

text
payment calls max 20 concurrent
report jobs max 4 workers

Prevents noisy dependency from taking all capacity.

Load shedding

When overloaded, reject early:

text
429
503

instead of accepting work until process OOMs.

Monitor queue depth/concurrency.

Database pool

Connection pool size times number of instances matters.

Example:

text
20 connections per instance
× 100 pods
= 2000 DB connections

Scale app and database together.

Do not max pool per process without global capacity planning.

Zero-downtime deploy

Sequence:

text
new instance starts
becomes ready
traffic shifts
old instance marked unready
SIGTERM
drain
exit

Readiness + graceful shutdown are essential.

Uncaught errors

Fatal unexpected:

log;
mark unhealthy/drain if possible;
exit;
supervisor restart.

Do not catch globally and continue forever.

Startup migration caution

Running database schema migration automatically in every replica at startup can cause races/long deploys.

Use explicit migration job/process when database type requires migrations.

Mongo schema migration strategies later.

Node watch and nodemon

Development:

bash
node --watch src/server.js

or nodemon.

Do not use dev watcher as production process supervisor.

PM2 basics

Can:

restart;
manage logs;
run multiple processes;
startup scripts.

If your deployment already has Kubernetes/systemd, evaluate whether PM2 adds value or duplicated supervision.

Container basics

Image should:

pin supported runtime base;
install lockfile deterministically;
run non-root where feasible;
copy only needed files;
avoid secrets;
use production dependencies/build artifacts;
define stop signal/health at platform layer.

Container security is broader DevOps topic but Node developer must understand lifecycle.

Common mistakes

listen before dependencies ready;
liveness checks DB and causes restart storm;
no shutdown timeout;
calling process.exit immediately on SIGTERM;
in-memory shared state across replicas;
nested supervisors;
no downstream timeouts;
retries unsafe writes;
connection pool multiplied across replicas;
secrets baked into image;
no release ID;
dev watcher in production.

Exercises

Write explicit start() lifecycle.
Add SIGTERM drain with deadline.
Add /live and /ready.
Make readiness false before server close.
Abort poller on shutdown.
Calculate DB pool capacity for 50 instances.
Design timeout/retry policy for upstream API.
Simulate retry storm and add backoff/jitter.
Draft Docker production checklist.
Draw zero-downtime deployment flow.

Mastery checklist

Explain:

startup sequencing;
graceful shutdown;
signals/deadline;
liveness/readiness;
stateless scaling;
supervisors;
secrets;
logging/release metadata;
timeouts/retries;
retry storms/circuit breaker;
pool capacity;
zero-downtime deploy.