143: MongoDB Backup, Recovery, Monitoring, Profiling, Maintenance, and Operational Diagnostics

Learning objectives

You will learn to:

distinguish replication from backup;
understand logical backup with mongodump/mongorestore;
understand snapshot backups and point-in-time recovery;
design recovery objectives;
test restores;
use monitoring metrics;
use database profiler/current operations carefully;
inspect slow queries;
understand connection capacity;
understand maintenance/index operations;
plan version upgrades and feature compatibility;
write an operational runbook.

Recovery starts with requirements

RPO

Recovery Point Objective:

How much data loss is acceptable?

Example:

text
RPO 5 minutes

RTO

Recovery Time Objective:

How long may recovery take?

Example:

text
RTO 30 minutes

Backup design must satisfy both.

“Daily backup” is not enough information.

Replication is not backup

Replica set mirrors changes.

Accidental:

javascript
db.orders.deleteMany({})

replicates.

Backup provides historical recovery.

Backup types

Logical

mongodump

Produces BSON/metadata dump.

Good for:

smaller deployments;
migration/export;
selected DB/collections.

Can be slow/large for huge DB.

Physical/snapshot

Storage-level or Atlas snapshots.

Efficient for large datasets.

Need consistency-aware supported mechanism.

Point-in-time recovery

Uses snapshots + continuous oplog-like history to restore near target timestamp.

Atlas offers managed PITR depending tier/configuration.

`mongodump`

Example:

bash
mongodump \
  --uri="$MONGODB_URI" \
  --out="./backup"

Do not put secret URI directly in shell history when avoidable.

Protect output.

`mongorestore`

bash
mongorestore \
  --uri="$TARGET_URI" \
  "./backup"

Restore into isolated environment first.

Do not test restore by overwriting production.

Namespace selection

Dump selected DB/collection can reduce scope.

But recovery consistency may require related collections together.

Example:

text
orders
payments
outbox

restored from different times can violate business invariants.

Design backup unit.

Consistent backups

On replica sets/shards, use supported procedures to ensure consistent point-in-time backup.

Do not copy data directory while server is running without supported snapshot mechanics.

Atlas managed backups reduce complexity.

Backup encryption

At rest and transit.

Keys separate/protected.

Backup access can be more dangerous than live DB because it contains broad historical data.

Retention

Example:

text
hourly 48h
daily 30d
monthly 12m

Depends compliance/business.

Retention costs storage and privacy obligations.

Expired data should be deleted from backup according to policy/legal requirements where feasible.

Restore testing

A backup not tested is an assumption.

Scheduled drill:

select backup;
restore isolated;
run integrity checks;
run app smoke tests;
measure time;
document failures;
destroy test environment safely.

Track actual RTO.

Point-in-time recovery drill

Simulate:

text
bad deploy 14:05
detected 14:22
restore to 14:04:30

Then reconcile external side effects after restore.

Database recovery can rewind:

text
order status

but cannot undo email/payment already sent externally.

Cross-system recovery requires business procedures.

MongoDB monitoring

Important dimensions:

text
connections
operations/sec
query latency
CPU
disk latency
disk space
WiredTiger cache
page faults/IO
replication lag
oplog window
locks/tickets
network
index usage
slow queries
shard balance

Atlas provides dashboards/alerts.

Self-managed uses monitoring stack.

WiredTiger cache

MongoDB storage engine uses cache.

Working set larger than memory causes more disk reads.

Do not interpret “Mongo uses lots of memory” as leak; database intentionally caches.

Monitor eviction/cache pressure.

Disk latency

Database performance depends strongly on storage.

High IOPS/latency causes:

query slowness;
replication lag;
checkpoint issues.

CPU tuning will not fix slow disk.

Connections

Drivers use pools.

Monitor current/available.

Too many app instances × huge pool can exhaust server.

A pool size of 100 per pod × 200 pods = 20,000 potential connections.

Capacity plan globally.

Slow query profiler

MongoDB profiler can collect detailed operation info.

Use carefully—profiling itself has overhead and may capture sensitive query values.

In production prefer slow-operation thresholds/diagnostic tools according to Mongo guidance.

Do not enable full profiling indefinitely without plan.

Current operations

Administrative commands can inspect current long-running operations.

Useful for incident:

text
what is stuck?
which namespace?
how long?

Killing operations is an emergency tool, not query optimization strategy.

`explain`

First tool for query performance.

Compare:

text
nReturned
keys examined
docs examined
execution time
sort

Don't start by increasing server RAM if query scans 50M docs for 10 results.

`$indexStats`

Can inspect index usage.

Use to identify candidates for cleanup.

But rare critical index may show low use.

Combine workload knowledge.

Database logs

Mongo logs:

connections;
slow operations;
elections;
checkpoints;
warnings/errors.

Centralize and protect.

Sensitive details may exist.

Query `comment`

Drivers can attach comments/metadata to operations in supported APIs.

Useful to identify application operation in profiler/logs.

Do not include sensitive user text.

Example logical:

text
comment: "GET /tasks list"

`maxTimeMS`

Bound expensive queries.

Timeout protects availability.

But if normal query always hits timeout, fix index/model.

Maintenance

Index creation/drop

Plan resource usage.

Compact/reclaim

Storage maintenance commands have significant impact and version-specific behavior.

Do not run old blog commands blindly.

Validation

validate command checks collection/storage consistency.

Use during diagnostics/maintenance according to guidance.

Roadmap includes validate() concept.

`validate`

Shell:

javascript
db.runCommand({
  validate: "tasks"
})

Can be resource intensive.

Do not schedule on hot production blindly.

Feature Compatibility Version (FCV)

During upgrades, Mongo uses FCV to control feature behavior.

MongoDB 8.3 upgrade from 8.0/8.2 requires prescribed paths/FCV state.

Follow official upgrade guide exactly.

Do not skip versions because “files are compatible.”

Upgrade plan

verify supported driver/Mongoose;
backup;
test staging;
review compatibility changes;
upgrade adjacent supported path;
verify members;
set FCV when instructed;
monitor;
have downgrade plan within supported window.

Patch versions

Security/reliability fixes arrive in patches.

Production should not remain on .0 indefinitely.

Track release notes/advisories.

Driver compatibility

Before DB upgrade, verify Node Mongo driver compatibility.

Do not upgrade server to feature not understood by old driver.

Change management

For major index/schema changes:

canary;
hidden index;
dual reads;
background migration;
metrics;
rollback.

Database changes are deployments.

Data integrity checks

Examples:

text
required counts
orphan references
tenant nulls
duplicate business keys
invalid type distributions
negative amounts
schemaVersion distribution

Build scripts/reporting.

Do not rely only on Mongo structural validation for business integrity.

Operational runbook

For “Mongo slow”:

text
1. check alerts/topology
2. confirm primary/election
3. query latency
4. CPU/disk/cache
5. connection saturation
6. slow ops/explain
7. recent deploy/index
8. replication lag
9. shard imbalance
10. mitigate

For “disk 90%”:

identify growth;
backup safety;
add capacity;
retention;
indexes;
logs;
avoid emergency deletes without plan.

Disaster recovery runbook

Include:

text
who declares incident
backup location
credentials/key access
restore command/process
target cluster
DNS/connection switch
integrity validation
external-system reconciliation
communications
postmortem

Common mistakes

no restore tests;
backup on same server only;
replication called backup;
full profiler always on;
giant connection pools;
ignore disk latency;
kill slow query instead of index;
upgrade without FCV plan;
use old maintenance commands;
no patch policy;
backup secrets exposed;
RPO/RTO undefined.

Exercises

Define RPO/RTO for three systems.
Run mongodump/mongorestore on test DB.
Time restore.
Build PITR incident scenario.
Monitor connections/cache/lag.
Use explain on slow query.
Inspect index stats.
Write upgrade checklist for 8.3.
Design connection pool capacity.
Write production Mongo runbook.

Mastery checklist

Explain:

logical/snapshot/PITR;
RPO/RTO;
restore drills;
monitoring;
cache/disk;
profiler/current ops;
connections;
validate;
FCV/upgrades;
patching;
runbooks.

143: MongoDB Backup, Recovery, Monitoring, Profiling, Maintenance, and Operational Diagnostics

Learning objectives

Recovery starts with requirements

RPO

RTO

Replication is not backup

Backup types

Logical

Physical/snapshot

Point-in-time recovery

mongodump

mongorestore

Namespace selection

Consistent backups

Backup encryption

Retention

Restore testing

Point-in-time recovery drill

MongoDB monitoring

WiredTiger cache

Disk latency

Connections

Slow query profiler

Current operations

explain

$indexStats

Database logs

Query comment

maxTimeMS

Maintenance

Index creation/drop

Compact/reclaim

Validation

validate

Feature Compatibility Version (FCV)

Upgrade plan

Patch versions

Driver compatibility

Change management

Data integrity checks

Operational runbook

Disaster recovery runbook

Common mistakes

Exercises

Mastery checklist

Official references

`mongodump`

`mongorestore`

`explain`

`$indexStats`

Query `comment`

`maxTimeMS`

`validate`