140: MongoDB Replica Sets and High Availability — Elections, Oplog, Failover, Read Scaling, and Recovery Semantics

Learning objectives

You will learn to:

understand replica-set architecture;
distinguish primary and secondary members;
understand replication oplog at a practical level;
understand elections;
understand failover behavior;
understand rollback risk;
understand majority commit point;
understand read preference trade-offs;
plan application behavior during failover;
understand connection-string discovery;
understand hidden/delayed/arbiter concepts at a high level;
avoid using replication as backup.

Replica set mental model

text
Primary
  │ writes
  ↓
oplog
 ↙   ↘
Secondary A
Secondary B

Clients normally send writes to primary.

Secondaries replicate primary operations.

If primary unavailable, eligible members can elect a new primary.

Why replication exists

high availability;
redundancy;
failover;
disaster-resilience building block;
read distribution for suitable workloads.

Replication is not backup.

If application deletes all records, deletion replicates too.

Primary

Accepts writes under normal replica-set operation.

Reads default to primary unless read preference changed.

Only one primary at a time per replica set.

Secondary

Replicates oplog and applies operations.

Can serve reads when read preference permits.

May lag.

Do not assume secondary is perfectly current.

Oplog

Replica set maintains capped operation log.

Secondaries follow it.

Oplog window represents how much history is retained.

If secondary falls too far behind beyond oplog history, it may need initial sync rather than catching up incrementally.

Monitor oplog window for recovery/maintenance expectations.

Replication lag

Difference between primary progress and secondary apply time.

Causes:

network;
slow disks;
heavy writes;
long operations;
resource saturation.

Secondary read may be stale.

Monitor.

Election

When primary unavailable, members coordinate election.

During election:

text
writes can temporarily fail/pause
driver discovers new primary
retryable operations may retry

Applications must expect transient errors.

Do not treat a 1-second failover as “database corruption.”

Driver discovery

Use replica-set/Atlas connection string listing/discovering topology.

Driver monitors cluster.

Do not pin application permanently to one node IP.

Example Atlas SRV:

text
mongodb+srv://cluster...

Driver selects server based on operation/read preference.

Majority

A write acknowledged with majority means majority of voting data-bearing members acknowledge according to replication semantics.

Majority commit point relates to durability/visibility.

This reduces rollback exposure.

Exact behavior depends on deployment/storage.

Rollback

If former primary accepted write not majority committed and loses leadership, conflicting operations may be rolled back during reconciliation.

This is why write concern matters for durability requirements.

Read concern majority

Reading majority helps avoid seeing data not majority committed.

Trade latency/availability.

Use business requirements.

Failover test

Production readiness requires testing:

sustained writes;
kill/step down primary;
observe client errors/retries;
new primary election;
verify no duplicates;
verify latency;
verify monitoring alerts.

Do not discover failover behavior first time in outage.

`rs.stepDown()`

Administrative command can trigger primary stepdown for maintenance/testing.

Use controlled environment and current docs.

Do not run blindly on production without operational plan.

Secondary reads

Can improve:

analytics/reporting;
geographically closer reads;
primary load.

But can harm:

read-after-write;
authorization;
inventory;
real-time queue.

Match endpoint semantics.

Nearest

Read preference nearest selects based on latency window/topology, not necessarily geographically “nearest” in simple terms.

Can return secondary data.

Do not use for correctness-critical reads solely for latency.

Hidden member

Replica-set member hidden from normal client reads.

Can be used for dedicated backup/reporting in some designs.

Still participates replication/elections depending votes/priority configuration.

Use current guidance.

Delayed member

Intentionally delays replication.

Can provide recovery window against some logical errors.

But not substitute for backups and can complicate elections/capacity.

Arbiter

Arbiter votes but stores no data.

Modern production architecture often prefers data-bearing voting members for redundancy.

Use arbiters only when architecture/guidance supports; understand durability implications.

Three-member replica set

Common baseline:

text
3 data-bearing members

can tolerate one member failure and still maintain majority.

Placement across failure domains matters.

Three members in same physical failure zone do not protect zone outage.

Failure domains

Place members across:

availability zones;
racks;
regions where latency allows.

Trade-off:

text
geographic resilience
vs
write latency

Majority write across far regions costs more latency.

Network partitions

Replica-set voting prevents both partitions from independently having valid primary if majority rules work correctly.

Minority side loses primary ability.

This prioritizes consistency.

Clients on minority partition may lose write availability.

Distributed systems trade-offs are real.

Write availability

To elect primary/ack majority, enough voting/data-bearing members must communicate.

If majority unavailable, writes can stop.

Do not promise 100% write availability under any partition.

Read availability

Depending read preference/read concern, some reads may continue from secondaries.

But serving stale reads may violate product semantics.

Availability is operation-specific.

Maintenance

Rolling restart:

text
secondary restart
catch up
next secondary
primary stepdown/restart

Managed services automate much.

Check replication lag before proceeding.

Backup from replica

Backups often use secondary/snapshots to reduce primary impact.

Still require consistency-aware snapshot method.

Do not copy live database files arbitrarily.

Connection pool during failover

Driver pools connections to topology.

Failover can cause transient pool churn/selection delays.

Set reasonable server selection/operation timeouts.

Do not set 100ms timeouts if elections can exceed that.

Health checks

Application readiness should consider database availability without causing restart loops.

Liveness should not kill process simply because Mongo primary is electing.

Use operation-aware readiness.

Monitoring

Track:

text
member state
replication lag
oplog window
elections
primary changes
connections
disk
CPU
cache
write latency
majority commit lag

Atlas provides managed metrics/alerts.

Replication and indexes

Indexes exist on members.

Index builds consume resources across replica set.

Plan production index changes.

Replication and TTL

TTL deletes execute/replicate according to Mongo internals.

Do not depend on exact deletion timestamp.

Replication and transactions

Transactions rely on replica-set/sharded deployment.

Standalone Mongo has different transaction capability limitations.

Use production-like topology in integration tests for transactions/change streams.

Local development

A single standalone is easy but hides:

transaction topology;
failover;
retryable behavior;
change streams.

For advanced tests, run local replica set/container or Atlas test cluster.

Disaster scenarios

Replica protects node failure.

It does not protect:

operator drops database;
bad migration;
ransomware with DB credentials;
corrupted app writes;
long-undetected logical error.

Backups/PITR needed.

Failure clinic

replication = backup misconception;
secondary used for auth check;
no failover tests;
app connects single hostname/node;
low timeout makes every election incident severe;
all members same failure domain;
arbiter chosen without durability reasoning;
no oplog/lag monitoring;
liveness restarts app during DB election;
standalone dev hides transaction topology behavior.

Exercises

Draw 3-member replica set.
Explain oplog.
Choose read preference for analytics versus checkout.
Simulate primary stepdown in test cluster.
Measure client retry behavior.
Calculate failure-domain placement.
Design monitoring alert for lag.
Explain why logical delete survives replicas.
Build readiness policy during election.
Document write concern/read preference matrix.

Mastery checklist

Explain:

primary/secondary;
oplog;
election;
failover;
lag;
majority;
rollback;
read preference;
failure domains;
partitions;
driver topology;
replication versus backup.