138: MongoDB Aggregation — Pipelines, Expressions, `$match`, `$group`, `$unwind`, `$lookup`, `$facet`, and Optimization

Learning objectives

You will learn to:

understand aggregation pipelines;
use common stages;
distinguish stages from expressions;
filter/project/transform;
group and calculate metrics;
unwind arrays;
join collections with $lookup;
build multi-result $facet pipelines;
understand accumulators;
use date/string/array expressions;
reason about stage ordering and index use;
use explain() for aggregation;
understand memory and disk-use concerns;
know when precomputation is better than repeated heavy aggregation.

Pipeline mental model

Documents flow through stages:

text
collection
↓
$match
↓
$project / $set
↓
$group
↓
$sort
↓
$limit
↓
result

Each stage consumes documents and emits documents.

Example:

javascript
db.orders.aggregate([
  {
    $match: {
      tenantId,
      status: "completed"
    }
  },
  {
    $group: {
      _id: "$customerId",
      totalPaise: {
        $sum: "$totalPaise"
      }
    }
  },
  {
    $sort: {
      totalPaise: -1
    }
  },
  {
    $limit: 10
  }
])

Stage versus expression

Stage:

javascript
{
  $match: {...}
}

Expression inside stage:

javascript
{
  $project: {
    total: {
      $multiply: [
        "$quantity",
        "$price"
      ]
    }
  }
}

Operators can exist in different contexts; read docs for exact syntax.

`$match`

Like query filter.

Place selective match early where possible:

javascript
{
  $match: {
    tenantId,
    createdAt: {
      $gte: start,
      $lt: end
    }
  }
}

Early match can use indexes when pipeline/query optimizer permits.

Do not start with expensive $lookup then filter tenant/date afterward if filter can happen first.

`$project`

Choose/reshape:

javascript
{
  $project: {
    _id: 0,
    orderId: "$_id",
    customerId: 1,
    totalPaise: 1
  }
}

Can compute expressions.

Do not project giant fields you do not need.

`$set` / `$addFields`

Add/replace fields:

javascript
{
  $set: {
    totalItems: {
      $sum: "$items.quantity"
    }
  }
}

Exact behavior of array expressions must be tested.

`$unset`

Remove fields.

Useful to drop sensitive/large fields during pipeline.

`$group`

javascript
{
  $group: {
    _id: "$status",
    count: {
      $sum: 1
    },
    totalPaise: {
      $sum: "$totalPaise"
    },
    averagePaise: {
      $avg: "$totalPaise"
    }
  }
}

Accumulators:

text
$sum
$avg
$min
$max
$first
$last
$push
$addToSet

Ordering matters for $first/$last; sort appropriately.

Group cardinality

Grouping by near-unique field creates many groups and memory use.

Example:

javascript
_group by requestId

may be pointless/expensive.

Understand desired aggregation.

`$sort`

javascript
{
  $sort: {
    totalPaise: -1
  }
}

If sort happens after group, index cannot directly sort grouped synthetic output.

This may require memory/disk.

Limit early when semantics allow.

`$limit`

javascript
{
  $limit: 20
}

Can reduce downstream work.

But moving limit before group changes result semantics.

Optimization must preserve meaning.

`$skip`

Useful but deep skip has same pagination concerns.

For reporting output, acceptable in small datasets.

For user-facing large pagination, range/cursor better.

`$unwind`

Document:

javascript
{
  items: [
    { sku: "A", quantity: 2 },
    { sku: "B", quantity: 1 }
  ]
}

Unwind:

javascript
{
  $unwind: "$items"
}

Produces one pipeline document per item.

Then:

javascript
{
  $group: {
    _id: "$items.sku",
    units: {
      $sum: "$items.quantity"
    }
  }
}

Preserve empty/null arrays

$unwind options can preserve null/empty and include array index.

Understand when missing items should disappear versus remain.

`$lookup`

Join-like stage.

Orders:

javascript
{
  customerId: ObjectId(...)
}

Customers collection:

javascript
{
  _id: ObjectId(...),
  name: "Maya"
}

Simple:

javascript
{
  $lookup: {
    from: "customers",
    localField: "customerId",
    foreignField: "_id",
    as: "customer"
  }
}

Produces array.

Then often:

javascript
{
  $unwind: "$customer"
}

`$lookup` pipeline

More powerful:

javascript
{
  $lookup: {
    from: "payments",
    let: {
      orderId: "$_id"
    },
    pipeline: [
      {
        $match: {
          $expr: {
            $eq: [
              "$orderId",
              "$$orderId"
            ]
          }
        }
      },
      {
        $project: {
          amountPaise: 1,
          status: 1
        }
      }
    ],
    as: "payments"
  }
}

Foreign collection indexes matter.

A bad lookup can behave like massive nested join.

`$expr`

Allows aggregation expressions in query matching:

javascript
{
  $match: {
    $expr: {
      $gt: [
        "$paidPaise",
        "$totalPaise"
      ]
    }
  }
}

Indexability depends on expression form/context.

Use explain.

`$facet`

Run multiple subpipelines on same input.

Example list + count:

javascript
[
  {
    $match: filter
  },
  {
    $facet: {
      data: [
        { $sort: { createdAt: -1 } },
        { $limit: 20 }
      ],
      meta: [
        { $count: "total" }
      ]
    }
  }
]

Convenient.

But exact total count on large filtered data can be expensive and may negate pagination efficiency.

Do not add total count because UI designer expects it without measuring.

`$count`

javascript
{
  $count: "total"
}

`$replaceRoot` / `$replaceWith`

Promote embedded document as root.

Useful after lookup/unwind/transformation.

`$map`

Transform arrays:

javascript
{
  $project: {
    itemNames: {
      $map: {
        input: "$items",
        as: "item",
        in: "$$item.name"
      }
    }
  }
}

`$filter`

javascript
{
  $project: {
    activeItems: {
      $filter: {
        input: "$items",
        as: "item",
        cond: {
          $eq: [
            "$$item.active",
            true
          ]
        }
      }
    }
  }
}

`$reduce`

Reduce array to value.

Can implement advanced transformations.

Prefer readable pipelines; deeply nested expressions become hard to maintain/test.

Conditional

text
$cond
$switch
$ifNull

Useful for classification/defaults.

String/date

Operators:

text
$toLower
$concat
$dateTrunc
$dateToString
$dateDiff

Be explicit about timezone.

Reporting grouped by “day” depends on business timezone.

Do not assume UTC midnight matches local business day.

Type conversion

text
$convert
$toString
$toInt
$toDecimal

Can clean historical inconsistent data, but repeated runtime conversion may prevent index use and hide schema problem.

Prefer data migration to consistent types.

Window functions

Modern MongoDB supports window stages/operators such as $setWindowFields.

Useful for:

running totals;
ranks;
moving averages.

Example concept:

text
partition by customer
sort by date
running total

Advanced analytics should be tested for memory/performance.

`$unionWith`

Combine collection/pipeline results.

Useful for cross-collection reporting.

Frequent unions may signal data architecture needs review.

`$out` and `$merge`

Write pipeline results to collection.

Useful for:

materialized projections;
ETL;
precomputation.

Dangerous/destructive semantics require careful permissions and rollout.

Do not expose arbitrary pipeline execution to public client.

Precomputation

Heavy report every request:

text
scan 20M orders
lookup 5 collections
group
sort

may be wrong architecture.

Options:

computed fields;
materialized summary collection;
scheduled aggregation;
event-driven projection;
analytics warehouse.

Mongo aggregation is powerful, not infinite free compute.

Memory and disk

Blocking stages like:

text
$sort
$group

may need memory.

Mongo can spill to disk under supported behavior/options.

Disk spill prevents memory failure but can be slow.

Design indexes/preaggregation.

Pipeline optimization

Mongo optimizer can reorder/coalesce stages.

Still write logically efficient pipeline:

selective $match;
reduce fields;
limit;
indexed lookups;
avoid exploding unwind unnecessarily.

Use explain to verify actual plan.

Explain

javascript
db.orders.explain("executionStats").aggregate([
  ...
])

Inspect:

cursor plan;
index;
docs/keys examined;
stage behavior;
execution stats.

Specific explain structure can vary by version.

Tenant security

Always include tenant filter before joins/reporting:

javascript
{
  $match: {
    tenantId: authTenantId
  }
}

Do not let public client pass tenant pipeline stage.

For $lookup, ensure joined data cannot cross tenant boundaries if foreign collection IDs are not globally isolated/secured.

Use tenant predicates as needed.

Aggregation injection

Never accept:

json
{
  "pipeline": [...]
}

from untrusted client and run directly.

Pipeline can:

access unintended fields;
perform expensive work;
write via stages under privileges;
exfiltrate data.

Expose a safe reporting DSL/allowlisted parameters instead.

Common mistakes

lookup before selective match;
exact count on huge list by default;
unwind exploding documents unexpectedly;
group by high-cardinality field;
no index foreignField;
timezone ignored;
runtime type conversion hides dirty schema;
total aggregation on request hot path;
arbitrary client pipeline;
missing tenant scope;
assuming aggregate pipeline order always equals physical execution without explain.

Exercises

Monthly order totals by status.
Unwind items and calculate top SKUs.
Lookup customer display names.
Add foreign index and compare.
Build facet list+count and measure.
Group by business-local day.
Use map/filter on embedded arrays.
Build running total with window functions.
Materialize a summary collection.
Explain pipeline and identify bottleneck.

Mastery checklist

Explain:

pipeline;
stages/expressions;
match/project/set;
group/accumulators;
sort/limit;
unwind;
lookup;
facet;
array/date expressions;
window functions;
merge/out;
optimization/explain;
precomputation;
tenant/injection safety.

138: MongoDB Aggregation — Pipelines, Expressions, `$match`, `$group`, `$unwind`, `$lookup`, `$facet`, and Optimization

Learning objectives

Pipeline mental model

Stage versus expression

$match

$project

$set / $addFields

$unset

$group

Group cardinality

$sort

$limit

$skip

$unwind