The Outlier Pattern

Level 5 — Data Modeling & Schema Design The schema design pattern that uses document embedding by default to optimize read performance for 99% of standard cases, but flags and routes overflow data to a referenced collection for the 1% of "outlier" documents that would otherwise exceed the 16MB limit.

1. Prerequisites

• Embedding vs. Referencing — The parent modeling choice.
• Document Size Limit (16 MB) — The physical boundary ceiling.

2. Term Category

Data Modeling (Heavy Tail Document Handling Pattern): The Outlier Pattern handles exceptional documents with massive relationship counts (e.g., celebrity social media followers) without blowing up standard document limits.

3. Explanation

Environment Context

• Universal Standard (Supported conceptually across NoSQL platforms. Designed to handle extreme data skew (Power Law / Pareto distributions) in social networks and e-commerce catalogs).

(1) Design Motivation — "Why did we design this?"

In high-scale applications, data distributions often follow a Power Law (a tiny percentage of records hold a massive percentage of the data):

• 99% of books get less than 50 reviews. A celebrity book gets 500,000 reviews.
• 99% of users have less than 200 followers. A celebrity user gets 80 million followers.

If you are designing a social network and choose:

• Pure Embedding: Nesting followers in user documents. It works for 99% of users, but when the celebrity signs up, their document immediately exceeds the 16MB limit and crashes the system.
• Pure Referencing: Storing followers in a separate collection. You prevent crashes, but you penalize the read performance of 99% of normal users who could have loaded their profiles instantly in a single query.

We designed The Outlier Pattern to resolve this skew.

You design your schema to use Embedding by default to keep the normal case fast.

However, you add an Overflow Flag (has_overflow: true) to the document.

When a user's follower list grows close to a threshold (e.g. 1,000 items), the application sets the flag to true, caps the local array, and writes all subsequent followers into a separate, referenced collection.

(2) Read and Write Paths

• Write Path: Check the local array size. If it is less than 1,000, push directly. If it is 1,000, set has_overflow: true and write the excess to the overflow collection.
• Read Path: Fetch the parent document. Check the has_overflow boolean. If false, return the document immediately. If true, fetch the local document and run a query to load the overflow records.

(3) Reality Metaphor (Filing Folders)

Imagine storing client contract pages in folders:

• Normal Case: You place contract pages inside a standard Plastic Folder. 99% of clients have 10 pages, which fits slimly in a desk drawer.
• Outlier Case: A massive corporate client has 10,000 pages. The plastic folder bursts.
• Outlier Pattern: You store the first 100 pages in the plastic folder. You write a bright red note on the cover: "CONTINUED IN BOX 4" (the overflow flag). You place the remaining 9,900 pages in a heavy cardboard shipping box in the backroom (referenced collection).
  • Your desk drawers stay neat and slim for 99% of normal files, while the outlier client is handled safely.

(4) Code Examples

Implementing the Outlier Pattern (Books and Reviews)

The book document has an embedded reviews array and an overflow indicator:

// Collection: books (Normal Case: 'Database Guide' has 2 reviews, embedded)
{
  _id: 101,
  title: "Database Guide",
  has_overflow: false, // Flag is false
  reviews: [
    { reviewer: "Alice", score: 5 },
    { reviewer: "Bob", score: 4 }
  ]
}

// Collection: books (Outlier Case: 'Mega Bestseller' has hit the 1,000 cap)
{
  _id: 202,
  title: "Mega Bestseller",
  has_overflow: true, // Flag is set to true!
  reviews: [
    // Array is capped at 1,000 reviews to prevent document bloat
    { reviewer: "Reviewer_1", score: 5 },
    // ... up to 1000
  ]
}

// Collection: reviews_overflow (Stores overflow reviews for books where has_overflow is true)
{
  _id: ObjectId("65fc71239b1d8b2e88a8d000"),
  book_id: 202, // Linked reference
  reviewer: "Reviewer_1001",
  score: 5,
  comment: "Great book!"
}

4. Common Mistakes & Pitfalls

Mistake 1: Prematurely normalizing the entire database schema to reference all data, strictly to avoid outlier scenarios

The mistake: Using referencing for all users, books, and articles, forcing slow joins on every homepage query because "one day a celebrity might sign up."

Why it's wrong: You sacrifice the high read performance of a document database for 99% of your users to protect against a 1% edge case.

This leads to high database CPU load and slow app responses.

Fix: Default to embedding. Build the outlier pattern check into your application controller so that normal users stay fast, while outliers are dynamically routed to overflow collections.

Mistake 2: Over-Engineering Outlier Handling for Collections Without True Heavy Outliers

The mistake: Implementing complex Outlier Pattern logic for collections where all documents have similar uniform array sizes.

Why it's wrong: The Outlier Pattern is designed for extreme power-law distributions (e.g. 99.9% of users have 100 followers, but 1 celebrity user has 50,000,000 followers). Do not add complexity unless true outliers exist.

Incorrect:

// Implementing outlier flags for uniform collections

Fix:

Use standard embedding or referencing unless extreme outlier distribution exists

Mistake 3: Failing to Handle Overflow Flag hasOutlier in Application Query Code

The mistake: Querying user.followers array without checking hasOutlier: true overflow flags.

Why it's wrong: When an outlier document overflows, additional data is stored in overflow documents. Application code must check hasOutlier: true and fetch overflow records.

Incorrect:

const followers = user.followers; // ❌ Misses overflow records if hasOutlier is true!

Fix:

if (user.hasOutlier) { const extra = await db.followers_overflow.find({ userId }); }

5. Practice Exercises

Exercise 1: Handling Heavy-Tail Outlier Accounts

Scenario: Model a social network where standard users have ~100 followers (embedded in document), but celebrity accounts have 10,000,000 followers using the Outlier Pattern.

Requirements:

1. Add boolean flag hasOutlier: true on celebrity user documents.
2. Store overflow followers in a separate followers collection.

// Standard User Document (Embedded Followers)
db.users.insertOne({
  username: "standard_user",
  hasOutlier: false,
  followers: [new ObjectId(), new ObjectId()]
});

// Outlier Celebrity User Document (Flagged + Overflow Collection)
db.users.insertOne({
  _id: new ObjectId("60c72b2f9b1d8b2c88888899"),
  username: "celebrity_user",
  hasOutlier: true,
  followers: [ /* Top 1,000 followers embedded */ ]
});

// Overflow followers stored in separate collection
db.followers.insertOne({
  userId: new ObjectId("60c72b2f9b1d8b2c88888899"),
  followerId: new ObjectId()
});

Exercise 2: Querying Outlier Relationships

Scenario: Write an application query fetching followers for a user, checking hasOutlier to determine whether to query followers collection.

Requirements:

1. Check hasOutlier flag to route queries.

const user = db.users.findOne({ username: "celebrity_user" });
let followersList = user.followers;

if (user.hasOutlier) {
  // Fetch overflow followers from separate collection
  const overflow = db.followers.find({ userId: user._id }).toArray();
  followersList = followersList.concat(overflow.map(f => f.followerId));
}

Exercise 3: Identifying Outliers with Aggregation Thresholds

Scenario: Identify documents in users whose followers array length exceeds 1,000 items to flag them as outliers.

Requirements:

1. Aggregate using $where or $expr with $size.

db.users.find({
  $expr: { $gt: [{ $size: "$followers" }, 1000] }
});

6. Related Terms

• MongoDB —
• Time-Series Collections — Related concept: Time-Series Collections.
• Document Size Limit (16 MB) — Related concept: Document Size Limit (16 MB).

7. Key Takeaways

• The Outlier Pattern keeps 99% of documents fast while protecting against the 1% edge cases.
• Embeds arrays by default for standard users.
• Caps the embedded array size and sets an overflow flag (e.g. has_overflow: true).
• Writes excess data to a separate, referenced collection to bypass the 16MB limit.
• Highly effective for handling data skew (Power Law distributions).
• Avoid premature normalization; default to embedding.
• Requires application-side logic to handle conditional double queries.