The Clean Break: Migrating a Production Graph from MongoDB to PostgreSQL Under Load

August 2026 · PostgreSQL, MongoDB, Database Architecture, Ruby on Rails, Retrospective, Coderwall · View Markdown

Around 2012, early-stage startups flocked to document databases. The promise was alluring: schema-less JSON storage, instant prototyping, and no upfront migration planning. Coderwall’s early data layer was built on MongoDB using the Mongoid ODM.

For the initial launch, MongoDB worked well enough. But by 2014, as Coderwall grew into a mature developer network with hundreds of thousands of user profiles, teams, protips, and badge endorsements, the cracks in the document model became impossible to ignore.

As part of leading the open-source transition for Coderwall’s founder, one of my core mandates was moving the entire data plane to PostgreSQL. Here is why the document model failed under load, and how we pulled off a zero-downtime relational migration.

Why Document Stores Broke Down on a Developer Graph

A developer reputation platform is inherently relational:

  • Users belong to Teams.
  • Teams have Members and aggregate company-wide Badges.
  • Users author Protips, which receive Likes, Comments, and Topic Tags.
  • Badges are awarded based on complex multi-repository activity criteria.

In MongoDB, representing these relationships forced two bad design choices:

  1. Massive Embedded Documents: Embedding comments and likes inside a Protip document caused documents to grow unpredictably. In MongoDB, when an embedded document exceeded its allocated memory block, the entire record had to be rewritten to a new disk sector, causing severe I/O spikes.
  2. Client-Side Joins: Storing ObjectIDs and referencing related documents meant the Rails application had to execute multiple N+1 queries over the wire to assemble a single user profile page.

Without foreign key constraints, data integrity quietly drifted. Orphaned records, broken references, and inconsistent aggregations accumulated in production.

The Two-Phase Migration Strategy

You cannot shut down a live, global platform for a multi-day database rewrite. We executed the transition across two pull requests (#226 and #227) using an active dual-read/dual-write pattern:

[ Incoming Request ]
         │
         ├──► Write to MongoDB (Legacy Source of Truth)
         └──► Write to PostgreSQL (ActiveRecord Shadow Table)

Phase 1: Migrating Team and User Schemas (PR #227)

We started with high-concurrency relational models where consistency mattered most: Teams, Memberships, and User Profiles.

  • Created explicit PostgreSQL schemas using ActiveRecord migrations with strict NOT NULL constraints, foreign keys, and indexes.
  • Wrote background backfill scripts using Sidekiq workers to stream records from MongoDB into Postgres in batches.
  • Verified row counts and checksums across both data stores.

Phase 2: Converting Remaining Mongoid References (PR #226)

Once core entities were synced and verified, we severed the remaining Mongoid dependencies:

  • Converted badge awards, opportunity listings, and protip relationships to ActiveRecord associations.
  • Switched production read queries to point exclusively to PostgreSQL.
  • Removed mongoid.yml, stripped MongoDB gems from the Gemfile, and dropped the legacy cluster.

Unlocking High Performance with Smarter Postgres Indexing

Moving to PostgreSQL didn’t just solve data integrity, it radically improved query performance and reduced hosting costs on Heroku.

Instead of running separate search infrastructure for simple string lookups, we leveraged native PostgreSQL indexing capabilities:

  1. Trigram & GiST Indexes for Fuzzy Search: Replacing expensive regex scans with trigram matching (pg_trgm) allowed sub-millisecond autocompletion on usernames, skills, and company networks.
  2. Partial Indexes for Active Records: Instead of indexing entire tables, we indexed only active, public opportunities and verified protips: 
    CREATE INDEX index_active_opportunities ON opportunities (created_at DESC) WHERE active = true;
  3. Compound Foreign Key Indexes: Indexing composite join pairs (team_id, user_id) eliminated table scans on high-traffic team directories.

💡 The Enduring Lesson for Modern Data Systems

In 2026, the tech industry is experiencing a similar wave of hype around specialized document engines, vector databases, and schemaless storage for AI workflows.

The lessons from Coderwall’s database migration remain unambiguous:

  1. Every system eventually becomes relational. If entities reference each other, you are building a relational graph. Trying to manage foreign keys and referential integrity in application code will always be slower and more bug-prone than letting a battle-tested relational engine enforce ACID guarantees.
  2. Operational simplicity wins. Consolidating from a multi-database setup (MongoDB + Redis + Postgres) down to PostgreSQL and Redis simplified the local onboarding setup for open-source contributors and cut production dyno costs immediately.
  3. Good schema design is an engineering superpower. Strict database types and constraints don’t slow down development, they prevent silent data corruption and give future maintainers a clear contract of how the system actually works.

In Part 3, we’ll look at the community tooling of the era: how UGl.st and Chicago Code Camp connected regional developer groups before centralized platforms took over.