How OpenAI Scaled Postgres to 800 Million Users (Step-by-Step)

Summary

The video explains how OpenAI’s PostgreSQL setup, as described in its article “Scaling Postgres to Power 800 Million ChatGPT Users,” can handle very large workloads. The presenter uses a diagram-based walkthrough and emphasizes that scaling involves more than adding database hardware.

Architecture and scaling techniques

• Start with vertical scaling: A single database can often be made more capable by adding resources. The presenter describes this as the simplest initial approach.
• Use one primary and many read replicas: OpenAI is described as keeping a single primary database for writes and operating roughly 50 read replicas. PostgreSQL’s write-ahead log (WAL) streams changes from the primary so replicas can apply them.
• Route reads and writes separately: Inserts, updates, and deletes go to the primary; read queries are distributed across replicas. This helps scale workloads where reads greatly outnumber writes.
• Plan for failure: Replicas may continue serving reads if the primary fails, though writes stop. A designated standby replica can be promoted to primary to reduce downtime.
• Pool connections with PgBouncer: Connection pooling avoids repeatedly creating database connections, which can become a significant bottleneck at scale. The presenter says OpenAI reported reducing connection-acquisition time from about 50 ms to 5 ms. He also describes a PgBouncer instance associated with each read replica.
• Consider cascading replication: Instead of having the primary send WAL directly to every replica, existing replicas can relay changes to additional replicas. The presenter says OpenAI was exploring this approach, not that it had already implemented it.
• Add caching and protect against stampedes: A cache can serve frequently requested data before it reaches the database. Strict locking can prevent many requests from simultaneously fetching an item after its cache entry expires. The presenter speculates that OpenAI may use Redis, but says the article does not identify the cache technology.
• Route traffic geographically: Incoming requests can be directed toward nearby available infrastructure.
• Optimize queries and control load: The video highlights appropriate indexing, avoiding unnecessary data retrieval, and doing some joins in memory to reduce database CPU work. Rate limits at the load balancer, PgBouncer, and application levels help prevent traffic bursts from overwhelming the databases.
• Manage schema changes carefully: Operations that could lock an entire database—such as changing a column’s data type—are described as prohibited for existing tables.

Guides and examples mentioned

• The video is a visual walkthrough of the scaling architecture rather than a step-by-step review of OpenAI’s article.
• The presenter points viewers to a prior video demonstrating cache-stampede prevention, including how the approach could extend to a distributed system.
• He also mentions a sample project using a primary database and a read replica with AWS RDS, shared in the pinned comment.

Main speakers and sources

• Milan Jovanović: Presenter and narrator; explains the architecture and relates it to systems he has worked on.
• OpenAI’s “Scaling Postgres to Power 800 Million ChatGPT Users” article: The infrastructure source discussed in the video.