Skip to content

Memcached at LiveJournal: A Technical Case Study

Technical Case Study Prompt Create a detailed technical case study of the creation of Memcache at Livejournal Use the STAR process 1. Situation - give context to the case study 2. Task - what was the challenge being addressed 3. Approach - What was the architectural approach? 4. Results - What were the results, both measurable and intangible

Executive Summary

In 2003, Brad Fitzpatrick and the LiveJournal team created Memcached, a distributed memory caching system that would become one of the foundational technologies for web-scale applications. What began as an internal solution to LiveJournal's database scaling crisis has since been adopted by major platforms including YouTube, Reddit, Facebook, Pinterest, Twitter, Wikipedia, and countless other web applications. This case study examines the technical challenges, architectural approach, and transformative results of Memcached's creation.

Situation: The Growing Pains of Early Social Media

Rapid User Growth and Platform Evolution

LiveJournal was started by Brad Fitzpatrick on April 15, 1999, as a way of keeping his high school friends updated on his activities. By 2003, the platform had grown to well over 4,000,000 accounts, over half of which were in active use, with built-in social networking, per-journal-entry security, message boards, and support for 20+ languages.

The platform's rapid growth created significant technical challenges:

  • Exponential User Base Growth: The rate of growth was faster than the server architecture could handle, necessitating an "invite code" system from September 2001 to December 2003 to control membership growth
  • High-Activity User Engagement: LiveJournal users were highly active, posting frequently and reading each other's journals extensively
  • Complex Social Features: The platform included sophisticated features like friend networks, commenting systems, and community journals that created complex relationship patterns in the database

Infrastructure Architecture Constraints

By 2003, LiveJournal was operating with: - Multiple MySQL Database Clusters: The site had evolved from one server to over sixty, adapting both code and architecture as the site grew - Web Server Scaling: The team could add web servers relatively easily, but database capacity remained the primary bottleneck - Traditional Database Architecture: Standard master-slave MySQL replication with read replicas

The team faced the classic early-2000s web scaling challenge: while web servers could be added relatively easily and cheaply, database scaling required expensive, high-end hardware and complex architectural changes.

Task: Solving the Database Performance Crisis

The Core Challenge: Database Overload

The main database problem was that when they got behind on buying new database clusters, existing user clusters became overloaded and too many writes happened on them, making them unable to do anything else usefully. This created a cascading performance problem:

Write Bottlenecks: High-volume write operations (new posts, comments, friend additions) were overwhelming the master databases, creating queues and delays.

Read Performance Degradation: As write load increased, read performance suffered dramatically. Simple operations like loading a user's friends page or displaying a journal entry became painfully slow.

Scaling Limitations: The performance bottleneck shifted forever to the databases, as they couldn't just keep adding more database servers like they could with web servers.

Specific Technical Pain Points

The LiveJournal team identified several critical performance issues:

  1. Friend Page Generation: Loading a user's friends page required multiple database queries to fetch recent posts from all friends, creating expensive JOIN operations
  2. Journal Display: Rendering individual journal entries required database access for posts, comments, user information, and metadata
  3. User Profile Lookups: User authentication, profile information, and preferences required frequent database access
  4. Comment Threading: The complex comment threading system required expensive recursive queries

Resource Constraints and Requirements

The team needed a solution that could: - Reduce Database Load: Dramatically decrease the number of database queries for common operations - Scale Horizontally: Utilize available memory across multiple servers rather than requiring expensive database hardware upgrades - Maintain Data Consistency: Handle cache invalidation appropriately when underlying data changed - Integrate Seamlessly: Work with existing LiveJournal codebase with minimal disruption

Approach: Creating a Distributed Memory Cache

Initial Concept and Design Philosophy

Brad Fitzpatrick developed the concept of a distributed memory cache that could run on any number of machines, with each cache having 512MB to 3GB of memory depending on the machine and its resources. The key insight was to create a simple, fast caching layer that could:

  • Distribute Load: Spread cached data across multiple servers using a consistent hashing algorithm
  • Maximize Memory Utilization: Use spare memory from existing servers rather than requiring dedicated caching hardware
  • Provide Simple APIs: Offer straightforward get/set operations that could be easily integrated into application code

Technical Architecture

Core Design Principles: 1. Simplicity Over Features: Focus on doing one thing (caching) extremely well rather than building a complex system 2. Distributed by Design: Use a client-server architecture where clients know all servers but servers do not communicate with each other 3. Consistent Hashing: Use a hash function to consistently map objects to specific cache servers based on unique object names or journal IDs 4. Memory-Only Storage: Keep all cached data in RAM for maximum performance

Protocol Design: The system used a simple key-value model with keys up to 250 bytes long and values up to 1 megabyte in size, exposing service on port 11211 with support for both TCP and UDP.

Development Iterations

Version 1 - Perl Implementation: Memcached was first developed on May 22, 2003, originally written in Perl. This initial version proved the concept but had performance limitations.

Version 2 - C Rewrite: The system was later rewritten in C by Anatoly Vorobey, who was employed by LiveJournal. This new version was "insanely fast" but initially had memory fragmentation issues when it allocated all available memory.

Memory Allocator Optimization: The team rewrote the memory allocator using modern OS allocator techniques, with help from various team members, to solve fragmentation problems.

Integration Strategy

The LiveJournal team implemented a comprehensive caching strategy:

Cache-Aside Pattern: Applications would first check the cache for requested data, and only query the database if the data wasn't found in cache.

Strategic Cache Key Design: Use descriptive keys like "userrow:userid" to cache user data, making cache management and debugging easier.

Gradual Rollout: The team converted all LiveJournal code to use the memcache whenever possible, prioritizing the most frequently accessed data patterns.

Results: Transformational Performance Improvements

Immediate Performance Gains

When the memory cache was working properly, it was so effective that the database servers were sitting around doing almost nothing. The goal was for most page views to be served from memory.

Specific Performance Improvements: - Database Load Reduction: Database query volume dropped dramatically for cached operations - Response Time Improvement: Page load times improved significantly when data could be served from memory - Scalability Enhancement: The system could handle much higher user loads without database upgrades

Architectural Impact

The introduction of Memcached fundamentally changed LiveJournal's architecture:

Shifted Bottlenecks: Once the memory cache was stable and deployed on many machines, the bottleneck shifted back to web servers, which could be easily added without worrying about maxing out database connections.

Operational Efficiency: The team could scale horizontally using commodity hardware with spare memory rather than investing in expensive database server upgrades.

Development Velocity: Developers could optimize performance by implementing caching strategies rather than complex database query optimization.

Long-term Industry Impact

Open Source Release: The LiveJournal team released Memcached as open source software under the BSD license, enabling widespread adoption.

Industry Adoption: Within a year of its release, Memcached was being used by major sites including Slashdot and Wikipedia. Today, it's used by YouTube, Reddit, Facebook, Pinterest, Twitter, Wikipedia, and Method Studios.

Cloud Platform Integration: Major cloud providers including Google App Engine, Google Cloud Platform, Microsoft Azure, IBM Bluemix, and Amazon Web Services offer Memcached as a managed service through APIs.

Measurable Business Outcomes

Operational Cost Reduction: LiveJournal could handle increased user load without proportional increases in database infrastructure costs.

User Experience Improvement: Faster page load times led to better user engagement and reduced user churn.

Competitive Advantage: The performance improvements helped LiveJournal maintain its position as a leading social platform during the critical growth period of the early 2000s.

Technical Legacy and Evolution

Ongoing Development: Development of Memcached continues on GitHub and is now primarily maintained by Dormando, who has maintained the project since 2007 and runs Cache Forge, a consulting company providing Memcached support.

Modern Enhancements: Today, Memcached supports multiple protocols including the original "text protocol" and the newer "meta protocol", with the project making more than a dozen releases in 2024.

Industry Influence: Memcached's success influenced the development of other caching systems and established distributed caching as a fundamental component of web-scale architecture.

Lessons Learned and Best Practices

Key Success Factors

  1. Focus on Core Problem: The team identified database load as the primary bottleneck and built a focused solution rather than a general-purpose system.

  2. Leverage Existing Resources: Rather than requiring new hardware, Memcached utilized spare memory across existing servers.

  3. Simple, Robust Design: The emphasis on simplicity made the system reliable and easy to debug.

  4. Open Source Strategy: Releasing Memcached as open source accelerated adoption and created a sustainable development community.

Technical Principles

  • Consistent Hashing: Distributing cache keys consistently across servers enabled seamless scaling.
  • Cache-Aside Pattern: Letting applications control cache logic provided flexibility while maintaining simplicity.
  • Memory-Only Storage: Accepting cache volatility in exchange for maximum performance proved to be the right trade-off.

Broader Implications

The Memcached case study demonstrates several important principles for scaling web applications:

  • Horizontal scaling strategies can be more cost-effective than vertical scaling for many workloads
  • Simple, focused solutions often outperform complex, feature-rich alternatives
  • Caching layers can dramatically improve application performance when implemented thoughtfully
  • Open source release can transform internal solutions into industry-standard technologies

The creation of Memcached at LiveJournal represents a pivotal moment in web architecture evolution, showing how practical engineering solutions to immediate scaling problems can become foundational technologies that enable the modern internet.

References

  1. Memcached - a distributed memory object caching system - 2003-2025 - Memcached Official Documentation - Official project page detailing Memcached's origins at LiveJournal and its current development status under Dormando's maintenance.
  2. LiveJournal's Backend and memcached: Past, Present, and Future - November 2004 - USENIX LISA Conference - Technical presentation by Brad Fitzpatrick and Lisa Phillips detailing LiveJournal's scaling challenges and Memcached's role in solving database performance issues.
  3. The evolution of Memcached - February 2025 - LWN.net - Comprehensive analysis of Memcached's 20+ year evolution from LiveJournal's internal tool to modern distributed caching system, including recent developments and protocol enhancements.
  4. Memcached - Updated January 2025 - Wikipedia - Detailed technical overview of Memcached's architecture, development history, and current usage across major platforms including YouTube, Facebook, and Twitter.
  5. LiveJournal - Updated June 2025 - Wikipedia - Comprehensive history of LiveJournal's growth from 1999 hobby project to major social platform, including scaling challenges that necessitated innovative solutions.
  6. A casual stroll through LiveJournal's backend - 2003 - LiveJournal Maintenance Blog - First-hand account from LiveJournal's engineering team describing database performance problems and the implementation of their distributed memory cache solution.
  7. LiveJournal Architecture - 2018 - High Scalability - Technical analysis of LiveJournal's architectural evolution and scaling strategies, highlighting the role of Memcached in solving database bottlenecks.
  8. Brad Fitzpatrick - Updated 2025 - Wikipedia - Biography of Memcached's creator, covering his work at LiveJournal, subsequent career at Google, and contributions to open source technologies.
  9. AMA with Brad Fitzpatrick, LiveJournal, Memcached & Camlistore Founder - 2020 - CodeFor.Cash - Interview with Brad Fitzpatrick discussing the early days of LiveJournal, the development challenges that led to Memcached, and insights into scaling web applications.
  10. Brad Fitzpatrick Personal Website - Updated 2025 - Personal Website - Brad Fitzpatrick's official website documenting his work history including the founding of Danga Interactive, development of LiveJournal and Memcached, and subsequent projects at Google and Tailscale.