LiveJournal's Secret Spinoffs

Artur Bergman, Engineering Manager, Six Apart
Brad Whitaker, Senior Software Engineer, Six Apart

Track: Web Apps
Date: Friday, July 28
Time: 11:35am - 12:20pm
Location: F150

Serving millions of page views a day, tens of thousands of concurrent users, scaling this large open source project spawned of three well-known and commonly used technologies for scaling out.

Learn how to deploy the trinity of Perlbal, memcached, and Mogilefs to solve your scaling problems.

Perlbal is a highly intelligent, super fast loadbalancer written in Perl with an extensive plugin support. It allows you to reduce the load on your application servers and free up valuable resources quicker, and includes features like queue support, which can prioritize your most important traffic.

memcache is a distributed hashing memory cache used by several large sites including Slashdot and Wikipedia. Its most notable feature is that all operations in the cache are O(1) and it is blazing fast. By taking your most common SQL queries and putting them into memcache, you can dramatically decrease the load on your databases.

With a realisation that normal filesystems are not designed to handle billions of small files in a reliable way came MogileFS. Used by several large users to keep track of millions of files, Mogile treats a disk as a throwaway, unreliable piece of hardware it actually is and achieves its speed and reliability by clustering across a large amount of machines.

The consistent theory is to use a large amount of cheap hardware to achieve cost-effective scalability in a pragmatic, cross-platform and cross-language way.