O'Reilly Open Source Convention - August 1-5, 2005 - Portland, Oregon
 Convention Coverage

Session

Scalable Computing with MapReduce
Doug Cutting, Yahoo!

Track: Java
Date: Wednesday, August 3rd, 2005
Time: 4:30pm - 5:15pm
Location: D135

Over the past few years, Google has published details of their infrastructure. Developers within Google are able to easily write algorithms that efficiently and reliably process many terabytes of data. To do this, they leverage two technologies: the Google File System (GFS), which implements reliable distributed storage; and MapReduce, a reliable distributed-processing layer built on GFS. Together, this platform facilitates tasks as diverse as log analysis and database construction. One can efficiently "grep" weeks of logs from a high-volume site, constructing concise summaries. One can build efficiently searchable indexes of huge datasets. Such tasks are easily implemented with little code. Scalability and reliability are handled by the system, so that developers can focus on algorithms.

The Nutch project has now implemented a similar platform in open source, so that folks outside of Google can enjoy these benefits. This talk outlines the platform's architecture and implementation, as well as shows how it may be used to solve real problems.



Diamond Sponsors

Computer Associates International Inc., (CA)
Hewlett Packard
SpikeSource
Sun Microsystems

Platinum Sponsors

Novell, Inc.

Gold Sponsors

ActiveState
IBM
Ticketmaster

Silver Sponsors

ActiveGrid
Alfresco
Black Duck Software
CollabNet
Covalent Technologies
Google
GroundWork Open Source Solutions
Intel Corporation
Mergere, Inc.
Microsoft
Oracle
Palamida
SourceLabs
SugarCRM
Yahoo! Inc.
Zend Technologies, Inc.

Media Sponsors

boing boing
C/C++ Users Journal
DevtownStation News
Digital ID World
Enterprise Open Source Journal
Free Software Magazine
InsideMac Radio
Integration Developer News
Linux Journal
LinuxQuestions.org
Open Enterprise Trends
Queue
SDForum
Software Association of Oregon
Version Tracker
Wi-Fi Technology Forum
Women's Technology Cluster
WorldWIT

In-Kind Sponsors

Dell Inc.
Gibson
Griffin Technology
Harman Multimedia
Smugmug

Sponsors

OSCON 2005 Sponsor Opportunities — Email us at

Download the OSCON 05 Sponsor/Exhibitor Prospectus

OSCON 2005 Media Sponsor Opportunities — Call Margi Levin at 707-827-7184 or email at

Press and Media

For media-related inquiries, contact Suzanne Axtell at

Conference News

Want to receive conference news? Sign up for our email newsletter.

O'Reilly Home | Privacy Policy

© 2005, O'Reilly Media, Inc.