[Baypiggies] mrjob - distributed computing in Python made easy
Jimmy Retzlaff
jimmy at retzlaff.com
Sat Oct 30 00:11:31 CEST 2010
Yelp has just open sourced mrjob. It's a package that makes doing
MapReduce in Python almost trivial. In as little as 10 lines of code
you can do MapReduce on your own computer. Add a handfull of lines of
configuration and you can be running your code unchanged in parallel
on dozens of machine on Amazon Elastic Map Reduce for a a few dollars
(a dozen "small" machines for an hour would be about $1). Dave, the
primary author just wrote up a post on our engineering blog:
http://engineeringblog.yelp.com/2010/10/mrjob-distributed-computing-for-everybody.html
I'm putting in a proposal to give a talk on mrjob at PyCon 2011.
Whether that's accepted or not, maybe I can give the same talk at
BayPIGgies in January or February if there's interest...
Jimmy
More information about the Baypiggies
mailing list