[Chicago] hadoop and python?

[John Stoner]

Hadoop is built in Java but it seems to have shell-based interfaces probably
useful across a variety of platforms. From the Hadoop Streaming page:

http://wiki.apache.org/hadoop/HadoopStreaming

Using the streaming system you can develop working hadoop jobs with *
extremely* limited knowldge of Java. At it's simplest your development task
is to write two shell scripts that work well together, let's call them *
shellMapper.sh* and *shellReducer.sh*. On a machine that doesn't even have
hadoop installed you can get first drafts of these working by writing them
to work in this way:

cat someInputFile | shellMapper.sh | shellReducer.sh > someOutputFile

With streaming, Hadoop basically becomes a system for making pipes from
shell-scripting work (with some fudging) on a cluster. There's a strong
logical correspondence between the unix shell scripting environment and
hadoop streaming jobs....



On Mon, Jun 28, 2010 at 11:52 AM, Christopher Allan Webber <
cwebber at dustycloud.org> wrote:

> Brian Ray <brianhray at gmail.com> writes:
>
> > It seems like "Cloud" computing needs some disambiguation.
>
> Understatement of the year :)
>
> Cloud computing is the computer industry's current buzzword baby.  Like
> all buzzwords it really doesn't mean anything because everyone's
> throwing their particular favorite issue/service into that bucket, to
> the point where it's overflowing with only tangentally related terms.
> Cloud computing more so than most buzzword buckets, even.
>
> I really think it's a good idea if people refer to the specific
> technology concepts they're advocating rather than ambiguous catch-alls.
> Non-relational databases, mapreduce, on-the-fly virtualization, software
> as a service... these are all very different ideas, and we're all much
> better off if people actually say what they mean.
>
>  - cwebb
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>



-- 
blogs:
http://www.generosity.org/stoner/
http://boogiepants.typepad.com/
'In knowledge is power; in  wisdom, humility.'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/chicago/attachments/20100628/f3f2c185/attachment.html>