[Chicago] What is cloud computing good for?

Pete pfein at pobox.com
Tue Aug 10 00:14:00 CEST 2010

On Aug 9, 2010, at 1:58 PM, skip at pobox.com wrote:
> You are correct.  I care not one whit about web apps.  I'm interested solely
> in single-threaded compute-intensive tasks.  Currently, we run the Torque
> batch queuing system on Solaris.  It works fine, though I am constantly
> having to scramble to find hardware.  I might have 10 cores available during
> the day, 50 at night.  That works well until I'm asked to run 4000 jobs one
> day.  Each job can run from a few minutes to several hours.

EC2 (or anything on a similar model) would be totally appropriate here - this is the exact type of problem it's designed for. As for bindings, there's also http://cloudsilverlining.org/ (not solid yet).

Skip, if your disk image is the same for all of these jobs, Amazon's excellent command line tools will suffice.  You just push your input data to S3, boot NN nodes w/ the image (multicore boxen are available, fwiw), run your jobs & push the output back to S3.  You can use amazon's Simple Queue Service (SQS) to coordinate task distribution. A SMOP, I'm sure. ;-)


More information about the Chicago mailing list