[Chicago] What is cloud computing good for?

Tal Liron tal.liron at threecrickets.com
Mon Aug 9 21:49:32 CEST 2010

  Hey guys,

We at crowdSPRING are moving our entire Django-based setup to the 
Amazonian cloud. We make good use of EC2, S3, EBS, and RDS (a relatively 
new service). Not only is it vastly cheaper than dedicated services, but 
it allows us astounding scalability.

It's not trivial. We've created a large set of Bash scripts to maintain 
and manage our cloud. For example, a single command can be used to spin 
up a new application node -- but a lot needs to be done: create a node 
and install all packages, have it join our redundant network RAID, join 
our monitoring tools, re-generate configuration files (we use PyTenjin 
for templating here) for our load balancer and front-end cache, restart 
them, etc. The bottom line is that within 2 minutes we have a new node 
behind the load balancer answering user requests. Within a few minutes 
we can double our capacity.

S3 presents it's own challenges -- we've instituted direct uploads to S3 
to bypass Django's standard storage API. Clients thus do not need to 
interact with our app servers in order to upload large files.

If there's interest, we can give a presentation on this, though note 
that there's not a lot that's Python-specific about it.

By the way, I'm surprised the discussion has not mentioned Google App 
Engine at all! I would definitely count that as "cloud," and it's 
absolutely the easiest path to it.


On 08/09/2010 02:26 PM, Garrett Smith wrote:
> On Mon, Aug 9, 2010 at 1:58 PM,<skip at pobox.com>  wrote:
>>     Massimo>  If you are not interested in web apps (and looks like you are
>>     Massimo>  not) you may be interested in this:
>>     Massimo>  http://www.picloud.com/
>> You are correct.  I care not one whit about web apps.  I'm interested solely
>> in single-threaded compute-intensive tasks.  Currently, we run the Torque
>> batch queuing system on Solaris.  It works fine, though I am constantly
>> having to scramble to find hardware.  I might have 10 cores available during
>> the day, 50 at night.  That works well until I'm asked to run 4000 jobs one
>> day.  Each job can run from a few minutes to several hours.
>> PiCloud looks interesting except it doesn't seem that your Python functions
>> can use C or C++ code which is not part of Python proper.  While I write
>> almost all my software in Python I rely heavily on internal libraries
>> written in C++ by other groups here at work.
> This is pretty recent from Amazon...
> http://aws.amazon.com/elasticmapreduce/
> I'm not actually the Amazon fanboy that I appear to be. But I
> impressed with how they've built AWS out over the last few years.
> For my own understanding, I parse "cloud" as "no physical
> infrastructure". This actually is a big deal, IMO -- not because it's
> new (the mainframe business was largely one of leasing compute cycles
> and data storage) -- but because fixed costs of using these resources
> are getting very close to $0.
> Jason's absolutely right -- at a certain point it's more cost
> effective to invest in your own infrastructure. Cloud offerings are
> generally a little bit cheaper than dedicated hardware but are also *a
> lot* less performant. But I've found that companies are willing to put
> up with this cost equation for the convenience of changing (or walking
> away from) their operations at any time.
> I generally recommend:
> - Develop and test in a virtualized (i.e. cloud) environment
> - When the economics over a multi-year period make sense, convert to a
> "real" infrastructure - this is simply a price/performance decision
> The cloud providers that can provide a combination of virtual and
> real/dedicated hardware -- that is, can simplify the migration from
> dev/test/early-production to long term production -- will have a super
> compelling value proposition.
> Garrett
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago

More information about the Chicago mailing list