From bradallen137 at gmail.com  Wed Feb  9 20:44:52 2011
From: bradallen137 at gmail.com (Brad Allen)
Date: Wed, 9 Feb 2011 13:44:52 -0600
Subject: [python-advocacy] addressing scalability concerns
Message-ID: <AANLkTi=D6Oh7HjtF8L5pu44Bqyp2haGdOvQnzL+w8aEn@mail.gmail.com>

Hello,

Today at work, one of our prospective customers raised concerns about
Python's scalability. We're sending them our stress test reports to
prove the horizontal scalability of our application, but we also
wanted to craft a good response.

I've posted it here, removing our product-specific details, in hopes
of getting some feedback. It would be nice to have a statement
somewhere on the Python website to address these kinds of concerns.

> 3.Python - They had some concerns about python's ability to scale and some "memory leaks". Please highlight the robustness of python and all the good stuff

Python has proven scalability at companies such as Google, YouTube,
NASA, Walt Disney Pictures, and Industrial Light & Magic (see
http://www.python.org/about/success/)

Nothing about the Python language encourages writing applications with
memory leaks. In fact, Python makes it easier to avoid memory leaks
than many programming languages, due to the clarity and sparseness of
the code in comparison to lower level languages where complexity
results in denser code. Python excels at making complexity manageable.
In addition, it has a built-in garbage collector which means
developers don't need to worry about allocating/deallocating memory.

The speed difference between Python and low level languages (such as
Java or C++) is often misunderstood. Python is written in C, and many
complex operations in Python actually occur at "C" speed. Bottlenecks
are more common in I/O than the CPU, and in cases where CPU matters,
we have good options available such as writing C extensions, or
scaling horizontally with multiple processes.

From d-beazley at sbcglobal.net  Wed Feb  9 21:02:57 2011
From: d-beazley at sbcglobal.net (David Beazley)
Date: Wed, 9 Feb 2011 14:02:57 -0600
Subject: [python-advocacy] addressing scalability concerns
In-Reply-To: <AANLkTi=D6Oh7HjtF8L5pu44Bqyp2haGdOvQnzL+w8aEn@mail.gmail.com>
References: <AANLkTi=D6Oh7HjtF8L5pu44Bqyp2haGdOvQnzL+w8aEn@mail.gmail.com>
Message-ID: <7721A7F8-2635-4FE7-9DF7-178F91F28AB2@sbcglobal.net>


On Feb 9, 2011, at 1:44 PM, Brad Allen wrote:

> Hello,
> 
> Today at work, one of our prospective customers raised concerns about
> Python's scalability. We're sending them our stress test reports to
> prove the horizontal scalability of our application, but we also
> wanted to craft a good response.
> 

How about a simple "No Python for you. Get out!"   ;-).

-Dave



From taleinat at gmail.com  Wed Feb  9 23:12:50 2011
From: taleinat at gmail.com (Tal Einat)
Date: Thu, 10 Feb 2011 00:12:50 +0200
Subject: [python-advocacy] addressing scalability concerns
In-Reply-To: <AANLkTi=D6Oh7HjtF8L5pu44Bqyp2haGdOvQnzL+w8aEn@mail.gmail.com>
References: <AANLkTi=D6Oh7HjtF8L5pu44Bqyp2haGdOvQnzL+w8aEn@mail.gmail.com>
Message-ID: <AANLkTi=cSx1dFXsyBPaFpwi+Gk01kaXBNyd5H7PA+_om@mail.gmail.com>

On Wed, Feb 9, 2011 at 9:44 PM, Brad Allen <bradallen137 at gmail.com> wrote:

> Nothing about the Python language encourages writing applications with
> memory leaks. In fact, Python makes it easier to avoid memory leaks
> than many programming languages, due to the clarity and sparseness of
> the code in comparison to lower level languages where complexity
> results in denser code. Python excels at making complexity manageable.
> In addition, it has a built-in garbage collector which means
> developers don't need to worry about allocating/deallocating memory.
>

You raise good points, but I'm not sure you're addressing their concerns.

By "memory leaks" they may be referring to the difficulty in predicting
memory usage for large & complex Python applications. In my workplace we
have built such a system and ran into memory consumption problems when
processing large amounts of data.

For instance, Python objects of certain types (such as ints, floats and
strings) use a memory allocation strategy where the memory is *never*
deallocated (it is re-used if/when such objects are created later). This
means that if at any time my program had many ints and/or floats and/or
strings live at once, the memory used to hold those objects will never be
freed until the process ends. There are various ways of working around this,
such as running calculations in separate processes or using array/ndarray
objects to store large numbers of similar objects, so this shouldn't be a
deal-breaker.

And of course there are the limitations of the Python garbage collector with
regard to dealing with cyclic references including objects with __del__
methods.

The way I see it, the ability to build a working large & complex system
using Python in a very short time with just a few programmers leaves more
than enough time and resources to deal with such issues if and when they
arise.


> The speed difference between Python and low level languages (such as
> Java or C++) is often misunderstood. Python is written in C, and many
> complex operations in Python actually occur at "C" speed. Bottlenecks
> are more common in I/O than the CPU, and in cases where CPU matters,
> we have good options available such as writing C extensions, or
> scaling horizontally with multiple processes.
>

>From my experience, using Python helps you to optimize only when and where
it is actually needed. You get a working version of the system up and
running quickly, therefore you can recognize the various bottlenecks and
deal with them sooner. I've worked on systems that do processing which takes
several weeks (on multiple processors on multiple computers) and the fact
that we were using Python never posed a serious handicap with regard to
optimizations.

- Tal Einat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/advocacy/attachments/20110210/3a33b322/attachment.html>