[python-advocacy] addressing scalability concerns

[Tal Einat]

On Wed, Feb 9, 2011 at 9:44 PM, Brad Allen <bradallen137 at gmail.com> wrote:

> Nothing about the Python language encourages writing applications with
> memory leaks. In fact, Python makes it easier to avoid memory leaks
> than many programming languages, due to the clarity and sparseness of
> the code in comparison to lower level languages where complexity
> results in denser code. Python excels at making complexity manageable.
> In addition, it has a built-in garbage collector which means
> developers don't need to worry about allocating/deallocating memory.
>

You raise good points, but I'm not sure you're addressing their concerns.

By "memory leaks" they may be referring to the difficulty in predicting
memory usage for large & complex Python applications. In my workplace we
have built such a system and ran into memory consumption problems when
processing large amounts of data.

For instance, Python objects of certain types (such as ints, floats and
strings) use a memory allocation strategy where the memory is *never*
deallocated (it is re-used if/when such objects are created later). This
means that if at any time my program had many ints and/or floats and/or
strings live at once, the memory used to hold those objects will never be
freed until the process ends. There are various ways of working around this,
such as running calculations in separate processes or using array/ndarray
objects to store large numbers of similar objects, so this shouldn't be a
deal-breaker.

And of course there are the limitations of the Python garbage collector with
regard to dealing with cyclic references including objects with __del__
methods.

The way I see it, the ability to build a working large & complex system
using Python in a very short time with just a few programmers leaves more
than enough time and resources to deal with such issues if and when they
arise.


> The speed difference between Python and low level languages (such as
> Java or C++) is often misunderstood. Python is written in C, and many
> complex operations in Python actually occur at "C" speed. Bottlenecks
> are more common in I/O than the CPU, and in cases where CPU matters,
> we have good options available such as writing C extensions, or
> scaling horizontally with multiple processes.
>

>From my experience, using Python helps you to optimize only when and where
it is actually needed. You get a working version of the system up and
running quickly, therefore you can recognize the various bottlenecks and
deal with them sooner. I've worked on systems that do processing which takes
several weeks (on multiple processors on multiple computers) and the fact
that we were using Python never posed a serious handicap with regard to
optimizations.

- Tal Einat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/advocacy/attachments/20110210/3a33b322/attachment.html>