[pypy-dev] Help with memory usage

[Matheus Salvia]

Thanks for the answer, Armin.
The application is corrently using about 15 processes, but this is a
configurable number.
By your information I can assume that the process overhead would be of
about 600MB,
which is not a negligible number but still doesn't fully explain all of the
memory usage.
The application is an information processor, and simplifying the algorithm
it would be something like:
- start a worker process pool
- receive messages from a queue
- delegate messages to the workers

The workers themselves have a lifecycle about like that:
- get message
- fetch data pointed by the message from database (apache cassandra)
- process data:
- - call a C library to do some preprocessing (Zorba XQuery thru CFFI)
- - cal a python module to do intermediate and post processing
- store resulting data to database (ElasticSearch)

The libraries I'm using are: pycassa, pyelasticsearch, cffi, boto and peewee
My operating system is Ubuntu 12.04 LTS x64

Last but not least, is there a way to change pypy from using a GC to using
refcount like CPython?

That was the information you needed?
Feel free to ask anything.

Again, thank you,
Matheus Salvia


2014-02-03 Armin Rigo <arigo at tunes.org>:

> Hi Matheus,
>
> On 3 February 2014 03:55, Matheus Salvia <matheus2740 at gmail.com> wrote:
> > When running my app under CPython, it uses about 1GB of memory, but when
> > running with pypy it goes up to almost 3GB.
>
> This is a question that doesn't have a single answer.  You need to
> give us a lot more information about what your application does,
> before we can start proposing ideas about why there is such a
> difference in your case.  In our own experience, the memory usage is
> usually very roughly around the same as CPython (a factor less than 2
> in one way or another).
>
> As a first (likely completely random) guess, you are starting a very
> large number of processes, each of which consumes 60MB instead of 20MB
> in CPython.  If I'm correct, then there is not much we can do; it's
> known that PyPy's baseline memory usage is higher than CPython's, due
> to the JIT which ends up storing a few dozens of MBs in any process.
>
>
> A bientôt,
>
> Armin.
>



-- 
--
 // Matheus Salvia
Desenvolvedor Mobile
Celular: +55 11 9-6446-2332
Skype: meta.faraday
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140203/d9f5aa16/attachment.html>