[Python-Dev] Multigigabyte memory usage in the OpenIndiana Buildbot

[Antoine Pitrou]

On Wed, 07 Sep 2011 13:38:23 +0200
Jesus Cea <jcea at jcea.es> wrote:
> 
> So, the problem is that a) "make test" takes quite a bit of RAM and b)
> the buildbot forks some "big" processes, so the virtual memory needed
> is BIG.

Note that buildbots run "make buildbottest", not "make test".

> So I have two questions:
> 
> 1. Can we reduce the memory footprint of the tests?. I can't
> understand why the python test process is taking so much memory.

Because the test suite will by construction load all the stdlib (minus
the few modules which don't have a test suite), and creates numerous
test scenarios. Depending on the memory allocator, fragmentation can
make it difficult to reclaim memory that has been formally freed after
a test is run.

If "-j" is used, tests get run in a separate process each, so that
approach might be an answer.

> 2. Why buildbot is "forking()" big processes?. Can we do something to
> change this?.

Because we need to test for various functionalities, such as os.fork()
and os.exec*(), but also the command-line behaviour of the interpreter,
the distutils module, the packaging module, the subprocess module, the
multiprocessing module... (this list is not exhaustive).

Regards

Antoine.