[pypy-dev] translate.py failes

[Armin Rigo]

Hi,

On Thu, Aug 28, 2008 at 10:34:51PM +0600, Vetoshkin Nikita wrote:
> Is there a direction, where I could digg to help? Something to begin with.
> I don't have enough experience but I'd like to take part (even very 
> small) in this project.

Sure.  Do you understand the issue?  It's related to the way a Unix
process starts a new subprocess: os.fork() followed by os.execv() in the
child process.  The issue is that os.fork() fails if the parent process'
total memory is more than half of your RAM+swap, because it tries to
create a copy of the process, and the copy would also need more than
half of your RAM+swap.  (In reality, os.fork() doesn't copy the memory
at all, but creates "shared pages" of memory so that memory pages are
duplicated only when one of the two processes really modifies it; but
still os.fork() has to mark all the memory as *potentially* used, and
raises MemoryError if there isn't enough.)

A solution might be to split translate.py in two processes: translate.py
would start a subprocess when it starts, and then do all the work
(consuming a lot of RAM); but when it needs to start new processes, e.g.
call gcc, it would instead ask the subprocess to start the new
processes.  More precisely, the code in pypy.translator.tool.cbuild
would stop using distutils directly, and instead ask the subprocess to
use distutils.  It looks even easy to do with the help of a
py.execnet.PopenGateway() created when translate.py starts (see the py
lib documentation at http://codespeak.net/py/).


A bientot,

Armin