[pypy-dev] Re: about forking in taskengine

[Samuele Pedroni]

Eric van Riet Paap wrote:

>hello Samuele,
>
>I have been reading your taskengine code with great interest.
>
>I was wondering what your plans with regards to forking are.
>Would it be possible to extend the option so we can specify after which task 
>to fork, instead of having some fixed points as we had before?
>
>  
>
yes, this is doable, although given how we use fork it makes more sense 
to have a fork-before

>What would be of great help (to me at least) is if multiple goals could be 
>specified (lets say: generate pypy-c and pypy-llvm). The program could then 
>be smart enough to fork just before the tasklists differ. With the current 
>translate_pypy there is no way to do this.
>
>  
>
a TranslationDriver is really meant to deal with one translation go, so 
this could be doable but
I don't think is worth the complexity. OTOH factoring code in 
translate_pypy.py to have more reusable
parts and creating a new script or adding a special option, you could 
implement the following logic
keeping TranslationDriver relatively straightforward:

compile-for-all-backends:
     drv = ...
     drv.proceed('rtype')
     for backend
          fork ->
             drv.proceed('compile_'+backend)

>An additional nicety would be if we could have an option where we specify how 
>many of these task to run in parallel (like 'make -j X') . I do realize some 
>things at the moment will not run nicely in parallel (compiling the same file 
>with different compiler options for instance), but that is another problem.
>
>cheers
>Eric
>  
>