about forking in taskengine

hello Samuele, I have been reading your taskengine code with great interest. I was wondering what your plans with regards to forking are. Would it be possible to extend the option so we can specify after which task to fork, instead of having some fixed points as we had before? What would be of great help (to me at least) is if multiple goals could be specified (lets say: generate pypy-c and pypy-llvm). The program could then be smart enough to fork just before the tasklists differ. With the current translate_pypy there is no way to do this. An additional nicety would be if we could have an option where we specify how many of these task to run in parallel (like 'make -j X') . I do realize some things at the moment will not run nicely in parallel (compiling the same file with different compiler options for instance), but that is another problem. cheers Eric

Eric van Riet Paap wrote:
hello Samuele,
I have been reading your taskengine code with great interest.
I was wondering what your plans with regards to forking are. Would it be possible to extend the option so we can specify after which task to fork, instead of having some fixed points as we had before?
yes, this is doable, although given how we use fork it makes more sense to have a fork-before
What would be of great help (to me at least) is if multiple goals could be specified (lets say: generate pypy-c and pypy-llvm). The program could then be smart enough to fork just before the tasklists differ. With the current translate_pypy there is no way to do this.
a TranslationDriver is really meant to deal with one translation go, so this could be doable but I don't think is worth the complexity. OTOH factoring code in translate_pypy.py to have more reusable parts and creating a new script or adding a special option, you could implement the following logic keeping TranslationDriver relatively straightforward: compile-for-all-backends: drv = ... drv.proceed('rtype') for backend fork -> drv.proceed('compile_'+backend)
An additional nicety would be if we could have an option where we specify how many of these task to run in parallel (like 'make -j X') . I do realize some things at the moment will not run nicely in parallel (compiling the same file with different compiler options for instance), but that is another problem.
cheers Eric
participants (2)
-
Eric van Riet Paap
-
Samuele Pedroni