[Distutils] parallelize setup.py develop/build

"Martin v. Löwis" martin at v.loewis.de
Sun Dec 18 20:47:01 CET 2011


> What I don't really like in fact is to be forced to call an external
> subprocess call just to run some other python code..

Why do you want to run this in parallel in the first place? If it's
for speed, and to use multiple processors efficiently, then you
absolutely need subprocesses.

> Going back to the original question, parallelizing many build/develop
> calls can be actually faster or is the same in case the hard disk is still only one?

I think your approach to parallelization would also suffer from the
global interpreter lock, in addition to probably experiencing an
IO bound. It may be that you can increase IO utilization with a
threaded approach also, but I really suggest that using subprocesses
is much better.

Regards,
Martin


More information about the Distutils-SIG mailing list