On Sun, Jul 21, 2013 at 08:25:46PM -0700, Alex Gaynor wrote:

No, there currently isn't a way to parallelize building.

Ok. Is it hard or just low priority?

Alex

PS: Attentive readers will note that technically the very last phase of compilation is parallelized.

Yes, and I guess if it weren't it would actually take a significant chunk of the compile time :) njh

On Mon, Jul 22, 2013 at 09:06:51AM +0200, Maciej Fijalkowski wrote:

On Mon, Jul 22, 2013 at 8:18 AM, Nathan Hurst wrote:

...

On Sun, Jul 21, 2013 at 08:25:46PM -0700, Alex Gaynor wrote:

...

No, there currently isn't a way to parallelize building.

Ok. Is it hard or just low priority?

hard. there is some work ongoing, but we need a working STM first ;-)

Thanks for the explanation Alex+Maciej - I'll just get used to waiting. Ok. I'm just learning about python threading (after reading about STMs - I've always used multiprocessing for compute heavy stuff and greenlets for servery stuff) and I tried this simple program: import threading class MyThread(threading.Thread): def run(self): while True: pass thread = MyThread() thread.start() thread.join() now clearly this program is never going to terminate without some kind of external interrupt. But the strange thing is, it doesn't terminate when I use control-C on it either (it does terminate correctly from eclipse.pydev with the stop button): njh $ pypy threadtoy.py ^C ^Z [1]+ Stopped pypy threadtoy.py njh $ kill % [1]+ Stopped pypy threadtoy.py njh $ [1]+ Terminated pypy threadtoy.py why does python and pypy do this? Is it part of the GIL problem? will STM fix it? regards, njh

On 22 July 2013 16:18, Nathan Hurst wrote:

On Sun, Jul 21, 2013 at 08:25:46PM -0700, Alex Gaynor wrote:

...

No, there currently isn't a way to parallelize building.

Ok. Is it hard or just low priority?

It's mostly pretty hard, although it's not the same reason through each of the stages involved - the short answer is that translation mostly involves walking over graphs, including the inter-procedural call graph, and propagating information. In one stage the graph is known mostly ahead of time, but otherwise it's being built as it goes, which means the job list probably stays pretty small. If we managed to make that stage parallel, we'd probably lose out on the fact that several of the steps work by mutating the model, so we'd either need a concurrency model that dealt with that or separate compilation to work. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement.