parallel building
Dear mailing list, I have looked around the web and through the various makefiles and pypy/goal/targetpypystandalone.py and I can't work out how to make pypy build using more cores. I have plenty of RAM (64GB) but the build is only using a single core. Is it possible? njh
No, there currently isn't a way to parallelize building. Alex PS: Attentive readers will note that technically the very last phase of compilation is parallelized. On Sun, Jul 21, 2013 at 8:08 PM, Nathan Hurst <njh@njhurst.com> wrote:
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
On Mon, Jul 22, 2013 at 09:06:51AM +0200, Maciej Fijalkowski wrote:
Thanks for the explanation Alex+Maciej - I'll just get used to waiting. Ok. I'm just learning about python threading (after reading about STMs - I've always used multiprocessing for compute heavy stuff and greenlets for servery stuff) and I tried this simple program: import threading class MyThread(threading.Thread): def run(self): while True: pass thread = MyThread() thread.start() thread.join() now clearly this program is never going to terminate without some kind of external interrupt. But the strange thing is, it doesn't terminate when I use control-C on it either (it does terminate correctly from eclipse.pydev with the stop button): njh $ pypy threadtoy.py ^C ^Z [1]+ Stopped pypy threadtoy.py njh $ kill % [1]+ Stopped pypy threadtoy.py njh $ [1]+ Terminated pypy threadtoy.py why does python and pypy do this? Is it part of the GIL problem? will STM fix it? regards, njh
Hi Nathan, On Mon, Jul 29, 2013 at 1:28 AM, Nathan Hurst <njh@njhurst.com> wrote:
why does python and pypy do this? Is it part of the GIL problem? will STM fix it?
PyPy does this because Python does it. It is not part of any other problem, just that thread.join() uses lock.acquire(), which cannot be interrupted in Python 2.x. This has been changed in some version of Python 3.x. A bientôt, Armin.
On 22 July 2013 16:18, Nathan Hurst <njh@njhurst.com> wrote:
It's mostly pretty hard, although it's not the same reason through each of the stages involved - the short answer is that translation mostly involves walking over graphs, including the inter-procedural call graph, and propagating information. In one stage the graph is known mostly ahead of time, but otherwise it's being built as it goes, which means the job list probably stays pretty small. If we managed to make that stage parallel, we'd probably lose out on the fact that several of the steps work by mutating the model, so we'd either need a concurrency model that dealt with that or separate compilation to work. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement.
No, there currently isn't a way to parallelize building. Alex PS: Attentive readers will note that technically the very last phase of compilation is parallelized. On Sun, Jul 21, 2013 at 8:08 PM, Nathan Hurst <njh@njhurst.com> wrote:
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
On Mon, Jul 22, 2013 at 09:06:51AM +0200, Maciej Fijalkowski wrote:
Thanks for the explanation Alex+Maciej - I'll just get used to waiting. Ok. I'm just learning about python threading (after reading about STMs - I've always used multiprocessing for compute heavy stuff and greenlets for servery stuff) and I tried this simple program: import threading class MyThread(threading.Thread): def run(self): while True: pass thread = MyThread() thread.start() thread.join() now clearly this program is never going to terminate without some kind of external interrupt. But the strange thing is, it doesn't terminate when I use control-C on it either (it does terminate correctly from eclipse.pydev with the stop button): njh $ pypy threadtoy.py ^C ^Z [1]+ Stopped pypy threadtoy.py njh $ kill % [1]+ Stopped pypy threadtoy.py njh $ [1]+ Terminated pypy threadtoy.py why does python and pypy do this? Is it part of the GIL problem? will STM fix it? regards, njh
Hi Nathan, On Mon, Jul 29, 2013 at 1:28 AM, Nathan Hurst <njh@njhurst.com> wrote:
why does python and pypy do this? Is it part of the GIL problem? will STM fix it?
PyPy does this because Python does it. It is not part of any other problem, just that thread.join() uses lock.acquire(), which cannot be interrupted in Python 2.x. This has been changed in some version of Python 3.x. A bientôt, Armin.
On 22 July 2013 16:18, Nathan Hurst <njh@njhurst.com> wrote:
It's mostly pretty hard, although it's not the same reason through each of the stages involved - the short answer is that translation mostly involves walking over graphs, including the inter-procedural call graph, and propagating information. In one stage the graph is known mostly ahead of time, but otherwise it's being built as it goes, which means the job list probably stays pretty small. If we managed to make that stage parallel, we'd probably lose out on the fact that several of the steps work by mutating the model, so we'd either need a concurrency model that dealt with that or separate compilation to work. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement.
participants (5)
-
Alex Gaynor -
Armin Rigo -
Maciej Fijalkowski -
Nathan Hurst -
William ML Leslie