What to do about this pending crisis is a complicated issue.
Oh, please. Calling this a crisis only causes people to ignore you.
I apologize if this statement is a little exaggerated. But, I do think this is a really critical problem that is going to affect certain groups of Python users and developers in adverse ways. Perhaps I have not made a very strong case that it is a true "crisis" though.
Take a look at this URL that Guido posted to the baypiggies list:
I actually agree with many of the comments made by the author. The quotes from John Dvorak and Wired are over the top. The author make good points about operating systems' abilities to handle threading on modest numbers of cores. It does work, even today, and will continue to work (for a while) as the number of cores increases. But the main point of the author is that operating systems DO have threading support that works reasonably well. Python is another story. The author even makes comments that provide a string critique of Python's (and Ruby's) threading capabilities: <quote> Modern programs tend to be moderately multithreaded, with individual threads dedicated to the GUI, to user I/O, to socket I/O, and often to computation. Multicore CPUs take advantage of this quite well. And we don't need any new technology to make sure multi-threaded programs are well-behaved - these techniques are pretty well understood, and in use in most software you use today. Modern languages like Java support threads and various concurrency issues right out of the box. C++ requires non-standard libraries, but all modern C++ environments worth their salt deal with multithreading in a fairly sane way. </quote> Modern programs tend to be moderately multithreaded.....except for Python. Modern languages like Java and C++ support threads and concurrency, even if those capabilities aren't built in at a low level (C++). I don't think the same thing can be said about Python. The GIL in CPython does in fact prevent threading from being a general solution for CPU bound parallelism. The author is wrong in one important respect: <quote> In this future view, by 2010 we should have the first eight-core systems. In 2014, we're up to 32 cores. By 2017, we've reached an incredible 128 core CPU on a desktop machine. </quote> I don't know where the author got this information, but it is way off. Here are some currently available examples: * Appro now offers a workstation with up to 4 quad core Opterons: http://www.appro.com/product/workstationxtreme_opteron.asp That is a 16 core system. It is 2007, not 2012. * Tilera offers a 64 core CPU that runs SMP Linux. * SiCortex offers a low power 648 processor Linux system that is organized into 6 core SMPs, each of which runs Linux. This week I am at a Department of Defense conference on multicore computing: http://www.ll.mit.edu/HPEC/2007/index.html There is broad agreement from all corners that multi/manycore CPUs will require new ways of expressing parallelism. But in all of the discussions about new approaches, people do assume that threads are a low-level building block that can and should be used for building the higher-level stuff. Intel's Threaded Building Blocks is a perfect example. While it is implemented using threads, it provides much higher level abstractions for developers to use in building applications that will scale well on multicore systems. I would _love_ to have such constructs in Python, but that is simply not possible. And process based solutions don't provide a solution for the many algorithms that require fine grained partitioning and fast data movement. For those of us who do use Python for high performance computing, these issues are critical. In fact, anytime I deal with fine grained parallel algorithms, I use C/C++. I do end up wrapping my low level C/C++ threaded code into Python, but this doesn't always map onto the problem well. The other group of people for whom this is a big issue, are general Python users that don't think they need parallelism. Eventually even these people will become frustrated that their Python codes are the same speed on a 1 core system as a 128 core system. To me this is a significant problem. Brian
Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/
The best way to get information on Usenet is not to ask a question, but to post the wrong information. _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas