On 7/25/10 11:42 PM, "Guido van Rossum" <guido@python.org> wrote:
On Sun, Jul 25, 2010 at 8:31 PM, Peter Portante <peter.a.portante@gmail.com> wrote:
FWIW: We use Python at Tabblo, straddled across Python 2.5.4 and 2.6.5. They work. And they work well. But we make light use of threads (mostly background I/O handling), and heavy use of multiple processes because we can't take advantage of our multi-core systems otherwise.
Isn't this an indication that the GIL is, in fact, not (much of) a problem?
Meaning, we have a working system, so the GIL is not (much of) a problem? Or that we have successfully spent a lot of time and effort rewriting embarrassingly parallel multithreaded algorithms into a bit more complex message passing multi-process algorithms because we can't get the language implementation to make efficient use of multiple CPUs, thus avoiding the GIL from being (much of) a problem? Perhaps we have to ask what does it mean to say the GIL is a problem? If what we mean is that the existence of the GIL does not cause a CPython based program to fail, then yes, it is not a problem at all. In fact, it is a testament to the level of excellence the code has achieved through the hard work folks have put in over the years. If what we mean is that the existence of the GIL prevents a multithreaded CPython application from taking advantage of multiple CPUs, then yes, it is a "problem". So the above statement says that the GIL is not a problem, and that it is THE problem, depending on your definition. :)
I wish those trying to get rid of the GIL well. But it may not be the panacea some folks are hoping for.
You are right, getting rid of the GIL is not a panacea for anything. Removing the GIL means that there will be other changes to the behavioral landscape of the language implementation which folks will have to learn and understand well to write multi-threaded programs that perform well. Anybody wishing to make a whole system run well must engage in that process of learning and discovery. Yet, shouldn't we be able to write a simple embarrassingly parallel multithreaded algorithm in python (no C-extensions) and have its execution use all the cores on a system using CPython? Python is a beautiful language in which to express algorithms. Having to resort to other languages, C extensions, or other implementations of Python, in order to express those algorithms that rely on execution entities sharing a coherent memory space is a limitation imposed by the existence of the GIL in CPython. Is that limitation worth the effort to remove? Perhaps. Perhaps not. Perhaps Jython, or IronPython, or other implementations of Python that don't have a GIL provide a path forward for folks that need that. Those implementations don't currently provide a path forward for what we are doing, so we avoid the use of threads with CPython.
Multi-threaded programming remains hard (and removing the GIL might actually make it harder).
Could we make a statement that the perceived difficulty of multithreaded programming would only increase if a CPython implementation had undocumented behaviors, or undefined behaviors that should be defined? In other words, the difficulty of multithreaded programming is independent of the existence of a/the GIL, but is dependent on the thorough documentation of all language implementation behaviors.
Jython and IronPython don't have a GIL, and I think PyPy may not either.
FWIW: We have considered switching to Jython because it does not have a GIL. Unfortunately, we'd have to find replacements for some of the C-extension modules we use. Sincerely, -peter
Does anyone have experience with GIL-free programming in one of those?