
Hello, I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind? Regards Antoine.

At work, I'm working on embedded systems (television set top boxes) with a Linux kernel with the GNU C library, and we do use threads! I'm not sure that Python runs on slower/smaller systems because they have other constrains like having very few memory, maybe no MMU and not using the glibc but µlibc for example. There is the "python-on-a-chip" project. It is written from scratch and is very different from CPython. I don't think that it uses threads. http://code.google.com/p/python-on-a-chip/ Victor

Antoine Pitrou <solipsis@pitrou.net> wrote:
_decimal is about 12% faster without threads, because the expensive thread local context can be disabled. On OpenBSD threading leads to strange problems like delayed signals in the REPL http://bugs.python.org/issue8714 . Without threads these problems don't occur. Stefan Krah

On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote:
I hope that the intent behind asking this question was more of being curious, rather then considering dropping --without-threads: unfortunately, multithreading was, still is and probably will remain troublesome on many supercomputing platforms. Often, once a new supercomputer is launched, as a developer you get a half-baked C/C++ compiler with threading support broken to the point when it's much easier to not use it altogether [*] rather than trying to work around the compiler quirks. Of course, the situation improves over the lifetime of each particular computer, but usually, when everything is halfway working, the computer itself becomes obsolete, so there is not much point in using it anymore. Moreover, these days there is a clear trend towards OpenMP, so it has become even harder to pressure the manufacturers to fix threads, because they have 101 argument why you should port your code to OpenMP instead. HTH. [*]: Another usual candidates for being broken beyond repair are the linker, especially when it comes to shared libraries, and support for advanced C++ language features, such as templates... -- Sincerely yours, Yury V. Zaytsev

Le Tue, 08 Jan 2013 10:28:25 +0100, "Yury V. Zaytsev" <yury@shurup.com> a écrit :
I was actually asking this question in the hope that we could perhaps simplify our range of build options (and the corresponding C #define's), but you made a convincing point that we should keep the --without-threads option :-) Thank you Antoine.

[ Weird, I can't see your original e-mail Antoine; hijacking Yury's reply instead. ] On Tue, Jan 08, 2013 at 01:28:25AM -0800, Yury V. Zaytsev wrote:
All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. Without it, they all wedge in some way or another. (That should be fixed*/investigated, but, until then, yeah, --without-threads allows for a slightly more useful (but still broken) test suite run on these platforms.) [*]: I suspect the problem with at least OpenBSD is that their userland pthreads implementation just doesn't cut it; there is no hope for the really technical tests that poke and prod at things like correct signal handling and whatnot. Trent.

On Tue, 2013-01-08 at 15:09 +0100, Antoine Pitrou wrote:
The original e-mail is quite old (it was sent in May) :-)
I'm sorry about that! I've just stumbled upon this thread and got scared that --without-threads might be going away soon... We've just been porting Python to a new supercomputer, so that we can use Python interface to our simulation software and, at the moment, this is the only way we can get it to work due to the deficiencies of the available software stack (which will hopefully be fixed in the future). Anyways, the point is that it's a recurrent problem, and we hit it every single time with each new machine, so I hope you can understand why I decided to post, even though the original message appeared back in May. -- Sincerely yours, Yury V. Zaytsev

On Wed, Jan 9, 2013 at 7:01 AM, Yury V. Zaytsev <yury@shurup.com> wrote:
Thank you for speaking up. Easier porting to new platforms is certainly one of the reasons we keep that capability, and it's helpful to have it directly confirmed that there *are* users out there that benefit from it :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Trent Nelson <trent@snakebite.org> wrote:
For OpenBSD the situation should be fixed in the latest release: http://www.openbsd.org/52.html#new I haven't tried it myself though. Stefan Krah

2013/1/8 Trent Nelson <trent@snakebite.org>:
Oooh yes, many bugs has been fixed by the implementation of threads in the kernel (rthreads in OpenBSD 5.2)! Just one recent example. On OpenBSD 4.9, FD_CLOEXEC doesn't work with fork()+exec() whereas it works with exec(); on OpenBSD 5.2, both cases work as expected. http://bugs.python.org/issue16850#msg179294 Victor

At work, I'm working on embedded systems (television set top boxes) with a Linux kernel with the GNU C library, and we do use threads! I'm not sure that Python runs on slower/smaller systems because they have other constrains like having very few memory, maybe no MMU and not using the glibc but µlibc for example. There is the "python-on-a-chip" project. It is written from scratch and is very different from CPython. I don't think that it uses threads. http://code.google.com/p/python-on-a-chip/ Victor

Antoine Pitrou <solipsis@pitrou.net> wrote:
_decimal is about 12% faster without threads, because the expensive thread local context can be disabled. On OpenBSD threading leads to strange problems like delayed signals in the REPL http://bugs.python.org/issue8714 . Without threads these problems don't occur. Stefan Krah

On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote:
I hope that the intent behind asking this question was more of being curious, rather then considering dropping --without-threads: unfortunately, multithreading was, still is and probably will remain troublesome on many supercomputing platforms. Often, once a new supercomputer is launched, as a developer you get a half-baked C/C++ compiler with threading support broken to the point when it's much easier to not use it altogether [*] rather than trying to work around the compiler quirks. Of course, the situation improves over the lifetime of each particular computer, but usually, when everything is halfway working, the computer itself becomes obsolete, so there is not much point in using it anymore. Moreover, these days there is a clear trend towards OpenMP, so it has become even harder to pressure the manufacturers to fix threads, because they have 101 argument why you should port your code to OpenMP instead. HTH. [*]: Another usual candidates for being broken beyond repair are the linker, especially when it comes to shared libraries, and support for advanced C++ language features, such as templates... -- Sincerely yours, Yury V. Zaytsev

Le Tue, 08 Jan 2013 10:28:25 +0100, "Yury V. Zaytsev" <yury@shurup.com> a écrit :
I was actually asking this question in the hope that we could perhaps simplify our range of build options (and the corresponding C #define's), but you made a convincing point that we should keep the --without-threads option :-) Thank you Antoine.

[ Weird, I can't see your original e-mail Antoine; hijacking Yury's reply instead. ] On Tue, Jan 08, 2013 at 01:28:25AM -0800, Yury V. Zaytsev wrote:
All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. Without it, they all wedge in some way or another. (That should be fixed*/investigated, but, until then, yeah, --without-threads allows for a slightly more useful (but still broken) test suite run on these platforms.) [*]: I suspect the problem with at least OpenBSD is that their userland pthreads implementation just doesn't cut it; there is no hope for the really technical tests that poke and prod at things like correct signal handling and whatnot. Trent.

On Tue, 2013-01-08 at 15:09 +0100, Antoine Pitrou wrote:
The original e-mail is quite old (it was sent in May) :-)
I'm sorry about that! I've just stumbled upon this thread and got scared that --without-threads might be going away soon... We've just been porting Python to a new supercomputer, so that we can use Python interface to our simulation software and, at the moment, this is the only way we can get it to work due to the deficiencies of the available software stack (which will hopefully be fixed in the future). Anyways, the point is that it's a recurrent problem, and we hit it every single time with each new machine, so I hope you can understand why I decided to post, even though the original message appeared back in May. -- Sincerely yours, Yury V. Zaytsev

On Wed, Jan 9, 2013 at 7:01 AM, Yury V. Zaytsev <yury@shurup.com> wrote:
Thank you for speaking up. Easier porting to new platforms is certainly one of the reasons we keep that capability, and it's helpful to have it directly confirmed that there *are* users out there that benefit from it :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Trent Nelson <trent@snakebite.org> wrote:
For OpenBSD the situation should be fixed in the latest release: http://www.openbsd.org/52.html#new I haven't tried it myself though. Stefan Krah

2013/1/8 Trent Nelson <trent@snakebite.org>:
Oooh yes, many bugs has been fixed by the implementation of threads in the kernel (rthreads in OpenBSD 5.2)! Just one recent example. On OpenBSD 4.9, FD_CLOEXEC doesn't work with fork()+exec() whereas it works with exec(); on OpenBSD 5.2, both cases work as expected. http://bugs.python.org/issue16850#msg179294 Victor
participants (10)
-
Antoine Pitrou
-
Dirkjan Ochtman
-
Kristján Valur Jónsson
-
Nick Coghlan
-
Stefan Behnel
-
Stefan Krah
-
Terry Reedy
-
Trent Nelson
-
Victor Stinner
-
Yury V. Zaytsev