Hello, I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind? Regards Antoine.
I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind?
At work, I'm working on embedded systems (television set top boxes) with a Linux kernel with the GNU C library, and we do use threads! I'm not sure that Python runs on slower/smaller systems because they have other constrains like having very few memory, maybe no MMU and not using the glibc but µlibc for example. There is the "python-on-a-chip" project. It is written from scratch and is very different from CPython. I don't think that it uses threads. http://code.google.com/p/python-on-a-chip/ Victor
On Mon, May 7, 2012 at 9:49 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind?
Gentoo (of course) allows users to build Python without threads; I'm not aware of anyone depending on that, but I sent out a quick question to gentoo-dev. Cheers, Dirkjan
I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind?
For EVE Online, we started out not using threads but relying solely on tasklets. We only added thread supports perhaps five years ago. Other embedded projects _might_ be omitting thread support for a leaner interpreter, but I'm not sure the difference is that large. K
Antoine Pitrou <solipsis@pitrou.net> wrote:
I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind?
_decimal is about 12% faster without threads, because the expensive thread local context can be disabled. On OpenBSD threading leads to strange problems like delayed signals in the REPL http://bugs.python.org/issue8714 . Without threads these problems don't occur. Stefan Krah
On Tue, 8 May 2012 19:40:32 +0200 Stefan Krah <stefan@bytereef.org> wrote:
Antoine Pitrou <solipsis@pitrou.net> wrote:
I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind?
_decimal is about 12% faster without threads, because the expensive thread local context can be disabled.
If you cached the last thread id along with the corresponding context, perhaps it could speed things up in most scenarios? Regards Antoine.
Antoine Pitrou <solipsis@pitrou.net> wrote:
_decimal is about 12% faster without threads, because the expensive thread local context can be disabled.
If you cached the last thread id along with the corresponding context, perhaps it could speed things up in most scenarios?
Nice. This reduces the speed difference to about 4%! Stefan Krah
On Wed, 9 May 2012 11:26:29 +0200 Stefan Krah <stefan@bytereef.org> wrote:
Antoine Pitrou <solipsis@pitrou.net> wrote:
_decimal is about 12% faster without threads, because the expensive thread local context can be disabled.
If you cached the last thread id along with the corresponding context, perhaps it could speed things up in most scenarios?
Nice. This reduces the speed difference to about 4%!
Note that you don't need the actual thread id, the Python thread state is sufficient: PyThreadState_GET should be a simply variable lookup in release builds. Regards Antoine.
Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 9 May 2012 11:26:29 +0200 Stefan Krah <stefan@bytereef.org> wrote:
Antoine Pitrou <solipsis@pitrou.net> wrote:
_decimal is about 12% faster without threads, because the expensive thread local context can be disabled.
If you cached the last thread id along with the corresponding context, perhaps it could speed things up in most scenarios?
Nice. This reduces the speed difference to about 4%!
Note that you don't need the actual thread id, the Python thread state is sufficient: PyThreadState_GET should be a simply variable lookup in release builds.
I've tried both ways now and the speed gain is roughly the same. Perhaps the interpreter as a whole is slightly faster --without-threads? That would explain the remaining speed difference of 4%. Stefan Krah
On Thu, 10 May 2012 20:23:08 +0200 Stefan Krah <stefan@bytereef.org> wrote:
Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 9 May 2012 11:26:29 +0200 Stefan Krah <stefan@bytereef.org> wrote:
Antoine Pitrou <solipsis@pitrou.net> wrote:
_decimal is about 12% faster without threads, because the expensive thread local context can be disabled.
If you cached the last thread id along with the corresponding context, perhaps it could speed things up in most scenarios?
Nice. This reduces the speed difference to about 4%!
Note that you don't need the actual thread id, the Python thread state is sufficient: PyThreadState_GET should be a simply variable lookup in release builds.
I've tried both ways now and the speed gain is roughly the same.
Perhaps the interpreter as a whole is slightly faster --without-threads? That would explain the remaining speed difference of 4%.
It may be. Can you try other benchmarks? Regards Antoine.
Stefan Krah <stefan@bytereef.org> wrote:
Nice. This reduces the speed difference to about 4%!
Note that you don't need the actual thread id, the Python thread state is sufficient: PyThreadState_GET should be a simply variable lookup in release builds.
I've tried both ways now and the speed gain is roughly the same.
Perhaps the interpreter as a whole is slightly faster --without-threads? That would explain the remaining speed difference of 4%.
Actually this seems to be the case: In the benchmark floats are also about 3% faster without threads. Stefan Krah
On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote:
I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind?
I hope that the intent behind asking this question was more of being curious, rather then considering dropping --without-threads: unfortunately, multithreading was, still is and probably will remain troublesome on many supercomputing platforms. Often, once a new supercomputer is launched, as a developer you get a half-baked C/C++ compiler with threading support broken to the point when it's much easier to not use it altogether [*] rather than trying to work around the compiler quirks. Of course, the situation improves over the lifetime of each particular computer, but usually, when everything is halfway working, the computer itself becomes obsolete, so there is not much point in using it anymore. Moreover, these days there is a clear trend towards OpenMP, so it has become even harder to pressure the manufacturers to fix threads, because they have 101 argument why you should port your code to OpenMP instead. HTH. [*]: Another usual candidates for being broken beyond repair are the linker, especially when it comes to shared libraries, and support for advanced C++ language features, such as templates... -- Sincerely yours, Yury V. Zaytsev
Le Tue, 08 Jan 2013 10:28:25 +0100, "Yury V. Zaytsev" <yury@shurup.com> a écrit :
On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote:
I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind?
I hope that the intent behind asking this question was more of being curious, rather then considering dropping --without-threads: unfortunately, multithreading was, still is and probably will remain troublesome on many supercomputing platforms.
I was actually asking this question in the hope that we could perhaps simplify our range of build options (and the corresponding C #define's), but you made a convincing point that we should keep the --without-threads option :-) Thank you Antoine.
[ Weird, I can't see your original e-mail Antoine; hijacking Yury's reply instead. ] On Tue, Jan 08, 2013 at 01:28:25AM -0800, Yury V. Zaytsev wrote:
On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote:
I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind?
All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. Without it, they all wedge in some way or another. (That should be fixed*/investigated, but, until then, yeah, --without-threads allows for a slightly more useful (but still broken) test suite run on these platforms.) [*]: I suspect the problem with at least OpenBSD is that their userland pthreads implementation just doesn't cut it; there is no hope for the really technical tests that poke and prod at things like correct signal handling and whatnot. Trent.
On Tue, 2013-01-08 at 15:09 +0100, Antoine Pitrou wrote:
The original e-mail is quite old (it was sent in May) :-)
I'm sorry about that! I've just stumbled upon this thread and got scared that --without-threads might be going away soon... We've just been porting Python to a new supercomputer, so that we can use Python interface to our simulation software and, at the moment, this is the only way we can get it to work due to the deficiencies of the available software stack (which will hopefully be fixed in the future). Anyways, the point is that it's a recurrent problem, and we hit it every single time with each new machine, so I hope you can understand why I decided to post, even though the original message appeared back in May. -- Sincerely yours, Yury V. Zaytsev
On Wed, Jan 9, 2013 at 7:01 AM, Yury V. Zaytsev <yury@shurup.com> wrote:
Anyways, the point is that it's a recurrent problem, and we hit it every single time with each new machine, so I hope you can understand why I decided to post, even though the original message appeared back in May.
Thank you for speaking up. Easier porting to new platforms is certainly one of the reasons we keep that capability, and it's helpful to have it directly confirmed that there *are* users out there that benefit from it :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 1/10/2013 8:54 AM, Nick Coghlan wrote:
On Wed, Jan 9, 2013 at 7:01 AM, Yury V. Zaytsev <yury@shurup.com> wrote:
Anyways, the point is that it's a recurrent problem, and we hit it every single time with each new machine, so I hope you can understand why I decided to post, even though the original message appeared back in May.
Thank you for speaking up. Easier porting to new platforms is certainly one of the reasons we keep that capability, and it's helpful to have it directly confirmed that there *are* users out there that benefit from it :)
If there is not currently a comment in the setup code explaining this, perhaps it would be a good idea, lest some future developer be tempted to delete the option. -- Terry Jan Reedy
Trent Nelson <trent@snakebite.org> wrote:
All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. Without it, they all wedge in some way or another. (That should be fixed*/investigated, but, until then, yeah, --without-threads allows for a slightly more useful (but still broken) test suite run on these platforms.)
[*]: I suspect the problem with at least OpenBSD is that their userland pthreads implementation just doesn't cut it; there is no hope for the really technical tests that poke and prod at things like correct signal handling and whatnot.
For OpenBSD the situation should be fixed in the latest release: http://www.openbsd.org/52.html#new I haven't tried it myself though. Stefan Krah
On Tue, Jan 08, 2013 at 06:15:45AM -0800, Stefan Krah wrote:
Trent Nelson <trent@snakebite.org> wrote:
All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. Without it, they all wedge in some way or another. (That should be fixed*/investigated, but, until then, yeah, --without-threads allows for a slightly more useful (but still broken) test suite run on these platforms.)
[*]: I suspect the problem with at least OpenBSD is that their userland pthreads implementation just doesn't cut it; there is no hope for the really technical tests that poke and prod at things like correct signal handling and whatnot.
For OpenBSD the situation should be fixed in the latest release:
http://www.openbsd.org/52.html#new
I haven't tried it myself though.
Interesting! I'll look into upgrading the existing Snakebite OpenBSD slaves (they're both at 5.1). Trent.
2013/1/8 Trent Nelson <trent@snakebite.org>:
On Tue, Jan 08, 2013 at 06:15:45AM -0800, Stefan Krah wrote:
Trent Nelson <trent@snakebite.org> wrote:
All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. Without it, they all wedge in some way or another. (That should be fixed*/investigated, but, until then, yeah, --without-threads allows for a slightly more useful (but still broken) test suite run on these platforms.)
[*]: I suspect the problem with at least OpenBSD is that their userland pthreads implementation just doesn't cut it; there is no hope for the really technical tests that poke and prod at things like correct signal handling and whatnot.
For OpenBSD the situation should be fixed in the latest release:
http://www.openbsd.org/52.html#new
I haven't tried it myself though.
Interesting! I'll look into upgrading the existing Snakebite OpenBSD slaves (they're both at 5.1).
Oooh yes, many bugs has been fixed by the implementation of threads in the kernel (rthreads in OpenBSD 5.2)! Just one recent example. On OpenBSD 4.9, FD_CLOEXEC doesn't work with fork()+exec() whereas it works with exec(); on OpenBSD 5.2, both cases work as expected. http://bugs.python.org/issue16850#msg179294 Victor
Yury V. Zaytsev <yury@shurup.com> wrote:
On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote:
I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind?
I hope that the intent behind asking this question was more of being curious, rather then considering dropping --without-threads: unfortunately, multithreading was, still is and probably will remain troublesome on many supercomputing platforms.
Often, once a new supercomputer is launched, as a developer you get a half-baked C/C++ compiler with threading support broken to the point when it's much easier to not use it altogether [*] rather than trying to work around the compiler quirks.
Out of curiosity: Do these incomplete compilers have any problem with either stdint.h or static inline functions in header files? Stefan Krah
Yury V. Zaytsev, 08.01.2013 10:28:
Moreover, these days there is a clear trend towards OpenMP, so it has become even harder to pressure the manufacturers to fix threads, because they have 101 argument why you should port your code to OpenMP instead.
I can't see OpenMP being an *alternative* to threads. You can happily acquire and release the GIL from OpenMP threads, for example. Stefan
participants (10)
-
Antoine Pitrou -
Dirkjan Ochtman -
Kristján Valur Jónsson -
Nick Coghlan -
Stefan Behnel -
Stefan Krah -
Terry Reedy -
Trent Nelson -
Victor Stinner -
Yury V. Zaytsev