Folklore that I remember so unreliably I avoid trying to repeat it here held that Python threading had problems on BSD and allied Unixes. What's the status of this? I suspect the answer is, "Everything works, and the only real problem ever was that signals have different semantics under Linux and *BSD." Anyone who can answer explicitly, though, would repre- sent a help to me.
On Thu, Jul 26, 2007 at 04:08:37PM +0000, Cameron Laird wrote:
Folklore that I remember so unreliably I avoid trying to repeat it here held that Python threading had problems on BSD and allied Unixes. What's the status of this? I suspect the answer is, "Everything works, and the only real problem ever was that signals have different semantics under Linux and *BSD." Anyone who can answer explicitly, though, would repre- sent a help to me.
This is just my personal opinion, but I suspect that this is perhaps because people have tried threading more in Python than in many other languages, because Python makes it particularly easy.
I've certainly had the experience that multithreaded stuff I have tried has sometimes had problems under various OSes (Linux, Solaris, OpenBSD, etc) due to operating system bugs with threading in general rather than Python problems per se.
Cameron Laird wrote:
Folklore that I remember so unreliably I avoid trying
to repeat it here
held that Python threading had problems on BSD and allied Unixes. What's
the status of this? I suspect the answer is, "Everything works, and the
only real problem ever was that signals have different semantics under
Linux and *BSD." Anyone who can answer explicitly, though, would repre-
sent a help to me.
I use Python with multithreading applications on FreeBSD
for several years, and really single problem I've discovered
is that default stack size for new threads is small for the
default recursion limit. It can be easily fixed in Python 2.5.
Apart from that everything works OK for me.
Cameron Laird schrieb:
Folklore that I remember so unreliably I avoid trying to repeat it here held that Python threading had problems on BSD and allied Unixes. What's the status of this?
The problem that people run into again and again is the stack size. The BSDs allow for so little stack so that even the quite conservative estimates of Python as to how many recursions you can have are incorrect, and you get an interpreter crash rather than a RuntimeError (as you should).
Furthermore, every time we decrease the that number, the next system release somehow manages to make the limit even smaller. This was never properly analyzed; I suspect that the stack usage of Python increases, either due to compiler changes or due to change to Python itself.
Another annoyance is the ongoing battle with Posix; the BSDs have not been very accepting towards Posix for many years. This resulted in an interpretation of Posix where defining _XOPEN_SOURCE hides many system interfaces, resulting in these system interfaces either not being present, or compilation to fail. I consider this a bug in the system: compilation should never fail if you define _XOPEN_SOURCE, and additional interfaces should be available if requested (that requires a way to request them). The work-around was to not define _XOPEN_SOURCE for those buggy system releases, hoping that the next release would fix the bug. Over the years, the maintainers of these systems seem to have come to a better understanding, so they offer various custom _SOURCE macros (_NETBSD_SOURCE, __BSD_VISIBLE). The latest addition here was OpenBSD, which now supports _BSD_SOURCE (apparently following a tradition set by GNU libc, and perhaps others). So I hope this is fixed for good now (except that FreeBSD may decide to break __BSD_VISIBLE the same way it got "broken" in OpenBSD, so we need to add their "official" feature selection macro once we find out what that is). The same problem exists of course on many other systems, but those solved the problem long ago (e.g. _GNU_SOURCE - glibc, _BSD_TYPES - Irix)
Regards, Martin
Martin v. Löwis wrote:
Cameron Laird schrieb:
Folklore that I remember so unreliably I avoid trying to repeat it here held that Python threading had problems on BSD and allied Unixes. What's the status of this?
The problem that people run into again and again is the stack size. The BSDs allow for so little stack so that even the quite conservative estimates of Python as to how many recursions you can have are incorrect, and you get an interpreter crash rather than a RuntimeError (as you should).
There are 2 aspects to the thread stack size issue:
- the stack size for the primary thread;
- the stack size for created threads.
I haven't done any investigating for FreeBSD 6.x and later, but I know that FreeBSD 4.x had a hard coded stack size of 1MB for the primary thread in a thread enabled application, which is what Martin's comment above particularly applies to. This affects code that doesn't use threads at all, and was particularly painful with REs prior to SRE being made non-recursive.
If you build the interpreter without thread support, stack space is instead controlled by session limits which are usually generous (typically 64MB).
I don't recall exactly FreeBSD's default stack size for threads created via pthread_create() but it is fairly small (32kB or 64kB comes to mind). Zope is one application known to be affected by this lack of stack size in created threads. At least the stack size for new threads can be adjusted at runtime, and a mechanism for doing this was added to Python 2.5.
Furthermore, every time we decrease the that number, the next system release somehow manages to make the limit even smaller. This was never properly analyzed; I suspect that the stack usage of Python increases, either due to compiler changes or due to change to Python itself.
I have seen examples of stack consumption increasing with increasing gcc version number, sometimes depending on optimisation choices.
Regards, Andrew.
--
Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia
-
"Martin v. Löwis" -
Andrew MacIntyre -
Cameron Laird -
Jon Ribbens -
Paul Colomiets