Mailman 3 September 2006 - Python-Dev

Re: [Python-Dev] Fwd: Problem withthe API for str.rpartition()
by Michael Chermside 06 Sep '06

06 Sep '06

Jim Jewett writes: > This change [in docs] looks wrong: > > PyDoc_STRVAR(rpartition__doc__, > -"S.rpartition(sep) -> (head, sep, tail)\n\ > +"S.rpartition(sep) -> (tail, sep, head)\n\ Raymond Hettinger replies: > It is correct. There may be some confusion in terminology. Head > and tail do not mean left-side or right-side. Instead, they refer to > the "small part chopped-off" and "the rest that is still choppable". > Think of head and tail in the sense of car and cdr. It is incorrect. The purpose of documentation is to explain things to users, and documentation which fails to achieve this is not "correct". The level of confusion generated by using "head" to refer to the last part of the string and "tail" to refer to the beginning, is quite significant. How about something like this: S.partition(sep) -> (head, sep, tail) S.rpartition(sep) -> (tail, sep, rest) Perhaps someone else can find something clearer than my suggestion, but in my own head, the terms "head" and "tail" are tighly bound with the idea of beginning and end (respectively) rather than with the idea of "small part chopped off" and "big part that is still choppable". -- Michael Chermside

15 26

Re: [Python-Dev] Signals, threads, blocking C functions
by Nick Maclaren 06 Sep '06

06 Sep '06

Johan Dahlin <jdahlin(a)async.com.br> wrote: > > Are you saying that we should let less commonly used platforms dictate > features and functionality for the popular ones? > I mean, who uses HP/UX, SCO and [insert your favorite flavor] as a modern > desktop system where this particular bug makes a difference? You haven't been following the thread. As I posted, this problem occurs to a greater or lesser degree on all platforms. This will be my last posting on the topic, but I shall try to explain. The first problem is in the hardware and operating system. A signal interrupts the thread, and passes control to a handler with a very partial environment and (usually) information on the environment when it was interrupted. If it interrupted the thread in the middle of a system call or other library routine that uses non-Python conventions, the registers and other state may be weird. There ARE solutions to this, but they are unbelievably foul, and even Linux on x86 gas had trouble with this. And, on return, everything has to be reversed entirely transparently! It is VERY common for there to be bugs in the C run-time system and not rare for there to be ones in the kernel (that area of Linux has been rewritten MANY times, for this reason). In many cases, the run-time system simply doesn't pretend to handle interrupts in arbitrary code (which is where the C undefined behaviour is used by vendors). The second problem is that what you can do depends both on what you were doing and how your 'primitive' is implemented. For example, if you call something that takes out even a very short term lock or uses a spin loop to emulate an atomic operation, you had better not use it if you interrupted code that was doing the same. Your thread may hang, crash or otherwise go bananas. Can you guarantee that even write is free of such things? No, and certainly not if you are using a debugger, a profiling library or even tracing system calls. I have often used programs that crashed as soon as I did one of those :-( Related to this is that it is EXTREMELY hard to write synchronisation primitives (mutexes etc.) that are interrupt-safe - MUCH harder than to write thread-safe ones - and few people are even aware of the issues. There was a thread on some Linux kernel mailing list about this, and even the kernel developers were having headaches thinking about the issues. Even if write is atomic, there are gotchas. What if the interrupted code is doing something to that file at the time? Are you SURE that an unexpected operation on it (in the same thread) won't cause the library function of program to get confused? And can you be sure that the write will terminate fast enough to not cause time-critical code to fail? And have you studied the exact semantics of blocking on pipes? They are truly horrible. So this is NOT a matter of platform X is safe and platform Y isn't. Even Linux x86 isn't entirely safe - or wasn't, the last time I heard. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

2 1

Re: [Python-Dev] Fwd: Problem withthe API for str.rpartition()
by Ron Adam 05 Sep '06

05 Sep '06

Ron Adam wrote: Correcting myself... > I hope this discussion is only about the words used and the > documentation and not about the actual order of what is received. I > would expect both the following should be true, and it is the current > behavior. > > ''.join(s.partition(sep)) -> s > ''.join(s.rpartition(sep)) -> s >>> 'abcd'.partition('x') ('abcd', '', '') >>> 'abcd'.rpartition('x') ('abcd', '', '') >>> Ok, I see Raymonds point, they are not what I expected. Although the above is still true, the returned value for the not found condition is inconsistent. _Ron

1 0

Re: [Python-Dev] Signals, threads, blocking C functions
by Nick Maclaren 05 Sep '06

05 Sep '06

"Gustavo Carneiro" <gjcarneiro(a)gmail.com> wrote: > > Anyway, I was speaking hypothetically. I'm pretty sure writing to a > pipe is async signal safe. It is the oldest trick in the book, > everyone uses it. I don't have to see a written signed contract to > know that it works. Ah. Well, I can assure you that it's not the oldest trick in the book, and not everyone uses it. > This is all the evidence that I need. And again I reiterate that > whether or not async safety can be achieved in practice for all > platforms is not Python's problem. I wish you the joy of trying to report a case where it doesn't work to a large vendor and get them to accept that it is a bug. > Although I believe writing to a > pipe is 100% reliable for most platforms. Even if it is not, any > mission critical application relying on signals for correct behaviour > should be rewritten to use unix sockets instead; end of argument. Er, no. There are lots of circumstances where that isn't feasible, such as wanting to close down an application cleanly when the scheduler sends it a SIGXCPU. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

3 2

Fwd: Problem withthe API for str.rpartition()
by Jim Jewett 05 Sep '06

05 Sep '06

Reversing the order of the return tuple will break the alignment with split/rsplit. Why not just change which of the three strings holds the remainder in the not-found case? In rc1, "d".rpartition(".") --> ('d', '', '') If that changes to "d".rpartition(".") --> ('', '', 'd') then (1) the loop will terminate (2) rpartition will be more parallel to partition (and split), (3) people who used rpartition without looping to termination (and therefore didn't catch the problem) will still be able to use their existing working code. (4) the existing docstring would remain correct, though it could still be improved. (It says "returns S and two empty strings", but doesn't specify the order.) -jJ

4 7

Re: [Python-Dev] no remaining issues blocking 2.5 release
by Neal Norwitz 05 Sep '06

05 Sep '06

Gustavo, Did you still want this addressed? Anthony and I made some comments on the bug/patch, but nothing has been updated. n -- On 8/15/06, Gustavo Niemeyer <gustavo(a)niemeyer.net> wrote: > > If you have issues, respond ASAP! The release candidate is planned to > > be cut this Thursday/Friday. There are only a few more days before > > code freeze. A branch will be made when the release candidate is cut. > > I'd like to see problem #1531862 fixed. The bug is clear and the > fix should be trivial. I can commit a fix tonight, if the subprocess > module author/maintainer is unavailable to check it out. > > -- > Gustavo Niemeyer > http://niemeyer.net >

2 1

Re: [Python-Dev] Signals, threads, blocking C functions
by Nick Maclaren 05 Sep '06

05 Sep '06

Jean-Paul Calderone <exarkun(a)divmod.com> wrote: > On Mon, 04 Sep 2006 17:24:56 +0100, > David Hopwood <david.nospam.hopwood@blueyon > der.co.uk> wrote: > >Jean-Paul Calderone wrote: > >> PyGTK would presumably implement its pending call callback by writing a > >> byte to a pipe which it is also passing to poll(). > > > >But doing that in a signal handler context invokes undefined behaviour > >according to POSIX. > > write(2) is explicitly listed as async-signal safe in IEEE Std 1003.1, 2004. > Was this changed in a later edition? Otherwise, I don't understand what you > mean by this. Try looking at the C90 or C99 standard, for a start :-( NOTHING may safely be done in a real signal handler, except possibly setting a value of type static volatile sig_atomic_t. And even that can be problematic. And note that POSIX defers to C on what the C languages defines. So, even if the function is async-signal-safe, the code that calls it can't be! POSIX's lists are complete fantasy, anyway. Look at the one that defines thread-safety, and then try to get your mind around what exit being thread-safe actually implies (especially with regard to atexit functions). Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

4 6

Cross-platform math functions?
by Nick Maclaren 05 Sep '06

05 Sep '06

Andreas Raab <andreas.raab(a)gmx.de> wrote: > > I'm curious if there is any interest in the Python community to achieve > better cross-platform math behavior. A quick test[1] shows a > non-surprising difference between the platform implementations. > Question: Is there any interest in changing the behavior to produce > identical results across platforms (for example by utilizing fdlibm > [2])? Since I have need for a set of cross-platform math functions I'll > probably start with a math-compatible fdlibm module (unless somebody has > done that already ;-) > > [1] Using Python 2.4: > >>> import math > >>> math.cos(1.0e32) > > WinXP: -0.39929634612021897 > LinuxX86: -0.49093671143542561 Well, I hope not, but I am afraid that there is :-( The word "better" is emotive and inaccurate. Such calculations are numerically meaningless, and merely encourage the confusion between consistency and correctness. There is a strong sense in which giving random results between -1 and 1 would be better. Now, I am not saying that you don't have a requirement for consistency but I am saying that confusing it with correctness (as has been fostered by IEEE 754, Java etc.) is harmful. One of the great advantages of the wide variety of arithmetics available in the 1970s is that numerical testing was easier and more reliable - if you got wildly different results on two platforms, you got a strong pointer to numerical problems. That viewpoint is regarded as heresy nowadays, but used not to be! Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

2 1

Re: [Python-Dev] Signals, threads, blocking C functions
by Nick Maclaren 05 Sep '06

05 Sep '06

"Adam Olsen" <rhamph(a)gmail.com> wrote: > On 9/4/06, Gustavo Carneiro <gjcarneiro(a)gmail.com> wrote: > > > Now, we've had this API for a long time already (at least 2.5 > > years). I'm pretty sure it works well enough on most *nix systems. > > Event if it works 99% of the times, it's way better than *failing* > > *100%* of the times, which is what happens now with Python. > > Failing 99% of the time is as bad as failing 100% of the time, if your > goal is to eliminate the short timeout on poll(). 1% is quite a lot, > and it would probably have an annoying tendency to trigger repeatedly > when the user does certain things (not reproducible by you of course). That can make it a lot WORSE that repeated failure. At least with hard failures, you have some hope of tracking them down in a reasonable time. The problem with exception handling code that goes off very rarely, under non-reproducible circumstances, is that it is almost untestable and that bugs in it are positive nightmares. I have been inflicted with quite a large number in my time, and have a fairly good success rate, but the number of people who know the tricks is decreasing. Consider the (real) case where an unpredictable process on a large server (64 CPUs) was failing about twice a week (detectably), with no indication of how many failures were giving wrong answers. We replaced dozens of DIMMs, took days of down time and got nowhere; it then went hard (i.e. one failure a day). After a week's total down time, with me spending 100% of my time on it and the vendor allocating an expert at high priority, we cracked it. We were very lucky to find it so fast. I could give you other examples that were/are there years and decades later, because the pain threshhold never got high enough to dedicate the time (and the VERY few people with experience). I know of at least one such problem in generic TCP/IP (i.e. on Linux, IRIX, AIX and possibly Solaris) that has been there for decades and causes occasional failure in most networked applications/protocols. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [Python-Dev] [Python-checkins] TRUNK IS UNFROZEN, available for 2.6 work if you are so inclined
by Neal Norwitz 05 Sep '06

05 Sep '06

On 8/18/06, Georg Brandl <g.brandl(a)gmx.net> wrote: > > I'd like to commit this. It fixes bug 1542051. > > Index: Objects/exceptions.c ... Georg, Did you still want to fix this? I don't remember anything happening with it. I don't see where _PyObject_GC_TRACK is called, so I'm not sure why _PyObject_GC_UNTRACK is necessary. You should probably add the patch to the bug report and we can discuss there. n

1 0