Mailman 3 March 2007 - Python-Dev

SoC AST generation question
by Jay Parlar March 20, 2007

March 20, 2007

I'm considering applying to be a student in this year's SoC, and the AST code generation in particular looks interesting to me (listed here: http://wiki.python.org/moin/CodingProjectIdeas/PythonCore). I was wondering a few things: 1) Who would most likely mentor this project? 2) I've never worked in the core before (but have been using Python as my primary language for about 6 years), so I'm wondering if the potential mentor thinks it'd even be feasible for me to jump at a project like this without prior knowledge. I'm interested in this project for two reasons. The first is that I'm still trying to pick my PhD thesis, and I'm leaning in the direction of automated code generation for embedded systems. I feel like working on this project would at least push me one way or another in terms of picking. I've done a major code generation tool before, but it was for a problem domain I was already an "expert" in, so I didn't develop any generic methods. Also, I've been wanting to contribute to Python core for awhile now, and just haven't had the opportunity to get my feet wet with the code. Thanks, Jay P.

4 6

Non-blocking sockets, asynchronous connects and select.select.
by Alan Kennedy March 20, 2007

March 20, 2007

Dear all, I'm working on a select implementation for jython (in combination with non-blocking sockets), and a set of unit tests for same. I decided to run all of the non-blocking unit tests, both server and client side, in the same thread; seems to be a reasonable thing to do. After all, avoiding threads is one of the great benefits of non-blocking sockets. Also, I'm writing the tests so that they'll hopefully pass on both cpython and jython. But I'm getting behaviour I don't expect on cpython, when a non-blocking accept and connect_ex is used, for the server and client sides respectively. The following piece of code outputs what I expect on my jython implementation, which is that both the server accept and client write calls would NOT block. But when I run the code on cpython, the code reports that both calls would block, i.e. that neither side of the socket will progress with the connection. The code is # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- import socket import select SERVER_ADDRESS = ("", 54321) server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server_socket.setblocking(0) server_socket.bind(SERVER_ADDRESS) server_socket.listen(5) client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) client_socket.setblocking(0) result = client_socket.connect_ex(SERVER_ADDRESS) rfds, wfds, xfds = select.select([server_socket], [client_socket], [], 1) if server_socket in rfds: print "Server socket: accept would not block" else: print "Server socket: accept would block" if client_socket in wfds: print "Client socket: write would not block" else: print "Client socket: write would block" server_socket.close() client_socket.close() # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Is there some call that I am missing, e.g. something along the lines of the java finishConnect() method on SocketChannel? http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/SocketChannel.htm… Is there any way to make the above code report, on cpython, that neither side of the socket would block? Am I missing some essential method call that would make either side progress to a connected socket? Thanks, Alan.

3 3

Re: [Python-Dev] Status of thread cancellation
by Nick Maclaren March 19, 2007

March 19, 2007

Jon Ribbens <jon+python-dev(a)unequivocal.co.uk> wrote: > > Can you elaborate on this? You can get zombie entries in the process > table if nobody's called 'wait()' on them, and you can (extremely > rarely) get unkillable process in 'disk-wait' state (usually due to > hardware failure or a kernel bug, I suspect), but I've never heard > of a process on a Unix-like system being unkillable due to something > to do with sockets (or any other kind of file descriptor for that > matter). How could a socket be 'jammed'? What does that even mean? Well, I have seen it hundreds of times on a dozen different Unices; it is very common. You don't always SEE the stuck process - sometimes the 'kill -9' causes the pid to become invisible to ps etc., and just occasionally it can continue to use CPU until the system is rebooted. That is rare, however, and it normally just hangs onto locks, memory and other such resources. Very often its vampiric status is visible only because such things haven't been freed, or when you poke through kernel structures. Sockets get jammed because they are used to connect to subprocesses or kernel threads, which in turn access unreliable I/O devices. If there is a glitch on the device, the error recovery very often fails to work, cleanly, and may wait for an event that will never occur or go into a loop (usually a sleep/poll loop). Typically, a HIGHER level then times out the failing error recovery, so that the normal programmer doesn't notice. But it very often fails to kill the lower level code. As far as applications are concerned, a jammed socket is one where the higher level recovery has NOT done that, and is waiting for the lower level to complete - which it isn't going to do! The other effect that ordinary programmers notice is a system very gradually starting to run down after days/weeks/months of continual operation. The state is cleared by rebooting. Regards, Nick Maclaren.

3 2

Re: [Python-Dev] Status of thread cancellation
by Nick Maclaren March 19, 2007

March 19, 2007

Grrk. I have done this myself, and been involved in one of the VERY few commercial projects that attempted to do it properly (IBM CEL, the other recent one being VMS). I am afraid that there are a lot of misapprehensions here. Several people have said things like: > The thing to model this on, I think, would be the > BSD sigmask mechanism, which lets you selectively > block certain signals to create a critical section > of code. A context manager could be used to make > its use easier and less error-prone (i.e. harder > to block async exceptions and then forget to unblock > them). No, no, no! That is an TRULY horrible! It works fairly well for things like device drivers, which are both structurally simple and with no higher level recovery mechanism, so that a failure turning into a hard hang is not catastrophic. But it is precisely what you DON'T want for complex applications, especially when a thread may need to call an external service 'non-interruptibly'. Think of updating a complex object in a multi-file database, for example. Interrupting half-way through leaves the database in a mess, but blocking interrupts while (possibly remote) file updates complete is asking for a hang. You also see it in horrible GUI (including raw mode text) programs that won't accept interrupts until you have completed the action they think you have started. One of the major advantages of networked systems is that you can usually log in remotely and kill -9 the damn process! The way that I, IBM and DEC approached it was by the classic callback mechanism, with a carefully designed way of promoting unhandled exceptions/interrupts. For example, the following is roughly what I did (somewhat extended, as I didn't do all of this for all exceptions): An event set a defined flag, which could be tested (and cleared) by the thread. If a second, similar event arrived (or it was not handled after a suitable time), the event was escalated. If so, a handler was called that HAD to return (again within a specific time). If a second, similar event arrived or it didn't return by a suitable time, the event was escalated. If so, another handler was called that COULDN'T return. If another event arrived, it returned, or it failed to close down the thread, the event was escalated. If so, the thread's built-in environment was closed down without giving the thread a chance to intervene. If that failed, the event was escalated. If so, the thread was frozen and process termination started. If clean termination failed, the event was escalated. If so, the run-time system produced a dump and killed itself. You can implement a BSD-style ignore by having an initial handler that just clears the flag and returns, but a third interrupt before it does so will force close-down. There was also a facility to escalate an exception at the point of generation, which could be useful for forcible closedown. There are a zillion variations of the above, but all mainframe experience is that callbacks are the only sane way to approach the problem IN APPLICATIONS. In kernel code, that is not so, which is why so many of the computer scientists design BSD-style handling (i.e. they think of kernel programming rather than very complex application programming). > Unconditionally killing a whole process is no big > problem because all the resources it's using get > cleaned up by the OS, and the effect on other > processes is minimal and well-defined (pipes and > sockets get EOF, etc.). But killing a thread can > leave the rest of the program in an awkward state. I wish that were so :-( Sockets, terminals etc. are stateful devices, and killing a process can leave them in a very unclean state. It is one of the most common causes of unkillable processes (the process can't go until its files do, and the socket is jammed). Many people can witness the horrible effects of ptys being left in 'echo off' or worse states, the X focus being left in a stuck override redirect window and so on. But you also have the multi-file database problem, which also applies to shared memory segments. Even if the process dies cleanly, it may be part of an application whose state is global across many processes. One common example is adding or deleting a user, where an unclean kill can leave the system in a very weird state. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

3 2

Re: [Python-Dev] Proposal to revert r54204 (splitext change)
by Taro March 19, 2007

March 19, 2007

<delurk> On Windows it's correct that splitext(".txt")[1] == splitext("foo.txt")[1] and an implementation in which this is not true would be considered buggy. On *ix it's correct that splitext(".txt")[1] != splitext("foo.txt")[1] and the current behaviour is considered buggy. Since programmer expectations are platform-specific, regardless of whether keywords are used or not, why not make the default behaviour platform-specific and document that it's so? Alternatively, if a new path implementation ever gets up, a more neutral solution might be to have a platform-specific Path.filetype, which could handle Mac resources.. </delurk> Cheers, -T

1 0

Two design questions
by Georg Brandl March 19, 2007

March 19, 2007

I'd like to have opinions on two issues: Patch #1682205: suggests to remove code that catches a TypeError and raises one with a different message if it occurs during iterable unpacking. This can mask completely unrelated TypeErrors, while the message in the case that the new error describes is already clear enough. Bug #1683368: object.__new__ rejects arguments if self.__class__.__init__ is object.__init__. Why not simply let object.__init__ do the argument checking itself? Georg

2 1

Re: [Python-Dev] Proposal to revert r54204 (splitext change)
by Phillip J. Eby March 19, 2007

March 19, 2007

At 12:53 PM 3/18/2007 -0700, Guido van Rossum wrote: >This is an experiment for me as well; if you all would prefer me to >stay out of it, I will. With respect to the specific change, it seems to me that there is an emerging consensus for adding keyword arguments to support the new behaviors, so I'm not sure a pronouncement is needed. As far as I'm aware, the main question left open is whether the default behavior should change in a future version of Python, and if so, which version. >To be concrete, I think that if Phillip had written a different kind >of post instead of the one where he requests the reversal of the >submit (only parenthetically mentioning Martin) then perhaps Martin >wouldn't have felt the need to dig in and defend his position, and the >issue might have been resolved quicker and at less emotional expense. Martin's position was already abundantly clear; the fact that he had checked in the change despite prior opposition demonstrated that a personal appeal was already moot -- the "digging in" had already taken place a week or two prior, when the use cases were first presented and objections were first raised, and Martin simply dropped the discussion and checked it in anyway. He left my last message in that discussion (laying out a detailed rationale for rejecting the change) without a reply: http://mail.python.org/pipermail/python-dev/2007-March/071798.html So I was absolutely stunned when I found the change had been checked in, anyway. To be concrete, if Martin had spent less time trying to discredit and discard the use cases of the people being "polled" about the question, a compromise could perhaps have been reached *before* he applied the patch, and the second discussion would never have needed to happen. In other words, the second discussion was the *result* of Martin "digging in" and ignoring objections, not the cause of it. >I'm trying to stay out of the feature discussion, but I would like to >point out that a policy that, in the sake of some strict definition of >backwards compatibility, forces us to introduce new APIs (or new >optional parameters to existing ones, which is really the same thing) >at a high rate is also doomed to have an overall detrimental effect on >the language -- who know, perhaps more so than the occasional >incompatible change. I don't advocate a mechanically-enforced policy, either. But it seems to me that when a behavior is documented and has valid use cases, changing the behavior to benefit people who *didn't* pay any attention to the documentation or test their code for corner cases is punishing the vigilant to aid the ignorant, and that seems unwise for us as a community. Likewise, attempting to retroactively fix latent bugs for one group at the cost of introducing latent bugs for another group doesn't seem like a net improvement.

3 5

Encouraging developers
by A.M. Kuchling March 19, 2007

March 19, 2007

>From <http://ivory.idyll.org/blog/mar-07/five-things-I-hate-about-python>: 4. The patch mafia. I like everyone on python-dev that I meet, but somehow it is annoyingly difficult to get a patch into Python. Like threading, and the stdlib, this is a mixed blessing: you certainly don't want every Joe Schmoe checking in whatever crud he wants. However, the barrier is high enough that I no longer have much interest in spending the time to shepherd a patch through. Yes, this is probably all my fault -- but I still hate it! FWIW, I have a related perception that we aren't getting new core developers. These two problems are probably related: people don't get patches processed and don't become core developers, and we don't have enough core developers to process patches in a timely way. And so we're stuck. Any ideas for fixing this problem? Tangentially related: At PyCon, there was general agreement that exposing a read-only Bazaar/Mercurial/git/whatever version of the repository wouldn't be too much effort, and might make things easier for external people developing patches. Thomas Wouters apparently has private scripts that perform the conversion. What needs to be done to move ahead with this idea? --amk

33 106

Rewrite of cmath module?
by Mark Dickinson March 18, 2007

March 18, 2007

The current cmath module has some significant numerical problems, particularly in the inverse trig and inverse hyperbolic trig functions. I've listed a few of these below, but for now I'll just note the following upsetting result: >>> asin(-1e8) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: math range error (The true value is somewhere around -pi/2 +/- 19.1138j, which is hardly deserving of an overflow error.) I'm wondering whether anyone would be interested in a rewrite of the cmath module. I have a drop-in replacement written in pure Python, based largely on the formulas given by Kahan in his `Much Ado about Nothing's Sign Bit' article, which I believe eliminates the vast majority of the current numerical problems. Besides the numerical fixes, the only significant difference from the current behaviour is that the branch cuts for asinh have been moved. (The docs suggest that the current branch cuts for asinh should be regarded as buggy). I'm offering to translate this into C and add appropriate documentation and test cases, but I have a couple of questions before I start. (1) Is there actually any interest in fixing this module? I guess that the cmath module can't be used that much, otherwise these problems would have surfaced before. (2) (Disregard if the answer to question 1 is `No'.) What should be done about branch cuts? The value of a function on a branch cut is going to depend on signs of zeros, so it'll be pretty much impossible to make any guarantees about the behaviour. Even so, there are two approaches that seem feasible: (a) One can ensure that on a platform that happens to have IEEE-754 compliant hardware and a signed-zero-friendly math library, the complex functions `do the right thing': i.e., are continuous on both sides of each of the branch cuts, using the sign of zero on a branch cut to determine which side of the branch cut the user is interested in. Doing this portably in C89 involves some trickery, but it seems possible. For other platforms behaviour on the branch cuts would be undefined. Alternatively: (b) Force collapsing of all signed zeros to `positive' zero, so that the functions are continuous only on one side of the branch cuts, and the continuity is exactly as specified in the current documentation (though probably including the suggested changes for atan and atanh.) This is mostly what happens with the current implementation, though it's not clear to me whether that's by design or by accident. (c) Some other approach that I haven't considered. Which of these approaches is preferable? The Python module that I have currently does (a). (3) Is it worth trying to be consistent in exceptional cases? The current math module documentation notes that math.log(0) might produce -inf on one platform and raise ValueError on another, so being consistent about whether a non-representable result gives a NaN, an infinity or a Python exception seems as though it would be tricky. I should also note that I've made no effort to do the right thing when the argument to any of the functions contains a NaN or an infinity in either the real or imaginary part. Here are a few examples of the current cmath problems, on my machine (OS X 10.4.8/PowerPC). (*) As already mentioned in the docs, the current branch cuts for asinh are somewhat unconventional, and should probably be fixed. I believe that this is a fairly trivial fix. (*) asinh gives inaccurate answers in the portion of the complex plane between the current branch cuts, when the real part is large and negative. >>> asinh(-1e7+0.9j) (-16.811242831518271+8.9158318101199366e-08j) >>> asinh(-1e8+0.9j) (-19.113827924512311+0j) (The imaginary parts here should be approximately 8.999999999999e-8 and 8.999999999999e-9 respectively.) (*) asinh gives inaccurate results for small x: >>> asinh(1e-12) (1.000088900582091e-12+0j) >>> asinh(1e-20) (4.4408920985006257e-16+0j) (asinh(x) ~ x - 1/6 x^3 + O(x^5) for small x, so the correct values are 1e-12 and 1e-20 to within machine precision.) (*) acos behaves badly for large reals: >>> acos(1e7) 16.805431370234086j # true value ~ 16.811242831518262597j >>> acos(1e8) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: math range error (*) tan and tanh overflow unnecessarily, even though tan(z) and tanh(z) should be representable for all representable complex arguments. >>> tan(1000j) (nannanj) (Actual value: 1j, to within machine precision.) (*) log and sqrt overflow unnecessarily for arguments close to the limits of double precision: >>> z = 1.4e308+1.4e308j >>> sqrt(z) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: math range error >>> log(z) (inf+0.78539816339744828j) (*) sqrt gives inaccurate answers for very small arguments (where the real and imaginary parts are denormalized numbers): >>> z = .5e-323 + .5e-323j >>> sqrt(z) (2.2227587494850775e-162+0j) (Correct value for sqrt(z) is close to 2.442109e-162 + 1.011554e-162j.) (*) exp, cosh and sinh overflow unnecessarily for some values. For example: >>> exp(710+pi/4j) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: math range error The correct value is around 1.579673e308 + 1.579673e308j, which should be representable in IEEE double precision. This last problem is almost not worth fixing: these functions are going to overflow eventually anyway, and whether that happens when the real part of the argument is 709.0 or 710.0 isn't going to make a huge difference to anyone's life. In contrast, functions like acos, log and sqrt should have representable output for *any* finite representable complex number (with the obvious exception of 0 for log), so it's a bit embarrassing if they fail for some inputs. One more thing: since this is my first post to python-dev I should probably introduce myself. I'm a mathematician, working mostly in number theory. I learnt programming and numerical analysis the hard way, coding service-life prediction algorithms for rocket motors in Fortran 77 during several long summers. I've been using Python for more than ten years, mainly for teaching but also occasionally for research purposes. Mark Dickinson

3 2

Re: [Python-Dev] Proposal to revert r54204 (splitext change)
by Delaney, Timothy (Tim) March 18, 2007

March 18, 2007

Delaney, Timothy (Tim) wrote: >> From me - definitely. Damned Outlook, reformatting sent emails! That statement was obviously from me ... Tim Delaney

1 0