Mailman 3 June 2008 - Python-Dev

doctest, exec and __module__
by Martijn Faassen 26 Jun '08

26 Jun '08

Hi there, I've just witnessed an interesting consequence of the way doctest works. I ran into an issue when doctesting an aspect of SQLAlchemy, where the following guard clause tripped me up: # In the normal call flow, a request for any of the 3 basic collection # types is transformed into one of our trivial subclasses # (e.g. InstrumentedList). Catch anything else that sneaks in here... if cls.__module__ == '__builtin__': raise sa_exc.ArgumentError( "Can not instrument a built-in type. Use a " "subclass, even a trivial one.") My class, coming in as cls here, was defined in a doctest, like this: >>> class Foo(object): ... pass It turns out that doctest compiles and executes this bit of code using a line like this: # Don't blink! This is where the user's code gets run. exec compile(example.source, filename, "single", compileflags, 1) in test.globs This places new key/value pairs into a dictionary, in this case test.globs. Unfortunately when the execution results in a class definition, it'll have its __module__ attribute set to '__builtin__'. Try as I might, I couldn't convince exec to do it any differently. I can't think of an nice way to work around this problem either. The ugly workaround in the doctest itself works: >>> Foo.__module__ = 'whatever' That isn't very nice though. You could also iterate through all the values in the dictionary after each exec, and then check whether it's a class, and if so, set its __module__ to something else than __builtin__, but that doesn't feel very pretty (or efficient) either. Any ideas? Am I missing something? Is there really no way to control this behavior with exec? I'd like to somehow fix doctest.py so it doesn't set the __module__ to '__builtin__' for everything. '__main__' would be nicer, as that's what the interpreter shell does, and a doctest example already looks like the interpreter shell. While the above SQLAlchemy code is hardly pretty, I can't think of any better way to put in a safeguard like that either. Regards, Martijn

2 2

Last "bug weekend"
by Facundo Batista 26 Jun '08

26 Jun '08

Hi all! In Python Argentina we get together in two places, Buenos Aires and Santa Fe, and worked out around 10-12 bugs, which is around the half of the total for both days, so I'm very happy, :) We even had a Python logo shaped cake, see, see [1]. Did you participate? What do you think it's the outcome for both days? Regards, [1] http://www.taniquetil.com.ar/facundo/imgs/tortalogopy.jpeg -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

1 0

Re: [Python-Dev] [Python-checkins] r64424 - in python/trunk: Include/object.h Lib/test/test_sys.py Misc/NEWS Objects/intobject.c Objects/longobject.c Objects/typeobject.c Python/bltinmodule.c
by Guido van Rossum 25 Jun '08

25 Jun '08

I'm curious why the addition of a new feature, past beta 1, was constrained entirely to a single tracker issue. Clearly not enough people were aware of it (or there wouldn't be the discussion about it here). Following the discussion in the issue tracker is really hard (since most of the discussion apparently refers to earlier versions). I also don't see any doc changes. If you want to keep the feature, at least write some documentation for it so everyone knows what is actually implemented, and why, without having to go to the source. --Guido On Tue, Jun 24, 2008 at 11:44 AM, Raymond Hettinger <python(a)rcn.com> wrote: >>>>>> bin(1.0) >>> >>> '0b1 * 2.0 ** 0' >>>>>> >> >> Oops. > > This was checked-in for Issue 3008 which has been under discussion and > development for a while. It was intentionally held until after the beta was > released so that the beta itself would be as stable as possible (in the > final week and a half most of the bots were red and it made no sense to > commit a feature request at that time). > > The patch allows a eval-able representation of a float that works across > platforms and is not dependent of %.17g, atof() or ftoa() routines. The > tool makes it much easier to examine or explain what floating point ops > really do. It was invaluable in the development of math.sum(). I've spent > several days of development time on this and would be really bummed if it > got reversed-out at this point. > > The use of __bin__ is not essential to its implementation so that > implementation detail can be changed if needed. > > > Raymond > > -- --Guido van Rossum (home page: http://www.python.org/~guido/)

5 6

Re: [Python-Dev] [Python-checkins] r64424 - in python/trunk: Include/object.h Lib/test/test_sys.py Misc/NEWS Objects/intobject.c Objects/longobject.c Objects/typeobject.c Python/bltinmodule.c
by Nick Coghlan 25 Jun '08

25 Jun '08

Eric Smith wrote: > Note that in > http://mail.python.org/pipermail/python-dev/2008-February/077062.html, > Guido advocates not adding the __bin__ machinery, which is what lead to > the simple implementation of bin() just calling PyNumber_ToBase and > relying on __index__. > > I don't think __bin__ should be added to 2.6. I don't see the point in > adding a feature in 2.6 that's implemented so differently in 3.0. It's > just asking for porting hassles. > > Instead, I think the approach used in 3.0 (r64451) should be used > instead. That is, if this feature exist at all. I'm -0 on adding > bin(), etc. to floats. The 3.0 approach means that non-float floating point types still can't be displayed properly by bin()/oct()/hex(). The current 2.6 approach means every such class has to implement its own equivalent of PyNumber_ToBase. Both are lousy solutions to a relative non-problem (IMO) that can easily be implemented using a separate display function for those applications which happen to need it. I'd prefer to see a proper PEP for this proposing a new slot that lets any class return an (integer_part, fraction_part) tuple of integers, and have PyNumber_ToBase take care of the actual string formatting. Introducing such a gratuitous incompatibility between 2.6 and 3.0 at this late stage of the process certainly seems highly undesirable to me. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

5 5

Python Windows Installers
by Michael Foord 25 Jun '08

25 Jun '08

Hello all, I'm just doing some housekeeping on a Windows install, and notice that the 'Publisher' of my Python 2.4 and 2.5 installs is shown as "Martin v. Lowis". Whilst I *personally* find this very reassuring I wonder if this is intended / ideal? All the best, Michael Foord

2 3

setitimer, getitimer wrapper (issue2240)
by Guilherme Polo 25 Jun '08

25 Jun '08

Hello, It has been some time since some discussion happened on http://bugs.python.org/issue2240, and I intended to compose this email several days ago but just did it now. This is an attempt to find someone who might have some information/idea that could help solving the current problem regarding the setitimer/getitimer wrapper under FreeBSD (and maybe somewhere else?). The current problem is that the virtual and prof itimers, in some occasions, may never finish, and as I remember from talking with Trent Nelson, the problem showed up when running all the tests but not when running the test_signal alone. So, that is it.. any solutions ? :) Regards, -- -- Guilherme H. Polo Goncalves

1 0

Suggested API change to PyOS_InputHook
by Michael Abbott 24 Jun '08

24 Jun '08

I have a problem with the PyOS_InputHook() API as implemented by Modules/readline.c: there is no way to communicate any interruption seen while waiting at the keyboard back to the process actually reading the line. I have solved this in my own application by creating a new module which reimplements the call_readline function and which has the following patch to readline_until_enter_or_signal(): FD_SET(fileno(rl_instream), &selectset); /* select resets selectset if no input was available */ has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, &timeout); - if(PyOS_InputHook) PyOS_InputHook(); + if (call_readline_InputHook && + call_readline_InputHook(fileno(rl_instream))) + { + has_input = -1; + PyErr_SetInterrupt(); + break; + } } if(has_input > 0) { rl_callback_read_char(); } - else if (errno == EINTR) { + else if (errno == EINTR || has_input < 0) { int s; #ifdef WITH_THREAD PyEval_RestoreThread(_PyOS_ReadlineTState); (I'm sorry, this isn't a proper patch, but hopefully if you've read this far you can figure out where it applies.) Here I've replaced PyOS_InputHook with effective type signature None->None with my own call_readline_InputHook which takes a file descriptor and returns a boolean, returning True iff an interrupt occurred. The file descriptor argument is a triviality, but the boolean return code is clearly crucial. Now, I don't know how this change might fit into mainstream Python, but I think there is a case for making this change. If the incompatibility was acceptable it would be better to change the signature of PyOS_InputHook (I declared my own variable call_readline_InputHook just to avoid accidents). I've checked the current svn trunk, and this change remains applicable. Please CC me in any replies: I am quite literally unable to cope with the volume of traffic that a subscription to python-dev will give me.

3 3

Proposal: Run GC less often
by "Martin v. Löwis" 23 Jun '08

23 Jun '08

Here is my proposal for making the GC run less often. The objective is to avoid the quadratic-time behavior if many objects are created and none of them is garbage. The youngest generation remains collected after 700 allocations, the middle generation after 10 young collections. Therefore, full GC remains considered every 7000 allocations. The two youngest generations don't cause quadratic behavior, as the number of objects in each generation is bounded. Currently, full GC is invoked every 10 middle collections. Under my proposal, 10 middle collections must have passed, PLUS the number of survivor objects from the middle generation must exceed 10% of the number of objects in the oldest generation. As a consequence, garbage collection becomes less frequent as the number of objects on the heap grows, resulting in an amortized O(N) overhead for garbage collection (under the access pattern stated above). To determine the number of survivor objects, counts are taken during the collection. Objects deallocated through reference counting still remain accounted for as survivors (likewise for determining the size of the oldest generation). I made a simulation assuming an initially-empty heap, and invocations of the collector every M=7000 objects. The table below is in units of M and shows the number of objects allocated, and the number of objects inspected since the start of the simulation, for the old and the new scheme (the asterisk indicates that a collection took place; lines with no collection are dropped): total old_inspected new_inspected 10 10* 10* 20 30* 30* 30 60* 60* 40 100* 100* 50 150* 150* 60 210* 210* 70 280* 280* 80 360* 360* 90 450* 450* 100 550* 450 102 550 552* 110 660* 552 115 660 667* 120 780* 667 129 780 796* 130 910* 796 140 1050* 796 ... 940 44650* 7724 942 44650 8666* 950 45600* 8666 960 46560* 8666 970 47530* 8666 980 48510* 8666 990 49500* 8666 1000 50500* 8666 ... 9990 4995000* 95887 As you can see, the old and the new scheme would start of equally until 100M objects have been allocated. The table shows how the old scheme grows quadratically, and the new scheme eventually approaches roughly a factor of ten between the number of objects and the number of times that GC considers an object. Applications with a small number of objects will see no change in behavior, also, applications that create lots of small cycles will continue to see them collected in the youngest or middle generations. Please let me know what you think. Regards, Martin P.S. I don't plan to implement this myself, so if you like the idea, go ahead.

7 10

forceful exit
by tomer filiba 23 Jun '08

23 Jun '08

hi i'm having trouble when forking child processes to serve sockets. the skeleton is something like the following: def work(): try: while True: s = listener.accept()[0] log("hello %s", s) if os.fork() == 0: try: serve_client(sock) finally: sys.exit() #### <<<< (1) else: log("forked child") sock.close() except KeyboardInterrupt: log("got ctrl+c") finally: log("server terminated") listener.close() the problem is that sys.exit() raises an exception, which propagates all the way up and is handled by the finallies... the real code does more than just printing logs, so i can't allow that. i'm forced to resort to os._exit, which does the trick but doesn't perform cleanup, which made me realize there's no clean way in python to *force* exit. i think there ought to be something like sys.forcedexit(), which collects all objects nicely and then exits immediately without letting anything propagate up. this will also solve another problem i came across, with threads. turns out only the main thread can kill the process -- if another thread issues sys.exit, it only kills itself. there's no clean way for a thread to terminate the process... i think there must be. i can contribute a patch in a matter of days... i think it's pretty straight forward (calling Py_Finalize and then exit). aye or nay? -tomer

1 0

Issue 2722
by Neil Muller 22 Jun '08

22 Jun '08

Could some one have a look at the suggested fix for issue 2722? While not a particularly common problem, it can be an annoyance, and the patch looks reasonable. -- Neil Muller

2 1