Mailman 3 February 2007 - Python-Dev

cross-compile patches
by Han-Wen Nienhuys 07 Mar '07

07 Mar '07

Hello all, A couple of weeks ago, I posted a collection of patches to Python 2.5 on SF.net (#1597850) that enable python to be cross-compiled using standard tools (ie. configure and environment settings), among others on Linux/Darwin to Mingw32/Linux/Freebsd. In response to Martin v Loewis' initial reply we have sent in contributor agreements, but nothing else has happened so far. I was wondering whether there is any way for me to speed up the review process. For example, I could post each patch separately and/or prepare patches to Python SVN. -- Han-Wen Nienhuys - hanwen(a)xs4all.nl - http://www.xs4all.nl/~hanwen

2 3

Lack of sequential decompression in the zipfile module
by Derek Shockey 07 Mar '07

07 Mar '07

Though I am an avid Python programmer, I've never forayed into the area of developing Python itself, so I'm not exactly sure how all this works. I was confused (and somewhat disturbed) to discover recently that the zipfile module offers only one-shot decompression of files, accessible only via the read() method. It is my understanding that the module will handle files of up to 4 GB in size, and the idea of decompressing 4 GB directly into memory makes me a little queasy. Other related modules (zlib, tarfile, gzip, bzip2) all offer sequential decompression, but this does not seem to be the case for zipfile (even though the underlying zlib makes it easy to do). Since I was writing a script to work with potentially very large zipped files, I took it upon myself to write an extract() method for zipfile, which is essentially an adaption of the read() method modeled after tarfile's extract(). I feel that this is something that should really be provided in the zipfile module to make it more usable. I'm wondering if this has been discussed before, or if anyone has ever viewed this as a problem. I can post the code I wrote as a patch, though I'm not sure if my file IO handling is as robust as it needs to be for the stdlib. I'd appreciate any insight into the issue or direction on where I might proceed from here so as to fix what I see as a significant problem. Thanks, Derek

7 8

PEP 306 changes (How to Change Python's Grammar)
by Jack Diederich 03 Mar '07

03 Mar '07

As I found when writing the class decorator patch PEP 306 hasn't been updated since the new AST was added. Here is a suggested replacement block for the Checklist section. AST hackers feel free to make suggestions. Checklist __ Grammar/Grammar: OK, you'd probably worked this one out :) __ Parser/Python.asdl may need changes to match the Grammar. Use Parser/asdl_c.py to regenerate Include/Python-ast.h __ Python/Python-ast.c may need changes to create the AST objects involved with the Grammar change. Lib/compiler/ast.py will need matching changes to the pure-python AST objects. __ Parser/pgen needs to be rerun to regenerate Include/graminit.h and Include/graminit.c __ Python/compile.c: You will need to create or modify the compiler_* functions for your productions. __ You may need to regenerate Lib/symbol.py and/or Lib/token.py and/or Lib/keyword.py __ The parser module. Add some of your new syntax to test_parser, bang on parsermodule.c until it passes. __ The compiler package. A good test is to compile the standard library and test suite with the compiler package and then check it runs. You did add some of your new syntax to the test suite, didn't you? There's a script in Tools/compiler that does this. __ If you've gone so far as to change the token structure of Python, then the tokenizer library module will need to be changed. __ Certain changes may require tweaks to the library module pyclbr. __ Documentation must be written!

4 3

Re: [Python-Dev] with_traceback
by Phillip J. Eby 03 Mar '07

03 Mar '07

At 03:38 PM 2/26/2007 -0700, Andrew Dalke wrote: >Guido's talk at PyCon said: > > > Use raise E(arg).with_traceback(tb) > > instead of raise E, arg, tb > >That seems strange to me because of the mutability. Looking through >the back discussions on this list I see Guido commented: > http://mail.python.org/pipermail/python-3000/2007-February/005689.html > > > Returning the mutated object is acceptable here > > because the *dominant* use case is creating and raising an exception > > in one go: > > > > raise FooException(<args>).with_traceback(<tb>) > >The 3 argument raise statement is rarely used, in my experience. >I believe most don't even know it exists, excepting mostly advanced >Python programmers and language lawyers. > >My concern when I saw Guido's keynote was the worry that >people do/might write code like this > >NO_END_OF_RECORD = ParserError("Cannot find end of record") > >def parse_record(input_file): > ... > raise NO_END_OF_RECORD > ... > > >That is, create instances at the top of the module, to be used >later. This code assume that the NO_END_OF_RECORD >exception instance is never modified. > >If the traceback is added to its __traceback__ attribute then >I see two problems if I were to write code like the above: > > - the traceback stays around "forever" > - the code is no longer thread-safe. Then don't do that, as it's bad style for Python 3.x. ;-) But do note that 3-argument raise should NOT be implemented this way in Python 2.x. 2.6 and other 2.x revisions should still retain the existing raise machinery, it's just that *catching* an exception using 3.x style ("except foo as bar:") should call with_traceback() at the time of the catch. This does mean you won't be able to port your code to 3.x style until you've gotten rid of shared exception instances from all your dependencies, but 3.x porting requires all your dependencies to be ported anyway. It should be sufficient in both 2.x and 3.x for with_traceback() to raise an error if the exception already has a traceback -- this should catch any exception instance reuse. >What is the correct way to rewrite this for use >with "with_traceback"? Is it > >def open_file_on_path(name): > # If nothing exists, raises an exception based on the > # first attempt > saved_err = None > for dirname in _PATH: > try: > return open(os.path.join(dirname, name)) > except Exception, err: > if not saved_err: > saved_err = err > saved_tb = sys.exc_info()[2] > raise saved_err.with_traceback(saved_err.__traceback__) No, it's more like this: try: for dirname in ... try: return ... except Exception as err: saved_err = err raise saved_err finally: del saved_err I've added the outer try-finally block to minimize the GC impact of the *original* code you showed, as the `saved_tb` would otherwise have created a cycle. That is, the addition is not because of the porting, it's just something that you should've had to start with. Anyway, the point here is that in 3.x style, most uses of 3-argument raise just disappear altogether. If you hold on to an exception instance, you have to be careful about it for GC, but no more so than in current Python. The "save one instance and use it forever" use case is new to me - I've never seen nor written code that uses it before now. It's definitely incompatible with 3.x style, though.

12 36

Re: [Python-Dev] NOARGS_NULL
by "Martin v. Löwis" 03 Mar '07

03 Mar '07

K D schrieb: > I thought it would be nice to have the code marked somehow so that it > was obvious what the additional (and obviously unused) parameter was. > The two references (NULL and METH_NOARGS) are both in upper-case and as > C is a case-sensitive language I think it's probably usual to do > case-sensitive greps etc when trying to understand code. Therefore, I > thought NOARGS_NULL as a name had enough information in it for someone > to "find their way" when looking at the code and wondering what that > unused parameter was declared for. Kev, The style-guide in C is that macro names are spelled in all-caps, so if you see an all-caps identifier, you expect it to be a macro name. Regards, Martin

2 1

Class destructor
by Nick Maclaren 02 Mar '07

02 Mar '07

I am gradually making progress with my binary floating-point software, but have had to rewrite several times as I have forgotten most of the details of how to do it! After 30 years, I can't say I am surprised. But I need to clean up workspace when a class (not object) is deallocated. I can't easily use attributes, as people suggested, because there is no anonymous storage built-in type. I could subvert one of the existing storage types (buffer, string etc.), but that is unclean. And I could write one, but that is excessive. So far, I have been unable to track down how to get something called when a class is destroyed. The obvious attempts all didn't work, in a variety of ways. Surely there must be a method? This could be in either Python or C. Thanks. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

3 2

Windows compiler for Python 2.6+
by Chris AtLee 01 Mar '07

01 Mar '07

I just got bitten by the runtime library incompatibility problem on windows when I tried to load a C extension compiled with MSVC 2005 (64-bit) into Python 2.5. I realize that Python2.5 will continue to use MSVC 2003 for compatibility reasons, but I was curious if any thought had been given to the future of the 2.x series. Cheers, Chris

3 4

Integer division operator can give float result?
by Greg Ewing 28 Feb '07

28 Feb '07

Is this intentional? I would have expected the // operator to always give an integer result. Python 2.3 (#1, Aug 5 2003, 15:52:30) [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 7.0 // 2 3.0 -- Greg

4 3

Re: [Python-Dev] Class destructor
by Nick Maclaren 28 Feb '07

28 Feb '07

"Phillip J. Eby" <pje(a)telecommunity.com> wrote: > > Well, you could use a custom metaclass with a tp_dealloc or whatever. Yes, I thought of that, but a custom metaclass to provide one callback is pretty fair overkill! > But I just mainly meant that a PyCObject is almost as good as a weakref > for certain purposes -- i.e. it's got a pointer and a callback. Ah. Yes. Thanks for suggesting it - if it is the simplest way, I may as well do it. > You could of course also use weak references, but that's a bit more > awkward as well. Yes. And they aren't a technology I have used (in Python), so I would have to find out about them. Attributes etc. I have already played with. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [Python-Dev] Class destructor
by Nick Maclaren 28 Feb '07

28 Feb '07

"Phillip J. Eby" <pje(a)telecommunity.com> wrote: > > >But I need to clean up workspace when a class (not object) is > >deallocated. I can't easily use attributes, as people suggested, > >because there is no anonymous storage built-in type. I could subvert > >one of the existing storage types (buffer, string etc.), but that is > >unclean. And I could write one, but that is excessive. > > > >So far, I have been unable to track down how to get something called > >when a class is destroyed. The obvious attempts all didn't work, in > >a variety of ways. Surely there must be a method? This could be in > >either Python or C. > > Have you tried a PyCObject? This is pretty much what they're for: Oh, yes, I use them in several places, but they don't really help. Their first problem is that they take a 'void *' and not a request for space, so I have to allocate and deallocate the space manually. Now, I could add a destructor to each of them and do that, but it isn't really much prettier than subverting one of the semi-generic storage types for an improper purpose! It would be a heck of a lot cleaner to deallocate all of my space in exactly the converse way that I allocate and initialise it. It would also all me to collect and log statistics, should I so choose. This could be VERY useful for tuning! I haven't done that, yet, but might well do so. All in all, what I need is some way to get a callback when a class object is destroyed. Well, actually, any time from its last use for object work and the time that its space is reclaimed - I don't need any more precise time than that. I suppose that I could add a C object as an attribute that points to a block of memory that contains copies of all my workspace pointers, and use the object deallocator to clean up. If all else fails, I will try that, but it seems a hell of a long way round for what I would have thought was a basic requirement. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

2 1