Mailman 3 December 2008 - Python-Dev

2.6 and 3.0 buildbot slaves
by "Martin v. Löwis" 19 Dec '08

19 Dec '08

I have now set up buildbot slaves for 2.6 and 3.0, and turned off the 2.5 ones. Regards, Martin

1 0

Can't have unbuffered text I/O in Python 3.0?
by Fabio Zadrozny 19 Dec '08

19 Dec '08

Hi, I'm currently having problems to get the output of Python 3.0 into the Eclipse console (integrating it into Pydev). The problem appears to be that stdout and stderr are not running unbuffered (even passing -u or trying to set PYTHONUNBUFFERED), and the content only appears to me when a flush() is done or when the process finishes. So, in the search of a solution, I found a suggestion from http://stackoverflow.com/questions/107705/python-output-buffering to use the following construct: sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) But that gives the error below in Python 3.0: sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) File "D:\bin\Python30\lib\os.py", line 659, in fdopen return io.open(fd, *args, **kwargs) File "D:\bin\Python30\lib\io.py", line 243, in open raise ValueError("can't have unbuffered text I/O") ValueError: can't have unbuffered text I/O So, I'd like to know if there's some way I can make it run unbuffered (to get the output contents without having to flush() after each write). Thanks, Fabio

3 3

Please test OSX installer
by "Martin v. Löwis" 19 Dec '08

19 Dec '08

I just created an OSX installer for 2.5.3c1. As it's the first time I do that, I'd appreciate if somebody could test it and report whether it works (as well as the 2.5.2 one did). http://www.python.org/download/releases/2.5.3/ Regards, Martin

3 3

try/except in io.py
by Kristján Valur Jónsson 19 Dec '08

19 Dec '08

Greetings! Yesterday, I committed revision r67843 to py3k. Re-enablign the windows CRT runtime checks showed me that close() was beeing called with an invalid file descriptor. Now, the problem was was in tokenizer.c, but the reason this wasn't caught earlier was, 1) Incorrect error checking for close() in _fileio.c, which I fixed, and 2) Line 384 in io.py, where all exceptions are caught for self.close(). Fixing 1 and patching 2 would bring the problem to light when running the test_imp.py part of the testsuite and, indeed, applying the fix to tokenizer.c would then remove it again. I am a bit worried about 2) thoug. I didn't modify that, but having a catch all clause just to be clean on system exit seems shaky to me. I wonder, is there a way to make such behaviour, if it is indeed necessary, just to be active when exit is in progress? Something like: try: self.close() except: try: if not sys.exiting(): raise except: pass Or better yet, do as we have done often here, just catch the particular problem that occurs during shutdown, most often name error: try: self.close() except (AttributeError, NameError): pass What do you think?

3 4

[ANN] Python 2.4.6 and 2.5.3 (final)
by "Martin v. Löwis" 19 Dec '08

19 Dec '08

On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.4.6 and 2.5.3 (final). 2.5.3 is the last bug fix release of Python 2.5. Future 2.5.x releases will only include security fixes. According to the release notes, about 80 bugs and patches have been addressed since Python 2.5.2, many of them improving the stability of the interpreter, and improving its portability. Since the release candidate, the only change was an update to the Macintosh packaging procedure. 2.4.6 includes only a small number of security fixes. Python 2.6 is the latest version of Python, we're making this release for people who are still running Python 2.4. See the release notes at the website (also available as Misc/NEWS in the source distribution) for details of bugs fixed; most of them prevent interpreter crashes (and now cause proper Python exceptions in cases where the interpreter may have crashed before). For more information on Python 2.4.6 and 2.5.3, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.4.6 http://www.python.org/2.5.3 Highlights of the previous major Python releases are available from the Python 2.5 page, at http://www.python.org/2.4/highlights.html http://www.python.org/2.5/highlights.html Enjoy this release, Martin Martin v. Loewis martin(a)v.loewis.de Python Release Manager (on behalf of the entire python-dev team)

1 0

Summary of Python tracker Issues
by Python tracker 19 Dec '08

19 Dec '08

ACTIVITY SUMMARY (12/12/08 - 12/19/08) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2266 open (+37) / 14258 closed (+20) / 16524 total (+57) Open issues with patches: 762 Average duration of open issues: 704 days. Median duration of open issues: 2530 days. Open Issues Breakdown open 2248 (+37) pending 18 ( +0) Issues Created Or Reopened (58) _______________________________ Doctest module does not work with zipped packages 12/15/08 CLOSED http://bugs.python.org/issue4197 reopened ncoghlan patch configparser DEFAULT 12/12/08 CLOSED http://bugs.python.org/issue4645 created shawn.ashlee distutils chokes on empty options arg in the setup function 12/12/08 http://bugs.python.org/issue4646 created theller patch, patch Builtin parser module fails to parse relative imports 12/12/08 CLOSED http://bugs.python.org/issue4647 created schluehk Fix n//x to n/x in the Docs 12/12/08 CLOSED http://bugs.python.org/issue4648 created Retro Fix a+b to a + b 12/13/08 CLOSED http://bugs.python.org/issue4649 created Retro getopt need re-factor... 12/13/08 http://bugs.python.org/issue4650 created wangchun getopt need re-factor... 12/13/08 CLOSED http://bugs.python.org/issue4651 created wangchun IDLE does not work with Unicode 12/13/08 http://bugs.python.org/issue4652 created zzyzx Patch to fix typos for Py3K 12/13/08 http://bugs.python.org/issue4653 created typo.pl os.path.realpath() get the wrong result 12/13/08 http://bugs.python.org/issue4654 created dirlt during Python installation, setup.py should not use .pydistutils 12/14/08 http://bugs.python.org/issue4655 created jah Python 3 tutorial has old information about dicts 12/14/08 CLOSED http://bugs.python.org/issue4656 created mdcowles Doctest gets line numbers wrongs with <> in name 12/14/08 http://bugs.python.org/issue4657 created ncoghlan missing closing bracket in Functional Programming HOWTO 12/14/08 CLOSED http://bugs.python.org/issue4658 created bgeron compilation warning in Modules/zipimport.c 12/14/08 http://bugs.python.org/issue4659 created pitrou multiprocessing.JoinableQueue task_done() issue 12/14/08 http://bugs.python.org/issue4660 created merrellb email.parser: impossible to read messages encoded in a different 12/14/08 http://bugs.python.org/issue4661 created dato posix module lacks several DeprecationWarning's 12/14/08 http://bugs.python.org/issue4662 created mishok13 patch Increase TextIOWrapper._CHUNK_SIZE 12/14/08 CLOSED http://bugs.python.org/issue4663 created pitrou Regression fix_imports does not refactor multiple imports correc 12/14/08 CLOSED http://bugs.python.org/issue4664 created lregebro Failure to compile trunk on Solaris10/SPARC using C++ compiler 12/14/08 CLOSED http://bugs.python.org/issue4665 created skip.montanaro test_bad_address in test_urllib2_localnet often fails 12/14/08 CLOSED http://bugs.python.org/issue4666 created pitrou Patch with a couple of 2.0isms in tutorial 12/14/08 CLOSED http://bugs.python.org/issue4667 created sgala patch examples in the functional howto are not consistent with 3.X beh 12/14/08 CLOSED http://bugs.python.org/issue4668 created sgala patch bytes,join and bytearray.join not in manual; help for bytes.join 12/15/08 http://bugs.python.org/issue4669 created sjmachin setup.py exception when db_setup_debug = True 12/15/08 http://bugs.python.org/issue4670 created djmdjm pydoc executes the code to be documented 12/15/08 http://bugs.python.org/issue4671 created Jim_C Distutils SWIG support blocks use of SWIG -outdir option 12/15/08 http://bugs.python.org/issue4672 created andybuckley Distutils should provide an uninstall command 12/15/08 http://bugs.python.org/issue4673 created andybuckley test_normalization failures on some buildbot 12/16/08 CLOSED http://bugs.python.org/issue4674 created pitrou urllib's splitpasswd does not accept newline chars in passwords 12/16/08 http://bugs.python.org/issue4675 created mibanescu patch python3 closes + home keys 12/16/08 http://bugs.python.org/issue4676 created Somelauw a list comprehensions tests for pybench 12/16/08 http://bugs.python.org/issue4677 created pitrou patch Unicode: multiple chars for high code points 12/16/08 CLOSED http://bugs.python.org/issue4678 created ede Fork + shelve causes shelve corruption and backtrace 12/16/08 http://bugs.python.org/issue4679 created calmofthestorm deque class should include high-water mark 12/17/08 CLOSED http://bugs.python.org/issue4680 created roysmith mmap offset should be off_t instead of ssize_t, and size calcula 12/17/08 http://bugs.python.org/issue4681 created saa patch 'b' formatter is actually unsigned char 12/17/08 http://bugs.python.org/issue4682 created vt urllib2.HTTPDigestAuthHandler fails on third hostname? 12/17/08 http://bugs.python.org/issue4683 created cmb sys.exit() exits program when non-daemonic threads are still run 12/17/08 http://bugs.python.org/issue4684 created eggy IDLE will not open (2.6.1 on WinXP pro) 12/17/08 http://bugs.python.org/issue4685 created Yo Exceptions in ConfigParser don't set .args 12/17/08 http://bugs.python.org/issue4686 created beazley GC stats not accurate because of debug overhead 12/17/08 http://bugs.python.org/issue4687 created pitrou patch GC optimization: don't track simple tuples and dicts 12/17/08 http://bugs.python.org/issue4688 created pitrou patch Typo in PyObjC URL on "GUI Programming on the Mac" 12/17/08 http://bugs.python.org/issue4689 created mevans asyncore calls handle_write() on closed sockets when use_poll=Tr 12/18/08 http://bugs.python.org/issue4690 created forest IDLE Code Caching Windows 12/18/08 CLOSED http://bugs.python.org/issue4691 created brandon.dixon Framework build fails if OS X on case-sensitive file system 12/18/08 CLOSED http://bugs.python.org/issue4692 created nad patch Idle for Python 3.0 is default even without doing make fullinsta 12/18/08 http://bugs.python.org/issue4693 created orsenthil _call_method() in multiprocessing documentation 12/18/08 CLOSED http://bugs.python.org/issue4694 created beazley Bad AF_PIPE address in multiprocessing documentation 12/18/08 http://bugs.python.org/issue4695 created beazley email module does not fold headers 12/18/08 http://bugs.python.org/issue4696 created bromine patch Clarification needed for subprocess convenience functions in Pyt 12/18/08 http://bugs.python.org/issue4697 created Erik Sternerson Solaris buildbot failure on trunk in test_hostshot 12/18/08 http://bugs.python.org/issue4698 created pitrou Typo in documentation of "signal" 12/19/08 CLOSED http://bugs.python.org/issue4699 created yam850 UnicodeEncodeError in license() 12/19/08 http://bugs.python.org/issue4700 created mnewman range objects becomes hashable after attribute access 12/19/08 http://bugs.python.org/issue4701 created hagen patch Issues Now Closed (54) ______________________ Thread local storage and PyGILState_* mucked up by os.fork() 15 days http://bugs.python.org/issue1683 loewis optimize list comprehensions 297 days http://bugs.python.org/issue2183 pitrou patch, patch gc.DEBUG_STATS reports invalid "elapsed" times 269 days http://bugs.python.org/issue2467 pitrou patch create a numbits() method for int and long types 148 days http://bugs.python.org/issue3439 marketdickinson patch, needs review use string_print() in gdb 116 days http://bugs.python.org/issue3632 amaury.forgeotdarc patch urllib.request and urllib.response cannot handle HTTP1.1 chunked 103 days http://bugs.python.org/issue3761 jhylton 2.6rc1: test_threading hangs on FreeBSD 6.3 i386 90 days http://bugs.python.org/issue3863 loewis patch _hotshot: invalid error control in logreader() 83 days http://bugs.python.org/issue3954 amaury.forgeotdarc patch __main__.__file__ not set correctly when -m switch gets __main__ 67 days http://bugs.python.org/issue4082 ncoghlan textwrap wordsep_re Unicode 53 days http://bugs.python.org/issue4163 pitrou patch Doctest module does not work with zipped packages 0 days http://bugs.python.org/issue4197 ncoghlan patch Pdb cannot access source code in zipped packages. 51 days http://bugs.python.org/issue4201 ncoghlan patch inspect.getsource doesn't work on functions imported from a zipf 47 days http://bugs.python.org/issue4223 ncoghlan cycle created by profile.run 40 days http://bugs.python.org/issue4273 darrenr [2.5 regression] ctypes fails to build on arm-linux-gnu 31 days http://bugs.python.org/issue4303 loewis (Tkinter) Please backport these 26 days http://bugs.python.org/issue4342 loewis A bug in ncurses.h still exists in FreeBSD 4.9 - 4.11 23 days http://bugs.python.org/issue4368 loewis patch Distutils Metadata Documentation Missing "platforms" Keyword 18 days http://bugs.python.org/issue4446 georg.brandl patch CVE-2008-5031 multiple integer overflows 13 days http://bugs.python.org/issue4469 loewis Speed up PyEval_EvalFrameEx when tracing is off. 12 days http://bugs.python.org/issue4477 jyasskin patch logging module __init__ uses has_key 9 days http://bugs.python.org/issue4523 benjamin.peterson patch Registry key not set if unattended installation used 13 days http://bugs.python.org/issue4567 loewis Improved optparse "varargs" callback example 9 days http://bugs.python.org/issue4568 georg.brandl patch reading UTF16-encoded text file crashes if \r on 64-char boundar 7 days http://bugs.python.org/issue4574 pitrou patch compiler: -3 warnings 8 days http://bugs.python.org/issue4578 georg.brandl patch segfault when mutating memoryview to array.array when array is r 11 days http://bugs.python.org/issue4583 pitrou patch, needs review new types example is out of date 7 days http://bugs.python.org/issue4595 georg.brandl 3.0 document tab interpretation change 6 days http://bugs.python.org/issue4603 georg.brandl 3.0 documentation mentions using maketrans from within the strin 4 days http://bugs.python.org/issue4605 benjamin.peterson Small error in "Extending Python with C or C++" 6 days http://bugs.python.org/issue4611 georg.brandl tarfile does not set the creation date and time of the extracted 3 days http://bugs.python.org/issue4616 lars.gustaebel optparse - dosn't distinguish between '--option' and '-option' 0 days http://bugs.python.org/issue4641 marketdickinson optparse - dosn't distinguish between '--option' and '-option' 0 days http://bugs.python.org/issue4642 marketdickinson Minor documentation fault in 2to3 script 0 days http://bugs.python.org/issue4644 benjamin.peterson configparser DEFAULT 2 days http://bugs.python.org/issue4645 loewis Builtin parser module fails to parse relative imports 0 days http://bugs.python.org/issue4647 benjamin.peterson Fix n//x to n/x in the Docs 0 days http://bugs.python.org/issue4648 rhettinger Fix a+b to a + b 1 days http://bugs.python.org/issue4649 gvanrossum getopt need re-factor... 0 days http://bugs.python.org/issue4651 gvanrossum Python 3 tutorial has old information about dicts 0 days http://bugs.python.org/issue4656 benjamin.peterson missing closing bracket in Functional Programming HOWTO 0 days http://bugs.python.org/issue4658 benjamin.peterson Increase TextIOWrapper._CHUNK_SIZE 1 days http://bugs.python.org/issue4663 pitrou Regression fix_imports does not refactor multiple imports correc 0 days http://bugs.python.org/issue4664 benjamin.peterson Failure to compile trunk on Solaris10/SPARC using C++ compiler 1 days http://bugs.python.org/issue4665 loewis test_bad_address in test_urllib2_localnet often fails 1 days http://bugs.python.org/issue4666 pitrou Patch with a couple of 2.0isms in tutorial 0 days http://bugs.python.org/issue4667 georg.brandl patch examples in the functional howto are not consistent with 3.X beh 0 days http://bugs.python.org/issue4668 georg.brandl patch test_normalization failures on some buildbot 0 days http://bugs.python.org/issue4674 pitrou Unicode: multiple chars for high code points 0 days http://bugs.python.org/issue4678 lemburg deque class should include high-water mark 1 days http://bugs.python.org/issue4680 tim_one IDLE Code Caching Windows 0 days http://bugs.python.org/issue4691 amaury.forgeotdarc Framework build fails if OS X on case-sensitive file system 0 days http://bugs.python.org/issue4692 marketdickinson patch _call_method() in multiprocessing documentation 0 days http://bugs.python.org/issue4694 benjamin.peterson Typo in documentation of "signal" 0 days http://bugs.python.org/issue4699 benjamin.peterson Top Issues Most Discussed (10) ______________________________ 49 create a numbits() method for int and long types 148 days closed http://bugs.python.org/issue3439 14 Optimize new io library 13 days open http://bugs.python.org/issue4561 13 GC optimization: don't track simple tuples and dicts 2 days open http://bugs.python.org/issue4688 11 urlopen returns extra, spurious bytes 8 days open http://bugs.python.org/issue4631 10 Py_IS_INFINITY defect causes test_cmath failure on x86 12 days open http://bugs.python.org/issue4575 10 Building a list of tuples has non-linear performance 73 days open http://bugs.python.org/issue4074 10 optimize list comprehensions 297 days closed http://bugs.python.org/issue2183 8 deque class should include high-water mark 1 days closed http://bugs.python.org/issue4680 8 datetime module missing some important methods 656 days open http://bugs.python.org/issue1673409 7 mmap offset should be off_t instead of ssize_t, and size calcul 2 days open http://bugs.python.org/issue4681

1 0

Py3k: magical dir()
by Dmitry Vasiliev 19 Dec '08

19 Dec '08

Hello! I think it's a strange behavior: Python 3.1a0 (py3k:67851, Dec 19 2008, 16:50:32) [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> hash(range(10)) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unhashable type: 'range' >>> dir(range(10)) ['__class__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__'] >>> hash(range(10)) -1211318616 >>> hash(range(1000)) -1211318472 -- Dmitry Vasiliev (dima at hlabs.spb.ru) http://hlabs.spb.ru

5 4

The endless GIL debate: why not remove thread support instead?
by Sturla Molden 18 Dec '08

18 Dec '08

Last month there was a discussion on Python-Dev regarding removal of reference counting to remove the GIL. I hope you forgive me for continuing the debate. I think reference counting is a good feature. It prevents huge piles of garbage from building up. It makes the interpreter run more smoothly. It is not just important for games and multimedia applications, but also servers under high load. Python does not pause to look for garbage like Java or .NET. It only pauses to look for dead reference cycles. This can be safely turned off temporarily; it can be turned off completely if you do not create reference cycles. With Java and .NET, no garbage is ever reclaimed except by the intermittent garbage collection. Python always reclaims an object when the reference count drops to zero whether the GC is enabled or not. This makes Python programs well-behaved. For this reason, I think removing reference counting is a genuinely bad idea. Even if the GIL is evil, this remedy is even worse. I am not a Python core developer; I am a research scientist who use Python because Matlab is (or used to be) a bad programming language, albeit a good computing environment. As most people who have worked with scientific computing know, there are better paradigms for concurrency than threads. In particular, there are message-passing systems like MPI and Erlang, and there are autovectorizing compilers for OpenMP and Fortran 90/95. There are special LAPACK, BLAS and FFT libraries for parallel computer architectures. There are fork-join systems like cilk and java.util.concurrent. Threads seem to be used only because mediocre programmers don't know what else to use. I genuinely think the use of threads should be discouraged. It leads to code that are full of bugs and difficult to maintain - race conditions, deadlocks, and livelocks are common pitfalls. Very few developers are capable of implementing efficient load-balancing by hand. Multi-threaded programs tend to scale badly because they are badly written. If the GIL discourages the abuse of threads, it serves a purpose albeit being evil like the Linux kernel's BKL. Python could be better off doing what tcl does. Allow each process to embed multiple interpreters; run each interpreter in its own thread. Implement a fast message-passing system between the interpreters (e.g. copy-on-write by making communicated objects immutable), and Python would be closer to Erlang than Java. I thus think the main offender is the thread and threading modules - not the GIL. Without thread support in the interpreter, there would be no threads. Without threads, there would be no need for a GIL. Both sources of evil can be removed by just removing thread support from the Python interpreter. In addition, it would make Python faster at executing linear code. Just copy the concurrency model of Erlang instead of Java and get rid of those nasty threads. In the meanwhile, I'll continue to experiment with multiprocessing. Removing reference counting to encourage the use of threads is like shooting ourselves in the leg twice. Thats my two cents on this issue. There is another issue to note as well: If you can endure a 200x loss of efficacy by using Python instead of Fortran, scalability on dual or quad-core processors may not be that important. Just move the bottlenecks out of Python and you are much better off. Regards, Sturla Molden

13 24

Python 3.0 urllib fails with chunked HTTP responses
by Jeremy Hylton 18 Dec '08

18 Dec '08

This bug is pretty serious, because urllib will insert garbage into the application-visible data for a chunked response. It simply ignores the fact that it's reading a chunked response and includes the chunked header data is payload data. The original bug was reported in September, but no one noticed it. It was reported again recently. http://bugs.python.org/issue3761 http://bugs.python.org/issue4631 I suspect we'd want to get a 3.0.1 out as soon as this is fixed, but that's not my call. Jeremy

3 6

Atomic instructions for reference count increment/decrement
by Arnar Birgisson 17 Dec '08

17 Dec '08

Hi all, I'm new here, so bear with me. I tried googling this, but the closest I came up with was a post from 2000. >From the discussion about getting rid of the GIL lately, what I read from it is that reference counting is the main obstacle. My question is, why aren't hardware supported atomic increments and decrements being used for the reference counters? As far as I'm told they are available on most modern platforms (on x86 it is the LOCK instruction prefix) and these incur little overhead. I'd be very happy with pointers to previous discussion on the matter or simple arguments why this would not apply to the Python reference counting mechanism. cheers, Arnar

2 1