Mailman 3 October 2010 - Python-Dev

My work on Python3 and non-ascii paths is done
by Victor Stinner Oct. 23, 2010

Oct. 23, 2010

Hi, Seven months after my first commit related to this issue, the full test suite of Python 3.2 pass with ASCII, ISO-8859-1 and UTF-8 locale encodings in a non- ascii source directory. It means that Python 3.2 now process correctly filenames in all modules, build scripts and other utilities, with any locale encoding. General changes: * Encode/decode filenames with the locale encoding, instead of utf-8, until the filesystem is set * mbcs encoding (Windows filesystem encoding) is now strict by default, whereas it ignores unencodable characters and replace undecodable bytes in Python 3.1. Old behaviour can still be used using the right error handler: 'ignore' to encode, 'replace' to decode. * tarfile uses utf-8 encoding on Windows (instead of mbcs), and the surrogateescape error handler on all OSes * sys.getfilesystemencoding() cannot be None anymore * Don't accept bytearray as filenames anymore Changes of the Python API: * Add os.environb: bytes version of os.environ, os.getenvb() function and os.supports_bytes_environ constant * Add os.fsencode() and os.fsdecode() functions * Remove sys.setfilesystemencoding() function Changes of the C API: * Add PyUnicode_EncodeFSDefault() function * Add PyUnicode_FSDecoder() ParseTuple converter * Add PySys_FormatStdout(), PySys_FormatStderr() and PyErr_WarnFormat() functions * Add PyUnicode_AsWideCharString() function: don't need a buffer size. * Add Py_UNICODE_strrchr(), Py_UNICODE_strcat(), PyUnicode_AsUnicodeCopy() and Py_UNICODE_strncmp() functions * PyUnicode_DecodeFSDefault() and PyUnicode_DecodeFSDefaultAndSize() use the surrogateescape error handler * File utilities: add _Py_wchar2char() (reverse of Py_char2wchar()), _Py_stat() and _Py_fopen() functions; move all file utilities to Python/fileutils.c * The format string of PyUnicode_FromFormat() and PyErr_Format() is now pure ASCII: raise an error on non-ascii character * PyUnicode_FSConverter() doesn't accept bytearray anymore Bugfixes: * Fix modules: tarfile, pickle, pickletools, ctypes, subprocess, bz2, ssl, profile, xmlrpclib, platform, libpython (gdb plugin), sqlite, distutils.log, locale, _warnings, zipimport, imp * Fix functions: os.exec*(), os.system(), ctypes.dlopen(), os.getenv(), os.get_exec_path() * Fix tests: test_gdb, test_httpservers, test_cmd_line, test_size, test_generic_path, test_subprocess, test_doctest, test_cmd_line_script * Fix utf-8 encoder to support error handlers producing unicode string (eg. 'backslashreplace') * Fix conversion from unicode to a wide character string if Py_UNICODE and wchar_t have different sizes: UTF-16 => UTF-32 or UTF-32 => UTF-16 * Fix Python command line parser if the the command line contains surrogates * Avoid _PyUnicode_AsString() because it returns NULL if the string contains surrogates, or catch the error * Fix regrtest.py to support surrogate characters in the current working directory and in the tracebacks I wrote also some tests and documentation. The most difficult part was to debug Python initialization (Py_InitializeEx and calculate_path) and the import machinery (import.c, zipimport.c), because gdb does sometimes crash (for various reasons) and because the import machinery is fragile and difficult to understand. A special thanks to Marc-Andre Lemburg, Martin v. Löwis, Antoine Pitrou and Amaury Forgeot d'Arc for their help, useful advices and code reviews! -- Bonus: short story of PYTHONFSENCODING --- In the middle of August, I created the PYTHONFSENCODING environment variable, as suggested by Marc-Andre Lemburg. Because of this variable and because Python used utf-8 until the filesystem encoding is known, I had to write ugly and fragile "redecode" functions to redecode all filenames of all objects (sys.path, sys.meta_path, sys.executable, sys.modules, all code objects, etc.). Then I found 4 issues related to PYTHONFSENCODING, inconsistencies between the filesystem encoding and the locale encoding. It was not easy to decide how to fix these issues, but at the end, we choosed to drop PYTHONFSENCODING variable, use the locale encoding as the filesystem encoding, and always use utf-8 as the filesystem encoding on Mac OS X. -- Victor Stinner http://www.haypocalc.com/

10 12

Distutils2 scripts
by Tarek Ziadé Oct. 22, 2010

Oct. 22, 2010

Hello, In the Distutils2 project, we'll have quite a few scripts that can be called via -m $ python -m distutils2.depgraph : shows a dependency graph $ python -m distutils2.install : installs a project $ python -m distutils2.run command : runs a distutils2 command etc.. The feedback I received for this is pretty clear: people want a single script that can be called directly. e.g. $ distutils2 depgraph $ distutils2 install $ distutils2 run command Fair enough, I can add that script in the project and get it installed when people install the project. I just wanted to make sure that once distutils2 is back in the stdlib, it's OK for us to add that script in the distribution. I remember that people did not want "yet another script" in there, so I just wanted to double-check before I take this path Thanks ! Tarek -- Tarek Ziadé | http://ziade.org

27 73

SVN rev. 85392 broke module handling in py3k
by Stefan Behnel Oct. 22, 2010

Oct. 22, 2010

Hi, since SVN rev. 85392, Cython's installation fails on the py3k branch with a weird globals error. I think it is related to some sys.modules magic that we do in order to support running Cython in Python 3 using lib2to3. Basically, what we do is, we import some parts of Cython at the beginning that are Py3 clean, specifically some distutils build_ext replacement for building Cython modules. Then we start up distutils, which first runs lib2to3 on Cython's sources to convert them into Py3 code. When it then gets to building the binary modules, we remove all Cython modules and packages from sys.modules and reimport their 2to3-ed sources so that we can run the complete compiler during the installation (to bootstrap parts of Cython into binary modules). Since the above revision, this process bails out with an error when accessing "os.path" because "os" is None. The "os" module is imported globally in our early-imported build_ext module, more or less like this: import os from distutils.command import build_ext as _build_ext class build_ext(_build_ext.build_ext): def build_extensions(self): print(os) # prints None! I suspect that the fact that we remove the modules from sys.modules somehow triggers the cleanup of these modules while there are still objects from these modules alive that refer to their globals. So, what I think is happening is that the module cleanup sets the module's globals to None before the objects from that module that refer to these globals have actually gone out of scope. Could someone (benjamin?) please look into this? Thanks, Stefan

4 12

python-2.6.6 coredump running newspipe
by Thomas Klausner Oct. 21, 2010

Oct. 21, 2010

Hi! I'm running newspipe-1.1.9, an RSS reader (http://newspipe.sourceforge.net/), on NetBSD-5.99.11/amd64 using Python-2.6.6. Sometimes, it core dumps with particular feeds in the configuration (I guess depending on the feed, because when I comment out the offending feed in the opml file, it runs through to completion). The backtrace looks like this: Core was generated by `python'. Program terminated with signal 10, Bus error. #0 0x00007f7ffdc35a21 in PyOS_snprintf (str=0x7f7ff5dfe3d8 "@", size=120, format=0x1 <Address 0x1 out of bounds>) at Python/mysnprintf.c:43 43 { (gdb) bt #0 0x00007f7ffdc35a21 in PyOS_snprintf (str=0x7f7ff5dfe3d8 "@", size=120, format=0x1 <Address 0x1 out of bounds>) at Python/mysnprintf.c:43 #1 0x00007f7ffdc471a6 in PyOS_ascii_formatd (buffer=0x7f7ff5dfe3d8 "@", buf_size=120, format=0x7f7ff5dfe388 "%.2f", d=0.15256118774414062) at Python/pystrtod.c:455 #2 0x00007f7ffdbaa7fa in formatfloat (buf=0x7f7ff5dfe3d8 "@", buflen=120, flags=16, prec=2, type=102, v=0x7f7ffcc6d510) at Objects/stringobject.c:4378 #3 0x00007f7ffdbabd32 in PyString_Format (format=0x7f7ffc8144e0, args=0x7f7ffcc6d510) at Objects/stringobject.c:4943 #4 0x00007f7ffdbaa3b0 in string_mod (v=0x7f7ffc8144e0, w=0x7f7ffcc6d510) at Objects/stringobject.c:4116 #5 0x00007f7ffdb459db in binary_op1 (v=0x7f7ffc8144e0, w=0x7f7ffcc6d510, op_slot=32) at Objects/abstract.c:917 #6 0x00007f7ffdb45c81 in binary_op (v=0x7f7ffc8144e0, w=0x7f7ffcc6d510, op_slot=32, op_name=0x7f7ffdc6c089 "%") at Objects/abstract.c:969 #7 0x00007f7ffdb467ad in PyNumber_Remainder (v=0x7f7ffc8144e0, w=0x7f7ffcc6d510) at Objects/abstract.c:1221 #8 0x00007f7ffdc08a03 in PyEval_EvalFrameEx (f=0x7f7fefa1dab0, throwflag=0) at Python/ceval.c:1180 #9 0x00007f7ffdc1175f in fast_function (func=0x7f7ff8a9bed8, pp_stack=0x7f7ff5dfeae8, n=1, na=1, nk=0) at Python/ceval.c:3836 #10 0x00007f7ffdc11565 in call_function (pp_stack=0x7f7ff5dfeae8, oparg=1) at Python/ceval.c:3771 #11 0x00007f7ffdc0d81f in PyEval_EvalFrameEx (f=0x7f7fee920420, throwflag=0) at Python/ceval.c:2412 #12 0x00007f7ffdc0f715 in PyEval_EvalCodeEx (co=0x7f7ffcc247b0, globals=0x7f7ffd1c5880, locals=0x0, args=0x7f7ff5b0aac8, argcount=8, kws=0x7f7ff5b0ab08, kwcount=0, defs=0x7f7ff8d3c4e8, defcount=5, closure=0x0) at Python/ceval.c:3000 #13 0x00007f7ffdc1184a in fast_function (func=0x7f7ff8a9cc80, pp_stack=0x7f7ff5dfeff8, n=8, na=8, nk=0) at Python/ceval.c:3846 #14 0x00007f7ffdc11565 in call_function (pp_stack=0x7f7ff5dfeff8, oparg=7) at Python/ceval.c:3771 #15 0x00007f7ffdc0d81f in PyEval_EvalFrameEx (f=0x7f7ff5b0a820, throwflag=0) at Python/ceval.c:2412 #16 0x00007f7ffdc1175f in fast_function (func=0x7f7ff8a9e140, pp_stack=0x7f7ff5dff358, n=1, na=1, nk=0) at Python/ceval.c:3836 #17 0x00007f7ffdc11565 in call_function (pp_stack=0x7f7ff5dff358, oparg=0) at Python/ceval.c:3771 #18 0x00007f7ffdc0d81f in PyEval_EvalFrameEx (f=0x7f7ff5b0a420, throwflag=0) at Python/ceval.c:2412 #19 0x00007f7ffdc1175f in fast_function (func=0x7f7ffca1db90, pp_stack=0x7f7ff5dff6b8, n=1, na=1, nk=0) at Python/ceval.c:3836 #20 0x00007f7ffdc11565 in call_function (pp_stack=0x7f7ff5dff6b8, oparg=0) at Python/ceval.c:3771 #21 0x00007f7ffdc0d81f in PyEval_EvalFrameEx (f=0x7f7ff5b03190, throwflag=0) at Python/ceval.c:2412 #22 0x00007f7ffdc0f715 in PyEval_EvalCodeEx (co=0x7f7ffca0d4e0, globals=0x7f7ffca473a0, locals=0x0, args=0x7f7ff04d3e68, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3000 #23 0x00007f7ffdb7a612 in function_call (func=0x7f7ffca1daa0, arg=0x7f7ff04d3e50, kw=0x0) at Objects/funcobject.c:524 #24 0x00007f7ffdb495e8 in PyObject_Call (func=0x7f7ffca1daa0, arg=0x7f7ff04d3e50, kw=0x0) at Objects/abstract.c:2492 #25 0x00007f7ffdb5eca0 in instancemethod_call (func=0x7f7ffca1daa0, arg=0x7f7ff04d3e50, kw=0x0) at Objects/classobject.c:2579 #26 0x00007f7ffdb495e8 in PyObject_Call (func=0x7f7ff8ac2a00, arg=0x7f7ffd112050, kw=0x0) at Objects/abstract.c:2492 #27 0x00007f7ffdc10cd3 in PyEval_CallObjectWithKeywords (func=0x7f7ff8ac2a00, arg=0x7f7ffd112050, kw=0x0) at Python/ceval.c:3619 #28 0x00007f7ffdc4e69f in t_bootstrap (boot_raw=0x7f7ffd1b4590) at ./Modules/threadmodule.c:428 #29 0x00007f7ffd90ba32 in pthread_setcancelstate () from /usr/lib/libpthread.so.1 #30 0x00007f7ffd26e9b0 in ___lwp_park50 () from /usr/lib/libc.so.12 #31 0x0000000000000000 in ?? () (gdb) fr 1 #1 0x00007f7ffdc471a6 in PyOS_ascii_formatd (buffer=0x7f7ff5dfe3d8 "@", buf_size=120, format=0x7f7ff5dfe388 "%.2f", d=0.15256118774414062) at Python/pystrtod.c:455 455 PyOS_snprintf(buffer, buf_size, format, d); (gdb) l 450 format = tmp_format; 451 } 452 453 454 /* Have PyOS_snprintf do the hard work */ 455 PyOS_snprintf(buffer, buf_size, format, d); 456 457 /* Do various fixups on the return string */ 458 459 /* Get the current locale, and find the decimal point string. (gdb) p format $1 = 0x7f7ff5dfe388 "%.2f" (gdb) fr 0 #0 0x00007f7ffdc35a21 in PyOS_snprintf (str=0x7f7ff5dfe3d8 "@", size=120, format=0x1 <Address 0x1 out of bounds>) at Python/mysnprintf.c:43 43 { (gdb) l 38 CAUTION: Unlike C99, str != NULL and size > 0 are required. 39 */ 40 41 int 42 PyOS_snprintf(char *str, size_t size, const char *format, ...) 43 { 44 int rc; 45 va_list va; 46 47 va_start(va, format); (gdb) It seems that the format argument is corrupted while calling PyOS_snprintf. Any ideas what could cause this or how to fix this? Thanks, Thomas

3 4

Re: [Python-Dev] [Python-checkins] r85733 - in python/branches/py3k: Doc/library/logging.rst Lib/logging/__init__.py
by Éric Araujo Oct. 21, 2010

Oct. 21, 2010

+``Filters` can be used by ``Handlers`` and ``Loggers`` for more ↑ missing ` here. Regards

1 0

hg migration schedule
by Dirkjan Ochtman Oct. 21, 2010

Oct. 21, 2010

Okay, we're getting somewhere here. I've been conferring with Georg on when we can do the conversion. He's done some testing with the buildbot setup, which seems to be mostly done. He also completed preliminary ports of the hooks used in the SVN repository. He managed to convince me that we're at a point where it would be good to post some kind of schedule, so here we go. 2010-11-13: availability of a test repo I'm working towards finally having a test repository that's close to what we'll have for the final repository. I hope to have it done sooner than by Nov 13, so it seems like a fairly realistic date. I'm not promising we'll also have docs done by then, but good documentation will be my focus as soon as I'm done with the test repository. I should also mention that I'll be visiting the US east cost (Boston and NYC) from Nov 13 to Nov 21, so I probably won't be spending much time on this that week. 2010-12-12: final conversion (tentative) Georg would really like the conversion to be done by the end of the year, and it seems that this is within reach. The aim here is not to disturb the 3.2 release process too much, of course, while still giving Georg more powerful tools to manage what's left of the release process. We're currently tracking progress in a todo.txt file in the pymigr repo [1] (and in the #python-dev IRC channel), so if you want to help out, you're more than welcome! We're also going to need at least one more big fight about workflow/organizational issues, but I'd like to hold off on that until after we have a test repo that can actually demonstrate the issues that are relevant to that discussion. Cheers, Dirkjan [1] http://hg.python.org/pymigr/file/tip/todo.txt

3 2

maximum recursion depth exceeded in lib\pstats.py
by Adam Bielański Oct. 21, 2010

Oct. 21, 2010

Hi all, There's a bug in Python 2.6, module lib\pstats.py, line 150. I'm pretty sure, however that it also exists in other versions, as I don't think that profiler differs very much between them. Let me paste a little piece of surrounding code: class Stats: (....) def add(self, *arg_list): if not arg_list: return self if len(arg_list) > 1: self.add(*arg_list[1:]) #Line 150, the problem is right here! other = arg_list[0] (... do some processing with first item from arg_list ...) This is the code for adding profiling stats from multiple files (names are on arg_list) so that they can be displayed in cumulated and readable form. As you can see it chops off the first item from the list and then uses recursion to process the rest of it. It's no wonder then that when you profiled a little too many function calls, you're bound to see RuntimeError with 'maximum recursion depth exceeded' message when you try using this. IMO this should be done with iteration instead, like: def add(self, *arg_list): if not arg_list: return self for other in arg_list: #Preserved variable name only for sake of comparison with previous snippet (... do some processing with each item from arg_list, merging results in some rightful manner ...) Kind regards, Adam Bielanski.

2 1

[patch] fpconst for python3
by Neal Becker Oct. 20, 2010

Oct. 20, 2010

Where should I send this patch? diff -u fpconst-0.7.2/fpconst.py fpconst-0.7.2.new/fpconst.py --- fpconst-0.7.2/fpconst.py 2005-02-24 12:42:03.000000000 -0500 +++ fpconst-0.7.2.new/fpconst.py 2010-10-19 20:55:07.407765664 -0400 @@ -40,18 +40,18 @@ ident = "$Id: fpconst.py,v 1.16 2005/02/24 17:42:03 warnes Exp $" import struct, operator +from functools import reduce # check endianess -_big_endian = struct.pack('i',1)[0] != '\x01' - +_big_endian = struct.pack('i',1)[0] != 1 # and define appropriate constants if(_big_endian): - NaN = struct.unpack('d', '\x7F\xF8\x00\x00\x00\x00\x00\x00')[0] - PosInf = struct.unpack('d', '\x7F\xF0\x00\x00\x00\x00\x00\x00')[0] + NaN = struct.unpack('d', b'\x7F\xF8\x00\x00\x00\x00\x00\x00')[0] + PosInf = struct.unpack('d', b'\x7F\xF0\x00\x00\x00\x00\x00\x00')[0] NegInf = -PosInf else: - NaN = struct.unpack('d', '\x00\x00\x00\x00\x00\x00\xf8\xff')[0] - PosInf = struct.unpack('d', '\x00\x00\x00\x00\x00\x00\xf0\x7f')[0] + NaN = struct.unpack('d', b'\x00\x00\x00\x00\x00\x00\xf8\xff')[0] + PosInf = struct.unpack('d', b'\x00\x00\x00\x00\x00\x00\xf0\x7f')[0] NegInf = -PosInf def _double_as_bytes(dval):

5 4

Re: [Python-Dev] [Python-checkins] r85739 - in python/branches/issue4388/Lib/test: script_helper.py test_cmd_line.py
by Nick Coghlan Oct. 20, 2010

Oct. 20, 2010

On Wed, Oct 20, 2010 at 10:06 PM, victor.stinner <python-checkins(a)python.org> wrote: > Modified: python/branches/issue4388/Lib/test/test_cmd_line.py > ============================================================================== > --- python/branches/issue4388/Lib/test/test_cmd_line.py (original) > +++ python/branches/issue4388/Lib/test/test_cmd_line.py Wed Oct 20 14:06:46 2010 > @@ -103,8 +103,15 @@ > if sys.getfilesystemencoding() != 'ascii': > if test.support.verbose: > import locale > + env = os.environ.copy() > + for key in ('LC_ALL', 'LC_CTYPE', 'LANG'): > + try: > + del env[key] > + except KeyError: > + pass > print('locale encoding = %s, filesystem encoding = %s' > - % (locale.getpreferredencoding(), sys.getfilesystemencoding())) > + % (locale.getpreferredencoding(), sys.getfilesystemencoding()), > + env=env) > command = "assert(ord('\xe9') == 0xe9)" > assert_python_ok('-c', command) Isn't that "env=env" argument meant to be down in the call to assert_python_ok, not in the print call? (your next checkin didn't change this) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Python-Dev] r85724 - in python/branches/py3k: Lib/logging/__init__.py Misc/NEWS
by Georg Brandl Oct. 19, 2010

Oct. 19, 2010

Am 19.10.2010 17:26, schrieb vinay.sajip: > Author: vinay.sajip > Date: Tue Oct 19 17:26:24 2010 > New Revision: 85724 > > Log: > logging: Added _logRecordClass, getLogRecordClass, setLogRecordClass to increase flexibility of LogRecord creation. Will this be documented? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

1 0