Mailman 3 December 2009 - Python-Dev

Issue 3745 backwards incompatibility
by Karen Tracey 15 Dec '09

15 Dec '09

In testing some existing code with the 2.7 alpha release, I've run into: TypeError: Unicode-objects must be encoded before hashing when the existing code tries to pass unicode objects to hashlib.sha1 and hashlib.md5. This is, I believe, due to changes made for issue 3745: http://bugs.python.org/issue3745 The issue states the need to reject unencoded strings based on the fact that one backend implementation (openssl) refused to accept them while another (_sha256) assumed a utf-8 encoding. The thing is, I cannot observe any such difference using Python 2.5 or 2.6. Instead of what is shown in the ticket (which was done on a Python 3, I believe) I see, when I adjust the demo test to use Python 2 syntax for "unencoded strings": Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import _hashlib >>> _hashlib.openssl_sha256(u"\xff") Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in position 0: ordinal not in range(128) >>> import _sha256 >>> _sha256.sha256(u'\xff') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in position 0: ordinal not in range(128) >>> (Sample from Windows because that's the only place I can get import _sha256 to work. The Ubuntu Linux I tried behaves the same way as above for the _hashlib version, while it doesn't appear to have _sha256 as an option.) So from what I can see the behavior wasn't inconsistent from backend-to-backend in Python 2 but rather fell in line with what I'm familiar with: if you pass unicode to some code that only wants bytes, the unicode object will get encoded to a bytestring using the system default encoding. No problems if the data can in fact always be encoded using that encoding, the error above if the data can't be encoded. Changing these functions to now require the caller to do the encoding explicitly ahead of time strikes me as introducing an inconsistency. Plus it introduces a backwards incompatibility in Python 2.7. Is this really necessary? Karen

3 3

Request commit privileges for Stefan Krah
by Mark Dickinson 15 Dec '09

15 Dec '09

I'd like to request that Stefan Krah be granted commit privileges to the Python svn repository, for the sole purpose of working on a (yet to be created) py3k-decimal-in-c branch. Stefan has produced a C library 'libmpdec' implementing (fast!) arbitrary precision decimal arithmetic, together with a Python extension module 'Cdecimal' that provides full compatibility with Python's decimal module. The current version of the code can be found at: http://www.bytereef.org/libmpdec-download.html It's currently licensed under the GNU Affero General Public License, but Stefan has indicated that he's willing to relicense it to the PSF under a contributor agreement. We'd like to start work on a branch that integrates Stefan's work into the Python source tree, with the hope that this work could be considered for inclusion in py3k when it's completed. While I haven't yet done anything resembling a thorough code review of Stefan's code, the code I have looked at has seemed very high quality, and gives me a high degree of confidence in its correctness. Stefan's also done a truly ridiculous amount of correctness and compatibility testing of this code, and in the process discovered several corner case bugs in Python's decimal module. (I'm not yet asking for a decision on *whether* the decimal module should be replaced with a C version; that decision should be easier to make once Stefan's code is fully integrated into the branch, and we have some timings and benchmarks to show the performance benefit. There are also various other questions about a C version of decimal that will eventually need to addressed, but I don't want to distract from the main point of this message.) Mark

4 4

Microsoft contributor agreement received?
by Dino Viehland 15 Dec '09

15 Dec '09

I'm not sure the best place to verify this so I'm starting here. I'm told we finally faxed in our contributor agreement (to the number listed at http://www.python.org/psf/contrib/) about a week and a half ago. I'd just like to make sure that someone has received it. Is anyone here able to confirm that it went through fine?

3 2

Splitting something into two steps produces different behavior from doing it in one fell swoop in Python 2.6.2
by Roy Hyunjin Han 15 Dec '09

15 Dec '09

While debugging a network algorithm in Python 2.6.2, I encountered some strange behavior and was wondering whether it has to do with some sort of code optimization that Python does behind the scenes. ************ After initialization: defaultdict(<type 'set'>, {1: set([1])}) Popping and updating in two steps: defaultdict(<type 'set'>, {1: set([1])}) ************ After initialization: defaultdict(<type 'set'>, {1: set([1])}) Popping and updating in one step: defaultdict(<type 'set'>, {}) import collections print '************' x = collections.defaultdict(set) x[1].update([1]) print 'After initialization: %s' % x items = x.pop(1) x[1].update(items) print 'Popping and updating in two steps: %s' % x print '************' y = collections.defaultdict(set) y[1].update([1]) print 'After initialization: %s' % y y[1].update(y.pop(1)) print 'Popping and updating in one step: %s' % y

4 5

Unittest/doctest formatting differences in 2.7a1?
by Lennart Regebro 13 Dec '09

13 Dec '09

I just ran the tests for zope.testing on Python 2.7, and the results are not good. It seems that there are multiple minor difference in the output formatting of the testresults between 2.7 and previous 2.x versions. The result is that all the tests that test testing (zope.testing includes a testrunner) fails. Is these changes necessary? It's going to be hell to test any form of testrunner under both 2.6 and 2.7 if the formatting of test results have changed. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

14 35

Re: [Python-Dev] First draft of "sysconfig"
by Brett Cannon 13 Dec '09

13 Dec '09

That's what I meant. [From my phone] On Dec 12, 2009 4:32 PM, "Eric Smith" <eric(a)trueblade.com> wrote: Tarek Ziadé wrote: > > On Sat, Dec 12, 2009 at 10:35 PM, Brett Cannon < brett(a)python.org> wrote: > [.... I don't really understand the complete context, but I think you want str.format(), not string.Formatter here. Eric.

1 0

Backslash at end of raw string
by Rob Cliffe 12 Dec '09

12 Dec '09

Python (e.g. 2.5) does not accept a backslash as the LAST character of a raw string: >>> r"\" File "<stdin>", line 1 r"\" ^ Syntax Error: EOL while scanning single-quoted string. path = r'\MyDir\MySubDir\' # raises same error path = r'\MyDir\MySubDir' # no error Isn't this a bug? Rob Cliffe

2 1

Summary of Python tracker Issues
by Python tracker 12 Dec '09

12 Dec '09

ACTIVITY SUMMARY (12/04/09 - 12/11/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2526 open (+34) / 16775 closed (+11) / 19301 total (+45) Open issues with patches: 1007 Average duration of open issues: 694 days. Median duration of open issues: 450 days. Open Issues Breakdown open 2492 (+34) pending 33 ( +0) Issues Created Or Reopened (45) _______________________________ Int/Long: some tests are duplicate and error messages refer to " 12/04/09 CLOSED http://bugs.python.org/issue7435 created flox patch Define 'object with assignable attributes' 12/04/09 http://bugs.python.org/issue7436 created tjreedy OS X 2.6.4 installer fails on 10.3 with two corrupted file names 12/05/09 http://bugs.python.org/issue7437 created ned.deily Allow to use a part of subprocess module during building Python 12/05/09 http://bugs.python.org/issue7438 created Arfrever patch, needs review Bug or expected behavior? I cannot tell. 12/05/09 CLOSED http://bugs.python.org/issue7439 created LambertDW distutils shows incorrect Python version in MSI installers 12/05/09 http://bugs.python.org/issue7440 created MarioVilas Py3.1: Fatal Python Error: Py_Initialize...unknown encoding: chc 12/05/09 http://bugs.python.org/issue7441 created lieryan decimal.py: format failure with locale specifier 12/05/09 http://bugs.python.org/issue7442 created skrah test.support.unlink issue on Windows platform 12/05/09 http://bugs.python.org/issue7443 created asvetlov patch, needs review Allow for a default method in the JSON decoder 12/05/09 CLOSED http://bugs.python.org/issue7444 created dhgoldman urllib2 (and urllib) should raise error for incomplete response 12/05/09 http://bugs.python.org/issue7445 created gotgenes http.cookies.BaseCookie (and SimpleCookie) can't be load from di 12/05/09 CLOSED http://bugs.python.org/issue7446 created tcourbon Sum() doc and behavior mismatch 12/06/09 http://bugs.python.org/issue7447 created tjreedy when piping output between subprocesses some fd is left open blo 12/06/09 http://bugs.python.org/issue7448 created nosklo A number tests "crash" if python is compiled --without-threads 12/06/09 http://bugs.python.org/issue7449 created r.david.murray patch, easy document that os.chmod accepts an octal digit mode 12/07/09 http://bugs.python.org/issue7450 created clutchski improve json decoding performance 12/07/09 http://bugs.python.org/issue7451 created pitrou patch Invalid mnemonic 'fnstcw' 12/07/09 CLOSED http://bugs.python.org/issue7452 created srid HPUX 11.00: socketmodule.c -- error 1588: "AI_PASSIVE" undefined 12/08/09 http://bugs.python.org/issue7453 created srid Solaris SPARC: _multiprocessing.so: symbol sem_timedwait: refere 12/08/09 http://bugs.python.org/issue7454 created srid cPickle: stack underflow in load_pop() 12/08/09 http://bugs.python.org/issue7455 created haypo patch rfc822.Message.getheaders undocumented 12/08/09 CLOSED http://bugs.python.org/issue7456 created tarek Adding a read_pkg_file to DistributionMetadata 12/08/09 CLOSED http://bugs.python.org/issue7457 created tarek crash in str.rfind() with an invalid start value 12/08/09 http://bugs.python.org/issue7458 created haypo patch Magic word incorrect in Python 3 12/08/09 CLOSED http://bugs.python.org/issue7459 created Arfrever patch extended slicing not sufficiently covered in docs 12/08/09 http://bugs.python.org/issue7460 created donlorenzo os.popen() objects don't support the context manager protocol 12/08/09 CLOSED http://bugs.python.org/issue7461 created pitrou patch Implement fastsearch algorithm for rfind/rindex 12/08/09 http://bugs.python.org/issue7462 created flox patch PyDateTime_IMPORT() causes compiler warnings 12/09/09 http://bugs.python.org/issue7463 created murrayc circular reference in HTTPResponse by urllib2 12/09/09 http://bugs.python.org/issue7464 created krisvale patch Call to another class's constructor in unittest.TestCase.setUp r 12/10/09 CLOSED http://bugs.python.org/issue7465 created awaltman Segmentation fault after about 20 seconds on lenovo T500 12/10/09 http://bugs.python.org/issue7466 created LambertDW patch The zipfile module does not check files' CRCs, including in ZipF 12/10/09 http://bugs.python.org/issue7467 created dougturk PyErr_Format documentation doesn't mention all format codes 12/10/09 http://bugs.python.org/issue7468 created ronaldoussoren Design and History FAQ entry on Floating Point does not mention 12/10/09 http://bugs.python.org/issue7469 created r.david.murray easy logger.StreamHandler emit encoding fallback is wrong 12/10/09 CLOSED http://bugs.python.org/issue7470 created cboos patch GZipFile.readline too slow 12/10/09 http://bugs.python.org/issue7471 created asnakelover email.encoders.encode_7or8bit(): typo "iso-2202". "iso-2022" is 12/10/09 http://bugs.python.org/issue7472 created ynkdir Compile error when building a 3-way universal framework when a 2 12/10/09 http://bugs.python.org/issue7473 created prniii multiprocessing.managers.SyncManager managed object creation fai 12/10/09 http://bugs.python.org/issue7474 created etzool codecs missing: base64 bz2 hex zlib ... 12/11/09 http://bugs.python.org/issue7475 reopened flox patch pipes.quote does not handle zero-length args correctly 12/10/09 http://bugs.python.org/issue7476 created jjwiseman kqueue timers don't work properly 12/11/09 http://bugs.python.org/issue7477 created hasan _sqlite3 doesn't catch PyDict_SetItem error 12/11/09 http://bugs.python.org/issue7478 created haypo patch os.lchmod is not present 12/11/09 http://bugs.python.org/issue7479 created bluegeek Issues Now Closed (24) ______________________ Update to latest ElementTree in Python 2.7 818 days http://bugs.python.org/issue1143 pitrou patch Can't pickle partial functions 483 days http://bugs.python.org/issue1398 alexandre.vassalotti Remove cmp parameter to list.sort() and builtin.sorted() 699 days http://bugs.python.org/issue1771 gvanrossum meaningful whitespace can be lost in rfc822_escape 682 days http://bugs.python.org/issue1923 tarek easy add ftp-tls support to ftplib - RFC 4217 667 days http://bugs.python.org/issue2054 pitrou patch I/O operation on closed socket: improve the error message 339 days http://bugs.python.org/issue4853 haypo patch cPickle - module object has no attribute 262 days http://bugs.python.org/issue5509 rb Implement a way to change the python process name 249 days http://bugs.python.org/issue5672 loewis patch _json crash on scanner/encoder initialization error 75 days http://bugs.python.org/issue6986 pitrou patch Problems running threaded Tkinter program under OS X IDLE 46 days http://bugs.python.org/issue7190 ronaldoussoren StringIO and with statement 1 days http://bugs.python.org/issue7426 alexandre.vassalotti Int/Long: some tests are duplicate and error messages refer to " 1 days http://bugs.python.org/issue7435 mark.dickinson patch Bug or expected behavior? I cannot tell. 1 days http://bugs.python.org/issue7439 brett.cannon Allow for a default method in the JSON decoder 0 days http://bugs.python.org/issue7444 bob.ippolito http.cookies.BaseCookie (and SimpleCookie) can't be load from di 0 days http://bugs.python.org/issue7446 amaury.forgeotdarc Invalid mnemonic 'fnstcw' 4 days http://bugs.python.org/issue7452 ronaldoussoren rfc822.Message.getheaders undocumented 0 days http://bugs.python.org/issue7456 tarek Adding a read_pkg_file to DistributionMetadata 0 days http://bugs.python.org/issue7457 tarek Magic word incorrect in Python 3 1 days http://bugs.python.org/issue7459 benjamin.peterson patch os.popen() objects don't support the context manager protocol 0 days http://bugs.python.org/issue7461 pitrou patch Call to another class's constructor in unittest.TestCase.setUp r 0 days http://bugs.python.org/issue7465 r.david.murray logger.StreamHandler emit encoding fallback is wrong 1 days http://bugs.python.org/issue7470 vinay.sajip patch add support for thread function result storage 1438 days http://bugs.python.org/issue1395552 mykhal patch sys.getfilesystemencoding 1293 days http://bugs.python.org/issue1495089 r.david.murray Top Issues Most Discussed (10) ______________________________ 27 IMAP4_SSL spin because of SSLSocket.suppress_ragged_eofs 219 days open http://bugs.python.org/issue5949 18 Invalid mnemonic 'fnstcw' 4 days closed http://bugs.python.org/issue7452 12 codecs missing: base64 bz2 hex zlib ... 0 days open http://bugs.python.org/issue7475 12 json C serializer performance tied to structure depth on some s 136 days open http://bugs.python.org/issue6594 11 crash in str.rfind() with an invalid start value 3 days open http://bugs.python.org/issue7458 9 Segmentation fault after about 20 seconds on lenovo T500 2 days open http://bugs.python.org/issue7466 9 Implement a way to change the python process name 249 days closed http://bugs.python.org/issue5672 9 csv module: add header row to DictWriter 1220 days open http://bugs.python.org/issue1537721 8 GZipFile.readline too slow 1 days open http://bugs.python.org/issue7471 8 Remove cmp parameter to list.sort() and builtin.sorted() 699 days closed http://bugs.python.org/issue1771

1 0

Re: [Python-Dev] Proposing PEP 386 for addition
by Michael Mysinger 10 Dec '09

10 Dec '09

More English language fixes: -In Python there are no real restriction yet on how a project should +In Python there are no real restrictions yet on how a project should -Furthermore, this will make OS packagers work easier when repackaging standards -compliant distributions, as of now it can be difficult to decide how two +Furthermore, this will make OS packagers' work easier when repackaging standards +compliant distributions, because as of now it can be difficult to decide how two -to support many or all existing versioning schemas. +to support all or even most of existing versioning schemas. -reasons and we cannot expect that to change. +reasons that we cannot expect to change. -version schemas and is a preferable alternative than supporting +version schemas and is a preferable alternative to supporting -there should be possible to express more than one versioning level +it should be possible to express more than one versioning level -The problem with this is that while it allows expressing requisite any -nesting level it doesn't allow to express special meaning versions -(pre and post-releases as well as development versions), as expressed in +The problem with this is that while it allows expressing any +nesting level it doesn't allow giving special meaning to versions +(pre- and post-releases as well as development versions) as expressed in -Also, notice that Distutils version classes are not really used in the +Also, note that Distutils version classes are not really used in the -which does not enforce any rule for the version, but +which does not enforce any rules for the version, but -Setuptools function is quite spread because it's used by tools +Setuptools function is quite widespread because it's used by tools -post-releases -- which apparently is used by a number of projects +post-releases -- which apparently are used by a number of projects This one is particularly critical to your intended meaning as I read it: -before a <tt class="docutils literal"><span class="pre">.post345</span></tt> marker. +before a <tt class="docutils literal"><span class="pre">.post456</span></tt> marker. Technical question: I don't know what notation this versioning schema was trying for, especially in regards to what the +'s mean: N.N[.N]+[abc]N[.N]+[.postN+][.devN+] Am I missing something here? You could maybe explain what the pluses mean in the PEP, and why some are inside the [] and others are outside. Or a regular expression version like this might be more specific. N.N(.N)?([abc]N(.N)?)?(.postN)?(.devN)? Or an linux help version like this may be more readible. N.N[.N][{abc}N[.N]][.postN][.devN] Cheers, Michael Mysinger (longtime python-dev lurker)

3 5

recursive closures - reference leak
by Kristján Valur Jónsson 10 Dec '09

10 Dec '09

Hello there. Consider this code: def factorial(n): def helper(n): if n: return n*helper(n-1) else: return 1 return helper(n) Ok, this is a bit contrived, but there are other valid reasons for calling a closure recursively. The problem with this is that once you have called factorial() once, you end up with a recursive cycle. "factorial" has become a cell object, referencing the "helper" function, which again refers to the outer cell object. This requires "gc" to clean up. Also, it is entirely non-obvious. the problem becomes worse if the inner function also refers to some large, temporary variable, since it will get caught up in the reference loop. A workaround is this. Replace the final line with: try: return helper(n) finally: helper = None This clears the cell before function exit. Note that it is not possible to "del" it, you get a syntax error for that. Now, the reason I'm posting this here, and not to python-lang, is this: Is there a way to implement this differently so that this pattern doesn't result in a cycle? Possible ideas would be: 1) clear "helper" on function exit. Not good if the internal function is exported, though. 2) Use have 'helper' store a weakref to the cell object Cheers, Kristján

7 12