Mailman 3 March 2011 - Python-Dev

About raising NotPortableWarning for CPython specific code
by Laura Creighton 12 Mar '11

12 Mar '11

For those of you not at the Language Summit at PyCON the day before yesterday, there was talk of identifying non-portable behaviour, such as relying on CPython's reference counting garbage collector to close files for you as soon as they become unreachable. And then warning about them. We have a real live user who has a large code base that relies on the CPython behaviour that an object's __radd__ method will take precedence over a list's inplace add behaviour. The thread with the whole gory details begins here: http://codespeak.net/pipermail/pypy-dev/2011q1/006958.html My inclination is to declare code that relies on this as broken, rather than patch every wretched container type in PyPy. Can this become blessed as a 'you shouldn't have done this'? Laura

7 10

another message to release
by R. David Murray 12 Mar '11

12 Mar '11

I believe the tracker sent an error message to python-dev as a result of a failed hook execution. If someone with the power would release that message so we can see what the error was, I'd appreciate it :) --David

1 0

Callable, non-descriptor class attributes.
by Thomas Wouters 12 Mar '11

12 Mar '11

One of the things brought up at the language summit (and I believe at the VM summit, although I wasn't there) was the unpredictable behaviour of callables turning into methods when they're class attributes. Specifically, things that are CFunctions in CPython (builtin functions, which are not descriptors and do not turn into methods) but could be plain Python functions in other Python implementations. (Or even turn into a plain function in CPython itself, if the other-implementation argument isn't persuasive enough for you.) To see the kind of thing that would be (potentially) problematic, see for example Lib/encodings/ascii.py: import codecs ### Codec APIs class Codec(codecs.Codec): # Note: Binding these as C functions will result in the class not # converting them to methods. This is intended. encode = codecs.ascii_encode decode = codecs.ascii_decode (I hope the irony of a two-line comment explaining the situation, as opposed to the self-documenting nature of staticmethod, isn't lost on anyone :) Obviously, if the codecs.ascii_encode and codecs.ascii_decode functions are replaced *for whatever reason* by Python functions, this will break. Because these kinds of class attributes are often used for dependency injection, the potentially problematic cases are even more common in tests -- the io testsuite, for example, specialcases pyio.open (in two different ways, no less) but omits the same for io.open for no apparent reason (other than that right now, in CPython, it isn't needed.) At the summit we discussed some potential improvements here: 1. Make staticmethod a callable object directly (it isn't, currently) and apply it to any Python function that replaces a (in CPython) CFunction. The change to staticmethod may be a good idea regardless, but the policy of making other implementations comply to this quirk in CPython seems (to me) like unnecessary descrimination[*]. 2. Make CFunctions turn into methods in CPython (after a period of warning about the impending change, obviously.) The actual *usecase* for this is hard to envision (I think Guido thought someone might do 'class str(str): len = len', which doesn't seem like that big an improvement :) so it would be a feature for symmetry's sake only. 3. Produce a CompatibilityWarning making sure that the programmer knows that this particular thing might not do the same thing in other Python implementations *or versions*. 4. Make it an error to have a callable class attribute that isn't a descriptor (although maybe we only discussed this one in my head.) To gauge the size of the problem I've implemented solution #3: I've uploaded a patch that adds the warning (currently as a DeprecationWarning) and fixes a bunch of the issues (not yet all of them): http://bugs.python.org/issue11470. The patch currently warns for callable, non-descriptor, *non-class* class attributes. The class check is necessary because injected classes and -- more popularly, in some ORMs -- nested classes would otherwise produce the warning. (Considering replacing a class with a function has lots of other effects, like isinstance checks failing, I don't think warning for that case is necessary anyway.) An alternate strategy would be to warn only about uses of CFunctions in these cases, but this misses out on a whole slew of potential issues (e.g. a module global that is currently a bound instance method.) If you look at the patch you'll notice it also fixes a couple of "dubious" cases -- usually in a module's own testsuite -- where one could certainly argue that the test is allowed to make the assumptions it's making. I don't think writing it explicitly future-proof detracts from the readability, stability and maintainability in most of these cases, although I should point out that multiprocessing's testsuite produces a bunch more warnings and isn't as easy to fix because of its use of locals().update() in a class statement. (Yes, really.) With regards to changing staticmethod, in an earlier iteration of the patch I modified staticmethod to be directly callable *and* to return self in its tp_descr_get, and this had a side-effect I hadn't considered: code that used a staticmethod-wrapped object in any way other than calling it, broke. Right now staticmethod is implemented as a generic container to inhibit descriptor behaviour, in spite of its name, so it can be used to protect any kind of descriptor in a class attribute. I'm not sure how common that use is, and whether we should try to provide a different wrapper for those cases. Unfortunately, without changing staticmethod to return itself, rather than the callable it's wrapping, in tp_descr_get, code like this: class C: open = builtin.open class D: open = C.open ... would still produce different behaviour when builtin.open changes from a CPython CFunction to a Python function wrapped in a staticmethod: C.open would produce the Python function, and D.open would then become a method after all. (This is why _pyio has OpenWrapper instead, I guess.) Considering all this, I personally think we should go for solution #3 or #4 (with a long migration path through #3, of course): add a warning like the one I implemented, add plenty of documentation explaining the issue (perhaps even pointing to a PEP in the warning,) and fix everything in the stdlib that produces the warning, even if it's hard to see how that particular case could become a problem. The warning should probably be of the same kind as DeprecationWarning to prevent end-users from being confused by it. Changing staticmethod to be directly callable may still be a good idea, but changing what its descr_get does is not. Making CFunctions turn into bound methods is something that seems of little value, but could be considered at a later point (after the warning has had time to flush out most of the problematic cases.) [*]-sorry-couldn't-resist'ly y'rs, -- Thomas Wouters <thomas(a)python.org> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

6 10

Failed issue tracker submission
by Python tracker 12 Mar '11

12 Mar '11

You are not permitted to edit property stage of class issue. Return-Path: <python-dev(a)python.org> X-Original-To: report(a)bugs.python.org Delivered-To: roundup+tracker(a)psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [82.94.164.166]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id E18F41CCEF for <report(a)bugs.python.org>; Sun, 13 Mar 2011 00:31:57 +0100 (CET) Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id A71FEEE9D3 for <report(a)bugs.python.org>; Sun, 13 Mar 2011 00:31:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1299972717; bh=Ibp6eWhVeNrklL/k3hji/5Ii1lOezo53S23METjDw1I=; h=Date:Message-Id:Content-Type:MIME-Version: Content-Transfer-Encoding:From:To:Subject; b=SFvsALnuTMjXJDF/BlVNobKujYGwQR3VvGbWp+BoRydAeuxb4zXAnjJI1eZl5owoM /Qj50uWq8D/7SlkGBPGrzYamqa2N4j0xyaWH4UZIhXYFJuO5aywdJY3/sNI3WlGRyC X0/K3wAlalv9IHS1LddFM+0jQau87LgY/CqpA8S8= Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 13 Mar 2011 00:31:57 +0100 Received: from dinsdale.python.org (svn.python.org [IPv6:2001:888:2000:d::a4]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.python.org (Postfix) with ESMTPS for <report(a)bugs.python.org>; Sun, 13 Mar 2011 00:31:57 +0100 (CET) Received: from localhost ([127.0.0.1] helo=dinsdale.python.org ident=hg) by dinsdale.python.org with esmtp (Exim 4.72) (envelope-from <python-dev(a)python.org>) id 1PyYHt-000633-JZ for report(a)bugs.python.org; Sun, 13 Mar 2011 00:31:57 +0100 Date: Sun, 13 Mar 2011 00:31:57 +0100 Message-Id: <E1PyYHt-000633-JZ(a)dinsdale.python.org> Content-Type: text/plain; charset="utf8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 From: python-dev(a)python.org To: report(a)bugs.python.org Subject: [issue2771] [status=closed; resolution=fixed; stage=committed/rejected] TmV3IGNoYW5nZXNldCA3N2Q0Nzg2YWFlMjkgYnkgQW50b2luZSBQaXRyb3UgaW4gYnJhbmNoICdk ZWZhdWx0JzoKY2xvc2VzICMyNzcxCmh0dHA6Ly9oZy5weXRob24ub3JnL3Rlc3QvcmV2Lzc3ZDQ3 ODZhYWUyOQo=

1 0

Failed issue tracker submission
by Python tracker 12 Mar '11

12 Mar '11

You are not permitted to edit property stage of class issue. Return-Path: <python-dev(a)python.org> X-Original-To: report(a)bugs.python.org Delivered-To: roundup+tracker(a)psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [82.94.164.166]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id D03D11CCEF for <report(a)bugs.python.org>; Sun, 13 Mar 2011 00:30:54 +0100 (CET) Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 91C45EEA3B for <report(a)bugs.python.org>; Sun, 13 Mar 2011 00:30:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1299972654; bh=YEfRd87zUD9hUNRPQ0gW/hovinbL3pZoGbabcdfUpGU=; h=Date:Message-Id:Content-Type:MIME-Version: Content-Transfer-Encoding:From:To:Subject; b=Lne+0Te3eUe78ACgJ/bd+w5w1kpdpCqG/g9KvpoyrohiMrDY0TU8ejIMhpYO8hx/W OtEH1COtBcujwb6w0O8pGyDLovo8D4+GFKJ+S2ziOn+kYEjyHYfiJZIbEj+BPswhXq f1LkEzapQxWETAOMqoj6xFbZ5S/+9zlTYqgtQSTU= Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 13 Mar 2011 00:30:54 +0100 Received: from dinsdale.python.org (svn.python.org [IPv6:2001:888:2000:d::a4]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.python.org (Postfix) with ESMTPS for <report(a)bugs.python.org>; Sun, 13 Mar 2011 00:30:54 +0100 (CET) Received: from localhost ([127.0.0.1] helo=dinsdale.python.org ident=hg) by dinsdale.python.org with esmtp (Exim 4.72) (envelope-from <python-dev(a)python.org>) id 1PyYGs-0005xP-GA for report(a)bugs.python.org; Sun, 13 Mar 2011 00:30:54 +0100 Date: Sun, 13 Mar 2011 00:30:54 +0100 Message-Id: <E1PyYGs-0005xP-GA(a)dinsdale.python.org> Content-Type: text/plain; charset="utf8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 From: python-dev(a)python.org To: report(a)bugs.python.org Subject: [issue2771] [status=closed; resolution=fixed; stage=committed/rejected] TmV3IGNoYW5nZXNldCBkMGY5N2QyN2I4ZjggYnkgQW50b2luZSBQaXRyb3UgaW4gYnJhbmNoICdk ZWZhdWx0JzoKY2xvc2VzICMyNzcxCmh0dHA6Ly9oZy5weXRob24ub3JnL3Rlc3QvcmV2L2QwZjk3 ZDI3YjhmOAo=

1 0

forward-porting from 3.1 to 3.2 to 3.3
by Eli Bendersky 12 Mar '11

12 Mar '11

The devguide's recommendation is to "forward-port" changes withing a major release line, i.e. if I need something in all 3.[123], then start with 3.1 and forward-port (by "hg merge <branch>") to 3.2 and then 3.3 Just to clarify - does this mean that all changesets that are applied to 3.2 eventually get to 3.3 as well? Since we just do "hg merge 3.2" in the 3.3 clone, I guess this must be true. But then what happens if there's a change in 3.2 that shouldn't get into 3.3? (say a bug fix for a feature that has disappeared)? Eli

5 9

Failed issue tracker submission
by Python tracker 12 Mar '11

12 Mar '11

You are not permitted to edit property stage of class issue. Return-Path: <python-dev(a)python.org> X-Original-To: report(a)bugs.python.org Delivered-To: roundup+tracker(a)psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [82.94.164.166]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id 075991CE7D for <report(a)bugs.python.org>; Sun, 13 Mar 2011 00:11:29 +0100 (CET) Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id C23D6EEB44 for <report(a)bugs.python.org>; Sun, 13 Mar 2011 00:11:28 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1299971488; bh=ZO2sKQ6jd1TrEYAI+tiEvF5y9+PgMMuHX2WWQDp0bnw=; h=Date:Message-Id:Content-Type:MIME-Version: Content-Transfer-Encoding:From:To:Subject; b=qcbzdeBjE9gfe3Ndft/j0SP7zLwuFGa++K1MsuhJ1Uor8zkuv61hyDeKzW0ZY6OB+ Go+whgYoMvp0qWvxE4kBNMfV5pVIlOLeTH3wcnL3jDmTdIKd60qfFKMdB3LpSzYPo3 fCY2j5P3m3chXfY9YTjJGJbK4ybtI0uC4kpA9xJU= Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 13 Mar 2011 00:11:28 +0100 Received: from dinsdale.python.org (svn.python.org [IPv6:2001:888:2000:d::a4]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.python.org (Postfix) with ESMTPS for <report(a)bugs.python.org>; Sun, 13 Mar 2011 00:11:28 +0100 (CET) Received: from localhost ([127.0.0.1] helo=dinsdale.python.org ident=hg) by dinsdale.python.org with esmtp (Exim 4.72) (envelope-from <python-dev(a)python.org>) id 1PyXy4-0003JY-Jx for report(a)bugs.python.org; Sun, 13 Mar 2011 00:11:28 +0100 Date: Sun, 13 Mar 2011 00:11:28 +0100 Message-Id: <E1PyXy4-0003JY-Jx(a)dinsdale.python.org> Content-Type: text/plain; charset="utf8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 From: python-dev(a)python.org To: report(a)bugs.python.org Subject: [issue11312] [status=closed; resolution=fixed; stage=committed/rejected] TmV3IGNoYW5nZXNldCBjMDE3Njk1YWNmMTkgYnkgQmVuamFtaW4gUGV0ZXJzb24gaW4gYnJhbmNo ICcyLjcnOgpjbGFyaWZ5IGNvbmRpdGlvbiBmb3IgcmVhZGxpbmUgdG8gcmV0dXJuIEVPRiAoY2xv c2VzICMxMTMxMikKaHR0cDovL2hnLnB5dGhvbi5vcmcvY3B5dGhvbi9yZXYvYzAxNzY5NWFjZjE5 Cg==

1 0

Introductions
by Ross Lagerwall 12 Mar '11

12 Mar '11

Hi, I have been offered commit rights for Python after making a few patches on subprocess and the os module. Antoine suggested that I should introduce myself on the python-dev list so here we go: I am a student from South Africa and decided to do some work on Python in my spare time. I think I chose the Python project because it definitely seems to be one of the friendliest projects around and the easiest to get started with - the dev guide helps a lot! Also, with the big standard library, there is place for people with all different levels of experience to contribute. Cheers Ross

4 3

Failed issue tracker submission
by Python tracker 12 Mar '11

12 Mar '11

You are not permitted to edit property stage of class issue. Return-Path: <python-dev(a)python.org> X-Original-To: report(a)bugs.python.org Delivered-To: roundup+tracker(a)psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [82.94.164.166]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id 2E3A11CD34 for <report(a)bugs.python.org>; Sat, 12 Mar 2011 18:58:26 +0100 (CET) Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id E301AEE981 for <report(a)bugs.python.org>; Sat, 12 Mar 2011 18:58:25 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1299952705; bh=qYZ+1soa1lYFkl/MBhE0kMaEkm6EP9xi0HD2EZzhmNM=; h=Date:Message-Id:Content-Type:MIME-Version: Content-Transfer-Encoding:From:To:Subject; b=wHQXREQx40bwINh4NztNuqvbAdeCrdv15TW2zjpZfVksU2wtp/dnIR2PMJhst3tak Awt/95dyrM5Q/9bbYgphd8Zs0Ldnhr8halDZNoPWv/cFbpDyAn4SUQUNBZzc02MGjC gxzVgik0GgKUPQrnX5YwTnofVxzLkEE2dqzDMP2U= Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 12 Mar 2011 18:58:25 +0100 Received: from dinsdale.python.org (svn.python.org [IPv6:2001:888:2000:d::a4]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.python.org (Postfix) with ESMTPS for <report(a)bugs.python.org>; Sat, 12 Mar 2011 18:58:25 +0100 (CET) Received: from localhost ([127.0.0.1] helo=dinsdale.python.org ident=hg) by dinsdale.python.org with esmtp (Exim 4.72) (envelope-from <python-dev(a)python.org>) id 1PyT56-0007cT-FG for report(a)bugs.python.org; Sat, 12 Mar 2011 18:58:24 +0100 Date: Sat, 12 Mar 2011 18:58:24 +0100 Message-Id: <E1PyT56-0007cT-FG(a)dinsdale.python.org> Content-Type: text/plain; charset="utf8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 From: python-dev(a)python.org To: report(a)bugs.python.org Subject: [issue11475] [status=closed; resolution=fixed; stage=committed/rejected] TmV3IGNoYW5nZXNldCA3ZmQwNjc3M2MzYzIgYnkgQmVuamFtaW4gUGV0ZXJzb24gaW4gYnJhbmNo ICczLjEnOgp0cnVuYyAtPiBtYXRoLnRydW5jIChjbG9zZXMgIzExNDc1KQpodHRwOi8vaGcucHl0 aG9uLm9yZy9jcHl0aG9uL3Jldi83ZmQwNjc3M2MzYzIKCgpOZXcgY2hhbmdlc2V0IDBlYjIzZGVk YjAzYyBieSBCZW5qYW1pbiBQZXRlcnNvbiBpbiBicmFuY2ggJzIuNyc6CnRydW5jIC0+IG1hdGgu dHJ1bmMgKGNsb3NlcyAjMTE0NzUpCmh0dHA6Ly9oZy5weXRob24ub3JnL2NweXRob24vcmV2LzBl YjIzZGVkYjAzYwo=

1 0

Builtin open() too slow
by Lukas Lueg 12 Mar '11

12 Mar '11

Hi, i've a storage engine that stores a lot of files (e.g. > 10.000) in one path. Running the code under cProfile, I found that with a total CPU-time of 1,118 seconds, 121 seconds are spent in 27.013 calls to open(). The number of calls is not the problem; however I find it *very* discomforting that Python spends about 2 minutes out of 18 minutes of cpu time just to get a file-handle after which it can spend some other time to read from them. May this be a problem with the way Python 2.7 gets filehandles from the OS or is it a problem with large directories itself? Best regards Lukas

3 2