Mailman 3 June 2011 - Python-Dev

[RELEASE] Python 2.7.2
by Benjamin Peterson 12 Jun '11

12 Jun '11

On behalf of the Python development team, I'm rosy to announce the immediate availability of Python 2.7.2. Since the release candidate 2 weeks ago, there have been 2 changes: 1. pyexpat.__version__ has be changed to be the Python version. 2. A regression from 3.1.3 in the handling of comments in the netrc module has been resolved. (see issue #12009). 2.7.2 is the second in bugfix release for the Python 2.7 series. 2.7 is the last major verison of the 2.x line and will be receiving only bug fixes while new feature development focuses on 3.x. The 2.7 series includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, set literals, dictionary views, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, a new sysconfig module, auto-numbering of fields in the str/unicode format method, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7.2 visit: http://www.python.org/download/releases/2.7.1/ The 2.7.2 changelog is at: http://hg.python.org/cpython/raw-file/eb3c9b74884c/Misc/NEWS 2.7 documentation can be found at: http://docs.python.org/2.7/ This is a production release, please report any bugs to http://bugs.python.org/ Enjoy and for those in the northern hemisphere, have a nice summer! -- Benjamin Peterson Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7.2's contributors)

2 1

Re: [Python-Dev] [Python-checkins] cpython (2.7): allow "fake" filenames in findsource (closes #9284)
by Nick Coghlan 12 Jun '11

12 Jun '11

On Sun, Jun 12, 2011 at 6:56 AM, benjamin.peterson <python-checkins(a)python.org> wrote: > summary: > allow "fake" filenames in findsource (closes #9284) > > This allows findsource() to work in doctests. > > A patch from Dirkjan Ochtman. Either this exception should be mentioned in the inspect.getsource() documentation or else doctest should be monkeypatching inspect as well as linecache. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 2

Re: [Python-Dev] [Python-checkins] cpython (3.1): onto 3.1.5
by Benjamin Peterson 11 Jun '11

11 Jun '11

2011/6/11 Terry Reedy <tjreedy(a)udel.edu>: > >> +What's New in Python 3.1.5? >> +=========================== >> + >> +*Release date: XXXX-XX-XX* >> + >> +Core and Builtins >> +----------------- >> + >> +Library >> +------- >> + >> + > > I presume that only security patches should be added. Indeed. -- Regards, Benjamin

1 0

Python jails
by Sam Edwards 11 Jun '11

11 Jun '11

Hello! This is my first posting to the python-dev list, so please forgive me if I violate any unspoken etiquette here. :) I was looking at Python 2.x's f_restricted frame flag (or, rather, the numerous ways around it) and noticed that most (all?) of the attacks to escape restricted execution involved the attacker grabbing something he wasn't supposed to have. IMO, Python's extensive introspection features make that a losing battle, since it's simply too easy to forget to blacklist something and the attacker finding it. Not only that, even with a perfect vacuum-sealed jail, an attacker can still bring down the interpreter by exhausting memory or consuming excess CPU. I think I might have a way of securely sealing-in untrusted code. It's a fairly nascent idea, though, and I haven't worked out all of the details yet, so I'm posting what I have so far for feedback and for others to try to poke holes in it. Absolutely nothing here is final. I'm just framing out what I generally had in mind. Obviously, it will need to be adjusted to be consistent with "the Python way" - my hope is that this can become a PEP. :) >>> # It all starts with the introduction of a new type, called a jail. (I haven't yet worked out whether it should be a builtin type, ... # or a module.) Unjailed code can create jails, which will run the untrusted code and keep strict limits on it. ... >>> j = jail() >>> dir(j) ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'acquire', 'getcpulimit', 'getcpuusage', 'getmemorylimit', 'getmemoryusage', 'gettimelimit', 'gettimeusage', 'release', 'setcpulimit', 'setmemorylimit', 'settimelimit'] >>> # The jail monitors three things: Memory (in bytes), real time (in seconds), and CPU time (also in seconds) ... # and it also allows you to impose limits on them. If any limit is non-zero, code in that jail may not exceed its limit. ... # Exceeding a memory limit will result in a MemoryError. I haven't decided what CPU/real time limits should raise. ... # The other two calls are "acquire" and "release," which allow you to seal (any) objects inside the jail, or bust them # out. Objects inside the jail (i.e. created by code in that jail) contribute their __sizeof__() to the j.getmemoryusage() ... >>> def stealPasswd(): ... return open('/etc/passwd','r').read() ... >>> j.acquire(stealPasswd) >>> j.getmemoryusage() # The stealPasswd function, its code, etc. are now locked away within the jail. 375 >>> stealPasswd() Traceback (most recent call last): File "<stdin>", line 1, in <module> JailError: tried to access an object outside of the jail The object in question is, of course, 'open'. Unlike the f_restricted model, the jail was freely able to grab the open() function, but was absolutely unable to touch it: It can't call it, set/get/delete attributes/items, or pass it as an argument to any functions. There are three criteria that determine whether an object can be accessed: a. The code accessing the object is not within a jail; or b. The object belongs to the same jail as the code accessing the object; or c. The object has an __access__ function, and theObject.__access__(theJail) returns True. For the jail to be able to access 'open', it needs to be given access explicitly. I haven't quite decided how this should work, but I had in mind the creation of a "guard" (essentially a proxy) that allows the jail to access the object. It belongs to the same jail as the guarded object (and is therefore impossible to create within a jail unless the guarded object belongs to the same jail), has a list of jails (or None for 'any') that the guard will allow to __access__ it (the guard is immutable, so jails can't mess with it even though they can access it), and what the guard will allow though it (read-write, read-only, call-within-jail, call-outside-jail). I have a couple remaining issues that I haven't quite sussed out: * How exactly do guards work? I had in mind a system of proxies (memory usage is a concern, especially in memory-limited jails - maybe allow __access__ to return specific modes of access rather than all-or-nothing?) that recursively return more guards after operations. (e.g., if I have a guard allowing read+call on sys, sys.stdout would return another guard allowing read+call on sys.stdout, likewise for sys.stdout.write) * How are objects sealed in the jail? j.acquire can lead to serious problems with lots of references getting recursively sealed in. Maybe disallow sealing in anything but code objects, or allow explicitly running code within a jail like j.execute(code, globals(), locals()), which works fine since any objects created by jailed code are also jailed. * How do imports work? Should __import__ be modified so that when a jail invokes it, the import runs normally (unjailed), and then returns the module with a special guard that allows read-only+call-within, but not on builtins? This has a nice advantage, since jailed code can import e.g. socket, and maybe even create a socket, but won't be able to do sock.connect(...), since socket.connect (which is running with jailed permissions) can't touch the builtin _socket module. * Is obj.__access__(j) the best way to decide access? It doesn't allow programmers much freedom to customize the jail policy since they can't modify __access__ for builtins. Maybe the jail should have the first chance (such as j.__access__(obj)), which allows programmers to subclass the jail, and the jail can fallback to obj.__access__(j) * How does Python keep track of what jail each frame is in? Maybe each frame can have a frame.f_jail, which references the jail object restricting that frame (or None for unjailed code) - frames' jails default to the jail holding the code object, or can be explicitly overridden (as in j.execute(code, globals(), locals())) * When are jails switched? Obviously, jailed code called from unjailed code (or even from other unjailed code) should be executed in the callee jail... But if a jailed caller is calling unjailed code, does the jail follow, or does the unjailed code run in an unjailed frame? How do programmers specify that? ...that's pretty much my two (erm, twenty) cents on the matter. Again, any feedback/adversarial reasoning you guys can offer is much appreciated.

5 5

Re: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to
by Georg Brandl 10 Jun '11

10 Jun '11

On 06/07/11 05:20, brett.cannon wrote: > http://hg.python.org/cpython/rev/fc282e375703 > changeset: 70695:fc282e375703 > user: Brett Cannon <brett(a)python.org> > date: Mon Jun 06 20:20:36 2011 -0700 > summary: > Remove some extraneous parentheses and swap the comparison order to > prevent accidental assignment. > > Silences a warning from LLVM/clang 2.9. Swapping the comparison order here seems a bit inconsistent to me. There are lots of others around (e.g. "len == 0" in the patch context below). Why is this one so special? I think that another developer even got told off once for these kinds of comparisons. I hope the Clang warning is only about the parentheses. Georg > files: > Modules/arraymodule.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c > --- a/Modules/arraymodule.c > +++ b/Modules/arraymodule.c > @@ -2091,7 +2091,7 @@ > if (len == 0) { > return PyUnicode_FromFormat("array('%c')", (int)typecode); > } > - if ((typecode == 'u')) > + if ('u' == typecode) > v = array_tounicode(a, NULL); > else > v = array_tolist(a, NULL);

12 14

Summary of Python tracker Issues
by Python tracker 10 Jun '11

10 Jun '11

ACTIVITY SUMMARY (2011-06-03 - 2011-06-10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2826 (+11) closed 21268 (+47) total 24094 (+58) Open issues with patches: 1236 Issues opened (45) ================== #5906: Risk of confusion in multiprocessing module - daemonic process http://bugs.python.org/issue5906 reopened by pakal #9516: sysconfig: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3" but http://bugs.python.org/issue9516 reopened by eric.araujo #12255: A few changes to .*ignore http://bugs.python.org/issue12255 opened by eric.araujo #12256: Link isinstance/issubclass doc to abc module http://bugs.python.org/issue12256 opened by eric.araujo #12257: Rework/replace use of DISTUTILS_USE_SDK in packaging http://bugs.python.org/issue12257 opened by eric.araujo #12258: Clean up bytes I/O in get_compiler_versions http://bugs.python.org/issue12258 opened by eric.araujo #12259: Test and document which compilers can be created on which plat http://bugs.python.org/issue12259 opened by eric.araujo #12260: Make install default to user site-packages http://bugs.python.org/issue12260 opened by eric.araujo #12261: urllib.parse docs still refer to urlparse http://bugs.python.org/issue12261 opened by tlesher #12263: punycode codec ignores the error handler argument http://bugs.python.org/issue12263 opened by haypo #12268: file readline, readlines & readall methods can lose data on EI http://bugs.python.org/issue12268 opened by gregory.p.smith #12271: Python 2.7.x on IA64 running SLES 11 SP1 http://bugs.python.org/issue12271 opened by v.claudic #12272: Python 2.7.1 version conflict for package "Tcl" on Windows 7 http://bugs.python.org/issue12272 opened by jackie3 #12273: Change ast.__version__ calculation to provide consistent order http://bugs.python.org/issue12273 opened by ncoghlan #12274: "Print window" menu on IDLE aborts whole application http://bugs.python.org/issue12274 opened by gagenellina #12275: urllib.request.HTTPRedirectHandler won't redirect to a URL wit http://bugs.python.org/issue12275 opened by lilydjwg #12276: 3.x ignores sys.tracebacklimit=0 http://bugs.python.org/issue12276 opened by gagenellina #12277: Missing comma in os.walk docs http://bugs.python.org/issue12277 opened by Retro #12278: Core mentorship mention in the devguide http://bugs.python.org/issue12278 opened by adam.woodbeck #12279: Add build_distinfo command to packaging http://bugs.python.org/issue12279 opened by eric.araujo #12281: bytes.decode('mbcs', 'ignore') does replace undecodable bytes http://bugs.python.org/issue12281 opened by haypo #12282: build process adds CWD (null entry) to PYTHONPATH if PYTHONPAT http://bugs.python.org/issue12282 opened by r.david.murray #12284: argparse.ArgumentParser: usage example option http://bugs.python.org/issue12284 opened by jonash #12285: Unexpected behavior for 0 or negative processes in multiproces http://bugs.python.org/issue12285 opened by jorgsk #12287: ossaudiodev: stack corruption with FD >= FD_SETSIZE http://bugs.python.org/issue12287 opened by neologix #12288: Python 2.7.1 tkSimpleDialog initialvalue http://bugs.python.org/issue12288 opened by busfault #12289: http.server.CGIHTTPRequestHandler doesn't check if a Python sc http://bugs.python.org/issue12289 opened by haypo #12290: __setstate__ is called for false values http://bugs.python.org/issue12290 opened by eltoder #12291: file written using marshal in 3.2 can be read by 2.7, but not http://bugs.python.org/issue12291 opened by vinay.sajip #12294: multiprocessing.Pool: Need a way to find out if work are finis http://bugs.python.org/issue12294 opened by mozbugbox #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 opened by eric.araujo #12296: Minor clarification in devguide http://bugs.python.org/issue12296 opened by eric.araujo #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 opened by eric.araujo #12299: Stop documenting functions added by site as builtins http://bugs.python.org/issue12299 opened by eric.araujo #12300: Document pydoc.help http://bugs.python.org/issue12300 opened by eric.araujo #12301: Use :data:`sys.thing` instead of ``sys.thing`` throughout http://bugs.python.org/issue12301 opened by eric.araujo #12302: packaging test command needs access to .dist-info http://bugs.python.org/issue12302 opened by michael.mulich #12303: expose sigwaitinfo() and sigtimedwait() in the signal module http://bugs.python.org/issue12303 opened by haypo #12304: expose signalfd(2) in the signal module http://bugs.python.org/issue12304 opened by haypo #12305: Building PEPs doesn't work on Python 3 http://bugs.python.org/issue12305 opened by ericsnow #12306: zlib: Expose zlibVersion to query runtime version of zlib http://bugs.python.org/issue12306 opened by torsten #12307: Inconsistent formatting of section titles in PEP 0 http://bugs.python.org/issue12307 opened by ericsnow #12308: Add link to PEP 0 for topical index in wiki http://bugs.python.org/issue12308 opened by ericsnow #12309: os.environ was modified by test_packaging http://bugs.python.org/issue12309 opened by haypo #12310: Segfault in test_multiprocessing http://bugs.python.org/issue12310 opened by haypo Most recent 15 issues with no replies (15) ========================================== #12309: os.environ was modified by test_packaging http://bugs.python.org/issue12309 #12308: Add link to PEP 0 for topical index in wiki http://bugs.python.org/issue12308 #12307: Inconsistent formatting of section titles in PEP 0 http://bugs.python.org/issue12307 #12306: zlib: Expose zlibVersion to query runtime version of zlib http://bugs.python.org/issue12306 #12305: Building PEPs doesn't work on Python 3 http://bugs.python.org/issue12305 #12303: expose sigwaitinfo() and sigtimedwait() in the signal module http://bugs.python.org/issue12303 #12301: Use :data:`sys.thing` instead of ``sys.thing`` throughout http://bugs.python.org/issue12301 #12300: Document pydoc.help http://bugs.python.org/issue12300 #12299: Stop documenting functions added by site as builtins http://bugs.python.org/issue12299 #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 #12296: Minor clarification in devguide http://bugs.python.org/issue12296 #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 #12294: multiprocessing.Pool: Need a way to find out if work are finis http://bugs.python.org/issue12294 #12288: Python 2.7.1 tkSimpleDialog initialvalue http://bugs.python.org/issue12288 #12272: Python 2.7.1 version conflict for package "Tcl" on Windows 7 http://bugs.python.org/issue12272 Most recent 15 issues waiting for review (15) ============================================= #12308: Add link to PEP 0 for topical index in wiki http://bugs.python.org/issue12308 #12306: zlib: Expose zlibVersion to query runtime version of zlib http://bugs.python.org/issue12306 #12304: expose signalfd(2) in the signal module http://bugs.python.org/issue12304 #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 #12296: Minor clarification in devguide http://bugs.python.org/issue12296 #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 #12291: file written using marshal in 3.2 can be read by 2.7, but not http://bugs.python.org/issue12291 #12289: http.server.CGIHTTPRequestHandler doesn't check if a Python sc http://bugs.python.org/issue12289 #12287: ossaudiodev: stack corruption with FD >= FD_SETSIZE http://bugs.python.org/issue12287 #12285: Unexpected behavior for 0 or negative processes in multiproces http://bugs.python.org/issue12285 #12281: bytes.decode('mbcs', 'ignore') does replace undecodable bytes http://bugs.python.org/issue12281 #12278: Core mentorship mention in the devguide http://bugs.python.org/issue12278 #12277: Missing comma in os.walk docs http://bugs.python.org/issue12277 #12268: file readline, readlines & readall methods can lose data on EI http://bugs.python.org/issue12268 #12261: urllib.parse docs still refer to urlparse http://bugs.python.org/issue12261 Top 10 most discussed issues (10) ================================= #12248: __dir__ semantics changed in Python 2.7.2 http://bugs.python.org/issue12248 18 msgs #12014: str.format parses replacement field incorrectly http://bugs.python.org/issue12014 12 msgs #12246: Warn when trying to install third-party module from an uninsta http://bugs.python.org/issue12246 12 msgs #8927: Handle version incompatibilities in dependencies http://bugs.python.org/issue8927 11 msgs #12211: Better document math.copysign behavior. http://bugs.python.org/issue12211 11 msgs #12226: use secured channel for uploading packages to pypi http://bugs.python.org/issue12226 11 msgs #11457: Expose nanosecond precision from system calls http://bugs.python.org/issue11457 10 msgs #7511: msvc9compiler.py: ValueError when trying to compile with VC Ex http://bugs.python.org/issue7511 8 msgs #12084: os.stat() on windows doesn't consider relative symlink http://bugs.python.org/issue12084 8 msgs #12291: file written using marshal in 3.2 can be read by 2.7, but not http://bugs.python.org/issue12291 8 msgs Issues closed (47) ================== #3771: test_httpservers intermittent failure, test_post and EINTR http://bugs.python.org/issue3771 closed by gregory.p.smith #3992: remove custom log module from distutils2 http://bugs.python.org/issue3992 closed by eric.araujo #4949: Constness in PyErr_NewException http://bugs.python.org/issue4949 closed by benjamin.peterson #7015: Getting call trace while executing "modules spam" at help prom http://bugs.python.org/issue7015 closed by eric.araujo #8407: expose pthread_sigmask(), pthread_kill(), sigpending() and sig http://bugs.python.org/issue8407 closed by haypo #9205: Parent process hanging in multiprocessing if children terminat http://bugs.python.org/issue9205 closed by pitrou #9344: please add posix.getgrouplist() http://bugs.python.org/issue9344 closed by rosslagerwall #10027: os.lstat/os.stat don't set st_nlink on Windows http://bugs.python.org/issue10027 closed by brian.curtin #10424: better error message from argparse when positionals missing http://bugs.python.org/issue10424 closed by r.david.murray #10645: Remove egg-info files in stdlib http://bugs.python.org/issue10645 closed by eric.araujo #10694: zipfile.py end of central directory detection not robust http://bugs.python.org/issue10694 closed by r.david.murray #10884: pkgutil EggInfoDistribution requirements for .egg-info metadat http://bugs.python.org/issue10884 closed by michael.mulich #11028: Implement the setup.py -> setup.cfg in mkcfg http://bugs.python.org/issue11028 closed by eric.araujo #11041: On the distutils2 documentation, 'requires-python' shouldn't b http://bugs.python.org/issue11041 closed by eric.araujo #11092: setup.cfg isn't packaged when running sdist http://bugs.python.org/issue11092 closed by eric.araujo #11203: gzip doc is behind http://bugs.python.org/issue11203 closed by teamnoir #11583: os.path.isdir() is slow on windows http://bugs.python.org/issue11583 closed by brian.curtin #11893: Obsolete SSLFakeFile in smtplib? http://bugs.python.org/issue11893 closed by pitrou #12019: Dead or buggy code in importlib.test.__main__ http://bugs.python.org/issue12019 closed by eric.araujo #12021: mmap.read requires an argument http://bugs.python.org/issue12021 closed by neologix #12040: Expose a Process.sentinel property (and fix polling loop in Pr http://bugs.python.org/issue12040 closed by pitrou #12080: decimal.py: performance in _power_exact http://bugs.python.org/issue12080 closed by mark.dickinson #12127: Inconsistent leading zero treatment http://bugs.python.org/issue12127 closed by mark.dickinson #12168: SysLogHandler incorrectly appends \000 to messages http://bugs.python.org/issue12168 closed by vinay.sajip #12214: platform module can't detect archlinux distribution http://bugs.python.org/issue12214 closed by eric.araujo #12224: problem with siginterrupt http://bugs.python.org/issue12224 closed by haypo #12234: unittest2 could enable regex debugging for more information http://bugs.python.org/issue12234 closed by r.david.murray #12237: Document how to open non-default webbrowser http://bugs.python.org/issue12237 closed by terry.reedy #12249: add missing command http://bugs.python.org/issue12249 closed by eric.araujo #12254: PEP-3107 has a wrong attribute name for function annotations http://bugs.python.org/issue12254 closed by eric.araujo #12262: Not Inheriting File Descriptors on Windows? http://bugs.python.org/issue12262 closed by 3vi1 #12264: parser.expr(): reference count error if more than one argument http://bugs.python.org/issue12264 closed by python-dev #12265: revamp argument errors http://bugs.python.org/issue12265 closed by python-dev #12266: str.capitalize contradicts oneself http://bugs.python.org/issue12266 closed by r.david.murray #12267: Difficult to compile python in Visual Studio 2010 express http://bugs.python.org/issue12267 closed by loewis #12269: IDLE 2.7 hangs on Mac OS X 10.6 http://bugs.python.org/issue12269 closed by ned.deily #12270: Please update copyright notice on bugs.python.org to 2011? http://bugs.python.org/issue12270 closed by ezio.melotti #12280: If statement http://bugs.python.org/issue12280 closed by brian.curtin #12283: python3.2 smtplib _quote_periods http://bugs.python.org/issue12283 closed by r.david.murray #12286: float('nan') breaks sort() method on a list of floats http://bugs.python.org/issue12286 closed by ezio.melotti #12292: bpython http://bugs.python.org/issue12292 closed by amaury.forgeotdarc #12293: wrong arguments passed to SMTP.sendmail in example http://bugs.python.org/issue12293 closed by r.david.murray #12298: Sphinx glitch in library/functions http://bugs.python.org/issue12298 closed by georg.brandl #12311: memory leak with self-referencing dict http://bugs.python.org/issue12311 closed by Albert.Zeyer #12312: is ok http://bugs.python.org/issue12312 closed by brian.curtin #1083299: Distutils doesn't pick up all the files it should. http://bugs.python.org/issue1083299 closed by eric.araujo #1699259: replacing char* with const char* in sysmodule.c/.h http://bugs.python.org/issue1699259 closed by sebastinas

1 0

Re: [Python-Dev] cpython (3.2): Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of os.stat.
by Georg Brandl 09 Jun '11

09 Jun '11

On 06/09/11 02:00, brian.curtin wrote: > http://hg.python.org/cpython/rev/88e318166eaf > changeset: 70713:88e318166eaf > branch: 3.2 > parent: 70700:0aa3064d1cef > user: Brian Curtin <brian(a)python.org> > date: Wed Jun 08 18:17:18 2011 -0500 > summary: > Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of os.stat. > > By changing to the Windows GetFileAttributes API in nt._isdir we can figure > out if the path is a directory without opening the file via os.stat. This has > the minor benefit of speeding up os.path.isdir by at least 2x for regular > files and 10-15x improvements were seen on symbolic links (which opened the > file multiple times during os.stat). Since os.path.isdir is used in > several places on interpreter startup, we get a minor speedup in startup time. > > files: > Lib/ntpath.py | 13 ++++++++++ > Misc/NEWS | 3 ++ > Modules/posixmodule.c | 37 +++++++++++++++++++++++++++++++ > 3 files changed, 53 insertions(+), 0 deletions(-) > > > diff --git a/Lib/ntpath.py b/Lib/ntpath.py > --- a/Lib/ntpath.py > +++ b/Lib/ntpath.py > @@ -672,3 +672,16 @@ > def sameopenfile(f1, f2): > """Test whether two file objects reference the same file""" > return _getfileinformation(f1) == _getfileinformation(f2) > + > + > +try: > + # The genericpath.isdir implementation uses os.stat and checks the mode > + # attribute to tell whether or not the path is a directory. > + # This is overkill on Windows - just pass the path to GetFileAttributes > + # and check the attribute from there. > + from nt import _isdir > +except ImportError: > + from genericpath import isdir as _isdir > + > +def isdir(path): > + return _isdir(path) Not that it matters, but ISTM that this would be faster as try: from nt import _isdir as isdir except ImportError: pass Georg

3 2

3.2.1 and Issue 12291
by Vinay Sajip 09 Jun '11

09 Jun '11

I just filed an issue which shows a serious breakage in the marshal module: "file written using marshal in 3.2 can be read by 2.7, but not 3.2 or 3.3" http://bugs.python.org/issue12291 Perhaps this issue should be marked as a release blocker for 3.2.1 - what do others (particularly Georg, of course) think? While it does appear to have been broken for some time, it seems a bit too serious to leave until a 3.2.2 release. Regards, Vinay Sajip

2 1

Can we improve support for abstract base classes with desciptors
by Darren Dale 09 Jun '11

09 Jun '11

I would like to try to address some shortfalls with the way python deals with abstract base classes containing descriptors. I originally was just concerned with improving support for defining abstract properties with the decorator syntax and converting between abstract and concrete properties, but recently realized that the problem extends to descriptors in general. ABCs ---- First, a bit of background may be in order. An abstract base class is defined by specifying its metaclass as ABCMeta (or a subclass thereof):: class MyABC(metaclass=ABCMeta): @abstractmethod def foo(self): pass When trying to instantiate MyABC or any of its subclasses, ABCMeta inspects the current class namespace for items tagged with __isabstractmethod__=True:: class ABCMeta(type): #[...] def __new__(mcls, name, bases, namespace): cls = super().__new__(mcls, name, bases, namespace) # Compute set of abstract method names abstracts = {name for name, value in namespace.items() if getattr(value, "__isabstractmethod__", False)} ABCMeta then checks if any of the base classes define any items tagged with __isabstractmethod__ and whether they remain abstract in the current class namespace:: for base in bases: for name in getattr(base, "__abstractmethods__", set()): value = getattr(cls, name, None) if getattr(value, "__isabstractmethod__", False): abstracts.add(name) cls.__abstractmethods__ = frozenset(abstracts) In Objects/typeobject.c, __abstractmethods__ is actually a descriptor, and setting it gives the type a chance to set an internal flag specifying if it has any abstract methods defined. When object_new is called in typeobject.c, the flag is checked and an error is raised if any abstract methods were identified. Issues with ABCs and descriptors -------------------------------- In order for this scheme to work, ABCMeta needs to identify all of the abstract methods, but there are some limitations when we consider descriptors. For example, Python's property is a composite object, whose behavior is defined by the getter, setter, and deleter methods with which it is composed. Since there is already an @abstractmethod decorator, I would have suspected that defining abstract properties would be intuitive:: class MyABC(metaclass=ABCMeta): @abstractmethod def _get_foo(self): pass @abstractmethod def _set_foo(self, val): pass foo = property(_get_foo, _set_foo) @property @abstractmethod def bar(self): pass @bar.setter @abstractmethod def bar(self, val): pass Ideally, one would want the flexibility of defining a concrete getter and an abstract setter, for example. However, ABCMeta does not inspect the descriptors of a class to see if they contain any abstract methods. It only inspects the descriptor itself for a True __isabstractmethod__ attribute. This places the burdon on every descriptor implementation to provide its own support for ABC compatibility. For example, support for abstract properties was attempted by adding abstractproperty to the abc module. abstractproperty subclasses the property builtin (as opposed to the relationship between every other abstract and concrete class in the python language). Here is the definition of abstractproperty, in its entirety (modulo docstrings):: class abstractproperty(property): __isabstractmethod__ = True A number of problems manifest with this approach, and I think they all can be traced to the fact that the abstractedness of a descriptor is currently not dependent upon the abstractedness of the methods with which it is composed. The documentation for abstractproperty doesn't suggest using @abstractmethod:: class C(metaclass=ABCMeta): def getx(self): ... def setx(self, value): ... x = abstractproperty(getx, setx) which leads to Issue #1: What is abstract about C.x? How does a subclass of C know whether it needs to override the getter or setter? Issue #2: The decorator syntax cannot be used to convert an abstract property into a concrete one. (This relates to Issue #1: how would a descriptor even know when such a conversion would be appropriate?) Running the following code:: from abc import ABCMeta, abstractmethod, abstractproperty class AbstractFoo(metaclass=ABCMeta): @abstractproperty def bar(self): return 1 @bar.setter def bar(self, val): pass class ConcreteFoo(AbstractFoo): @AbstractFoo.bar.getter def bar(self): return 1 @bar.setter def bar(self, val): pass foo = ConcreteFoo() yields:: TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods bar Issue #3: The following class *is* instantiable, even though AbstractFoo declared that a setter for bar is required:: class ConcreteFoo(AbstractFoo): @property def bar(self): pass Previous attempt to improve abc.abstractproperty ------------------------------------------------ It seems to me that the strategy used by abc.abstractproperty is fundamentally ill-advised. I explored the possibility of extending abstractproperty, redefining its getter, setter, and deleter methods such that they would work in conjunction with the @abstractmethod decorator and yield an instance of the builtin property once all abstract methods were replaced with concrete ones (http://bugs.python.org/issue11610). Issues #1 and #2 were addressed, but there were still problems with that approach. It did not address Issue #3, and it also introduced a new issue, #4:: class AbstractFoo(metaclass=ABCMeta): @abstractproperty # bar would be an abstractproperty, even though the getter is concrete def bar(self): return 1 @bar.setter # bar.setter inspected the getter and the new setter, did not identify # any abstract methods, and thus returned an instance of the built-in # property def bar(self, val): pass @bar.deleter # bar is a concrete property, its deleter decorator does not know it # is supposed to check for abstract methods, so it will return an # instance of the built-in property: @abstractmethod def bar(self): pass By the time the deleter was specified, bar was a concrete property, which does not know it should return an instance of abstractproperty (in part because the inheritance diagram for property/abstractproperty is inverted). Thus, AbstractFoo was instantiable, even though it shouldn't be. Finally, issue #5: the current approach taken by ABCMeta and abstractproperty places the burdon on descriptors to identify themselves to ABCMeta as abstract. Considering the issues encountered with abstractproperty, this may be an onerous requirement. There has been a fair amount of discussion at http://bugs.python.org/issue11610 , which can be summarized as a) concerns about maintaining backward compatibility, and b) objections to requiring @abstractmethod to specify that a method being passed to abstractproperty is abstract. Extending ABCMeta: A Promising Way Forward ------------------------------------------ I think the key is to focus on Issue #3. ABCMeta needs to be improved to recognize descriptor objects, both in the current namespace as well as the base classes, and to identify any abstract methods associated with the descriptors. I suggest the following approach in ABCMeta:: def __new__(mcls, name, bases, namespace): cls = super().__new__(mcls, name, bases, namespace) # Compute set of abstract method names def isdescriptor(val): return hasattr(val, '__get__') or hasattr(val, '__set__') \ or hasattr(val, '__delete__') def getabstracts(ns): return [name for name, value in ns.items() if getattr(value, "__isabstractmethod__", False)] abstracts = getabstracts(namespace) for item, val in namespace.items(): # further inspect descriptors for abstract methods: if isdescriptor(val): ## unfortunately, can't import inspect: #from inspect import getmembers #d = dict(getmembers(val)) ## so instead, use the following: d = dict((k, getattr(val, k, None)) for k in dir(val)) for name in getabstracts(d): # add the abstract descriptor methods to the list: abstracts.append('%s.%s'%(item, name)) for base in bases: for name in getattr(base, "__abstractmethods__", set()): if '.' in name: # base class identified a descriptor abstract method: k, v = name.split('.') desc = getattr(cls, k, None) val = getattr(desc, v, None) else: val = getattr(cls, name, None) if val is None or getattr(val, "__isabstractmethod__", False): abstracts.append(name) cls.__abstractmethods__ = frozenset(abstracts) I rolled everything into __new__ just to keep it as simple as possible for the sake of discussion. Python already provides the rest of the framework needed for descriptors to work properly with ABCs. This implementation actually works; I've tested it with an existing python-3.2 install:: from abc import ABCMeta, abstractmethod class AbstractFoo(metaclass=ABCMeta): @property @abstractmethod def bar(self): return 1 @bar.setter @abstractmethod def bar(self, val): pass >>> abstractfoo = AbstractFoo() Traceback (most recent call last): File "temp.py", line 17, in <module> abstractfoo = AbstractFoo() TypeError: Can't instantiate abstract class AbstractFoo with abstract methods bar.fget, bar.fset as expected. Note the more informative error message indicating what about the bar property is abstract. Also:: class ConcreteFoo(AbstractFoo): @AbstractFoo.bar.getter def bar(self): return 1 >>> foo = ConcreteFoo() Traceback (most recent call last): File "temp.py", line 24, in <module> foo = ConcreteFoo() TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods bar.fset So issue #1 is addressed, since we are explicitly specifying which descriptor methods are abstract. Issue #2 has been addressed, since the following class is instantiable:: class ConcreteFoo(AbstractFoo): @AbstractFoo.bar.getter def bar(self): return 1 @bar.setter def bar(self, val): pass Issue #3 is also addressed. In the following example, even though I redefine the bar property as a readonly property, ABCMeta recognizes that a setter is needed:: class ConcreteFoo(AbstractFoo): @property def bar(self): return 1 >>> foo = ConcreteFoo() Traceback (most recent call last): File "temp.py", line 24, in <module> foo = ConcreteFoo() TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods bar.fset Issue #4 (introduced in a previous attempt to solve the problem using abstractproperty) is also addressed:: class AbstractFoo(metaclass=ABCMeta): @property def bar(self): return 1 @bar.setter def bar(self, val): pass @bar.deleter @abstractmethod def bar(self): pass >>> abstractfoo = AbstractFoo() Traceback (most recent call last): File "temp.py", line 15, in <module> abstractfoo = AbstractFoo() TypeError: Can't instantiate abstract class AbstractFoo with abstract methods bar.fdel Finally, this approach addresses Issue #5 by holding ABCMeta responsible for identifying the abstractedness of descriptor methods, rather than placing that burdon on the desciptor objects to identify themselves as abstract. If ABCMeta were extended as above to identify abstract methods associated with descriptors, third parties would simply decorate methods used to compose the descriptors with @abstractmethod. This change to ABCMeta would not effect the behavior of abstractproperty, so backward compatibility would be maintained in that respect. But I think abstractproperty should be deprecated, or at the very least removed from the documentation. The documentation for @abstractmethod in >=python-3.3 should be extended to provide examples with properties/descriptors. The syntax would be backward compatible with older python versions, but with <python-3.3 ABCMeta would simply not recognize descriptors' abstract methods. This leads to one source of potential forward compatibility:: class AbstractFoo(metaclass=ABCMeta): @property @abstractmethod def bar(self): return 1 class ConcreteFoo(AbstractFoo): pass Both above classes would be instantiable with <python-3.3, but not with >=python3.3. In my opinion, this is a feature: python-3.3 has identified a bug in ConcreteFoo. The developer would not have tagged that method as abstract unless they had intended for consumers of AbstractFoo to provide a concrete implementation. Darren

2 5

The socket HOWTO
by Antoine Pitrou 08 Jun '11

08 Jun '11

Hello, I would like to suggest that we remove the socket HOWTO (currently at http://docs.python.org/dev/howto/sockets.html) My main issue with this document is that it doesn't seem to have a well-defined destination: - people who know sockets won't learn anything from it - but people who don't know sockets will probably find it clear as mud (for example, what's an "INET" or "STREAM" socket? what's "select"?) I have other issues, such as the style/tone it's written in. I'm sure the author had fun writing it but it doesn't fit well with the rest of the documentation. Also, the author gives a lot of "advice" without explaining or justifying it ("if somewhere in those input lists of sockets is one which has died a nasty death, the select will fail" -> is that really true? what is a "nasty death" and how is that supposed to happen? couldn't the author have put a 3-line example to demonstrate this supposed drawback and how it manifests?). And, finally, many statements seem arbitrary ("There’s no question that the fastest sockets code uses non-blocking sockets and select to multiplex them") or plain wrong ("threading support in Unixes varies both in API and quality. So the normal Unix solution is to fork a subprocess to deal with each connection"). I don't think giving misleading advice to users is really a good idea. And suggesting beginners they use non-blocking sockets without even *showing* how (or pointing to asyncore or Twisted) is a very bad idea. select() is not enough, you still have to be prepared to get EAGAIN or EWOULDBLOCK when calling recv() or send() (i.e. select() can give false positives). Oh and I think it's obsolete too, because the "class mysocket" concatenates the output of recv() with a str rather than a bytes object. Not to mention that features of the "class mysocket" can be had using a buffered socket.makefile() instead of writing custom code. (followed up from http://bugs.python.org/issue12126 at Eli's request) Regards Antoine.

17 35