Mailman 3 September 2011 - Python-Dev

PEP 393 memory savings update
by "Martin v. Löwis" Sept. 27, 2011

Sept. 27, 2011

I have redone my memory benchmark, and added a few new counters. The application is a very small Django application. The same source code of the app and Django itself is used on all Python versions. The full list of results is at http://www.dcl.hpi.uni-potsdam.de/home/loewis/djmemprof/ Here are some excerpts: A. 32-bit builds, storage for Unicode objects 3.x, 32-bit wchar_t: 6378540 3.x, 16-bit wchar_t: 3694694 PEP 393: 2216807 Compared to the previous results, there are now some significant savings even compared to a narrow unicode build. B. 3.x, number of strings by maxchar: ASCII: 35713 (1,300,000 chars) Latin-1: 235 (11,000 chars) BMP: 260 (700 chars) other: 0 total: 36,000 (1,310,000 chars) This explains why the savings for shortening ASCII objects are significant in this application. I have no good intuition how this effect would show for "real" applications. It may be that the percentage of ASCII strings (in number and chars) grows proportionally with the total number of strings; it may also be that the majority of these strings is a certain fixed overhead (resulting from Python identifiers and other interned strings). C. String-ish objects in 2.7 and 3.3-trunk: 2.x 3.x #unicode 370 36,000 #bytes 43,000 14,000 #total 43,400 50,000 len(unicode) 5,300 1,306,000 len(bytes) 2,040,000 860,000 len(total) 2,046,000 2,200,000 (Note: the computations in the results are slightly messed up: the number of bytes for bytes objectts is actually the sum of the lengths, not the sum of the sizeofs; this gets added in the "total" lines to the sum of sizeofs of unicode strings, which is non-sensical. The table above corrects this) As you can see, Python 3 creates more string objects in total. D. Memory consumption for 2.x, 3.x, PEP 393, accounting both unicode and bytes objects, using 32-bit builds and 32-bit wchar_t: 2.x: 3,620,000 bytes 3.x: 7,750,000 bytes PEP 393: 3,340,000 bytes This suggests that PEP 393 actually reduces memory consumption below what 2.7 uses. This is offset though by "other" (non-string) objects, which take 300KB more in 3.x. Regards, Martin

2 1

PyCon 2012 Proposals Due October 12
by Brian Curtin Sept. 27, 2011

Sept. 27, 2011

The deadline for PyCon 2012 tutorial, talk, and poster proposals is under 15 days away, so be sure to get your submissions in by October 12, 2011. Whether you’re a first-timer or an experienced veteran, PyCon is depends on you, the community, coming together to build the best conference schedule possible. Our call for proposals (http://us.pycon.org/2012/cfp/) lays out the details it takes to be included in the lineup for the conference in Santa Clara, CA on March 7-15, 2012. If you’re unsure of what to write about, our recent survey yielded a large list of potential talk topics (http://pycon.blogspot.com/2011/09/need-talk-ideas.html), and plenty of ideas for tutorials (INSERT TUTORIAL POST). We’ve also come up with general tips on proposal writing at http://pycon.blogspot.com/2011/08/writing-good-proposal.html to ensure everyone has the most complete proposal when it comes time for review. As always, the program committee wants to put together an incredible conference, so they’ll be working with submitters to fine tune proposal details and help you produce the best submissions. We’ve had plenty of great news to share since we first announced the call for proposals. Paul Graham of Y Combinator was recently announced as a keynote speaker (http://pycon.blogspot.com/2011/09/announcing-first-pycon-2012-keynote.html), making his return after a 2003 keynote. David Beazley, famous for his mind-blowing talks on CPython’s Global Interpreter Lock, was added to the plenary talk series (http://pycon.blogspot.com/2011/09/announcing-first-pycon-2012-plenary.html). Sponsors can now list their job openings on the “Job Fair” section of the PyCon site (http://pycon.blogspot.com/2011/09/announcing-pycon-2012-fair-page-sponsor.h…). We’re hard at work to bring you the best conference yet, so stay tuned to PyCon news at http://pycon.blogspot.com/ and on Twitter at https://twitter.com/#!/pycon. We recently eclipsed last year’s sponsorship count of 40 and are currently at a record 52 organizations supporting PyCon. If you or your organization are interested in sponsoring PyCon, we’d love to hear from you, so check out our sponsorship page (http://us.pycon.org/2012/sponsors/). A quick thanks to all of our awesome PyCon 2012 Sponsors: - Diamond Level: Google and Dropbox. - Platinum Level: New Relic, SurveyMonkey, Microsoft, Eventbrite, Nasuni and Gondor.io - Gold Level: Walt Disney Animation Studios, CCP Games, Linode, Enthought, Canonical, Dotcloud, Loggly, Revsys, ZeOmega, Bitly, ActiveState, JetBrains, Caktus, Disqus, Spotify, Snoball, Evite, and PlaidCloud - Silver Level: Imaginary Landscape, WiserTogether, Net-ng, Olark, AG Interactive, Bitbucket, Open Bastion, 10Gen, gocept, Lex Machina, fwix, github, toast driven, Aarki, Threadless, Cox Media, myYearBook, Accense Technology, Wingware, FreshBooks, and BigDoor - Lanyard: Dreamhost - Sprints: Reddit - FLOSS: OSU/OSL, OpenHatch The PyCon Organizers - http://us.pycon.org/2012 Jesse Noller - Chairman - jnoller(a)python.org Brian Curtin - Publicity Coordinator - brian(a)python.org

1 0

Re: [Python-Dev] [Python-checkins] cpython: Issue #12981: rewrite multiprocessing_{sendfd, recvfd} in Python.
by Nick Coghlan Sept. 26, 2011

Sept. 26, 2011

On Sun, Sep 25, 2011 at 4:04 AM, charles-francois.natali <python-checkins(a)python.org> wrote: > +if not(sys.platform == 'win32' or (hasattr(socket, 'CMSG_LEN') and > + hasattr(socket, 'SCM_RIGHTS'))): > raise ImportError('pickling of connections not supported') I'm pretty sure the functionality checks for CMSG_LEN and SCM_RIGHTS mean the platform check for Windows is now redundant. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 2

Re: [Python-Dev] range objects in 3.x
by Ethan Furman Sept. 23, 2011

Sept. 23, 2011

Benjamin Peterson wrote: > 2011/9/23 Ethan Furman <ethan(a)stoneleaf.us>: >> >> Follow-up question: since the original range returned lists, and comparisons >> do make sense for lists, should the new range also implement them? > > What would be the use-case? The only reason I'm aware of at the moment is to prevent loss of functionality from 2.x range to 3.x range. I'm -0 with a decision to not have range be orderable; but I understand there are bigger fish to fry. :) My original concern was that the comparison methods were there at all, but looking around I see object has them, so it makes sense to me now. I had thought I would have to implement them if I went ahead with an frange (for floats). >> I note >> that it does implement __contains__, __getitem__, count, and index in the >> same way that list does. > > That's because it implements the Sequence ABC. So the question becomes, Why does it implement the Sequence ABC? Because the original range returned a list and those operations made sense? ~Ethan~

6 7

Re: [Python-Dev] [Python-checkins] cpython (3.2): Issue #7732: Don't open a directory as a file anymore while importing a
by Éric Araujo Sept. 23, 2011

Sept. 23, 2011

Hi Victor, > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -10,6 +10,10 @@ > Core and Builtins > ----------------- > > +- Issue #7732: Don't open a directory as a file anymore while importing a > + module. Ignore the direcotry if its name matchs the module name (e.g. Typo: direcotry

2 1

Summary of Python tracker Issues
by Python tracker Sept. 23, 2011

Sept. 23, 2011

ACTIVITY SUMMARY (2011-09-16 - 2011-09-23) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3030 (+11) closed 21788 (+31) total 24818 (+42) Open issues with patches: 1299 Issues opened (34) ================== #11686: Update of some email/ __all__ lists http://bugs.python.org/issue11686 reopened by r.david.murray #11780: email.encoders are broken http://bugs.python.org/issue11780 reopened by r.david.murray #12991: Python 64-bit build on HP Itanium - Executable built successfu http://bugs.python.org/issue12991 reopened by wah meng #12997: sqlite3: PRAGMA foreign_keys = ON doesn't work http://bugs.python.org/issue12997 opened by Mark.Bucciarelli #12998: Memory leak with CTypes Structure http://bugs.python.org/issue12998 opened by a01 #12999: _XOPEN_SOURCE and _XOPEN_SOURCE_EXTENDED usage on Solaris http://bugs.python.org/issue12999 opened by neologix #13000: unhandled exception at install http://bugs.python.org/issue13000 opened by jorge.seifert #13001: test_socket.testRecvmsgTrunc failure on FreeBSD 7.2 buildbot http://bugs.python.org/issue13001 opened by neologix #13004: pprint: add option to truncate sequences http://bugs.python.org/issue13004 opened by terry.reedy #13008: syntax error when pasting valid snippet into console without e http://bugs.python.org/issue13008 opened by techtonik #13009: Remove documentation in distutils2 repo http://bugs.python.org/issue13009 opened by eric.araujo #13011: Frozen programs require the original build directory in order http://bugs.python.org/issue13011 opened by malcolmp #13012: Allow keyword argument in str.splitlines() http://bugs.python.org/issue13012 opened by mark.dickinson #13013: _ctypes.c: refleak http://bugs.python.org/issue13013 opened by Suman.Saha #13014: _ssl.c: refleak http://bugs.python.org/issue13014 opened by Suman.Saha #13015: _collectionsmodule.c: refleak http://bugs.python.org/issue13015 opened by Suman.Saha #13016: selectmodule.c: refleak http://bugs.python.org/issue13016 opened by Suman.Saha #13017: pyexpat.c: refleak http://bugs.python.org/issue13017 opened by Suman.Saha #13018: dictobject.c: refleak http://bugs.python.org/issue13018 opened by Suman.Saha #13019: bytearrayobject.c: refleak http://bugs.python.org/issue13019 opened by Suman.Saha #13020: structseq.c: refleak http://bugs.python.org/issue13020 opened by Suman.Saha #13023: argparse should allow displaying argument default values in ad http://bugs.python.org/issue13023 opened by denilsonsa #13024: cgitb uses stdout encoding http://bugs.python.org/issue13024 opened by haypo #13025: mimetypes should read the rule file using UTF-8, not the local http://bugs.python.org/issue13025 opened by haypo #13026: Dis module - documentation of MAKE_FUNCTION http://bugs.python.org/issue13026 opened by arno #13027: python 2.6.6 interpreter core dumps on modules command from he http://bugs.python.org/issue13027 opened by Balachandran.Sivakumar #13028: python wastes linux users time by checking for dylib on each d http://bugs.python.org/issue13028 opened by fzvqedi #13029: test_strptime fails on Windows 7 french http://bugs.python.org/issue13029 opened by haypo #13030: Be more generic when identifying the Windows main dir in insta http://bugs.python.org/issue13030 opened by sandro.tosi #13031: [PATCH] small speed-up for tarfile.py when unzipping tarballs http://bugs.python.org/issue13031 opened by jpeel #13032: h2py.py can fail with UnicodeDecodeError http://bugs.python.org/issue13032 opened by Arfrever #13033: recursive chown for shutils http://bugs.python.org/issue13033 opened by Low.Kian.Seong #13034: Python does not read Alternative Subject Names from SSL certif http://bugs.python.org/issue13034 opened by atrasatti #13035: "maintainer" value clear the "author" value when registering http://bugs.python.org/issue13035 opened by jab Most recent 15 issues with no replies (15) ========================================== #13035: "maintainer" value clear the "author" value when registering http://bugs.python.org/issue13035 #13034: Python does not read Alternative Subject Names from SSL certif http://bugs.python.org/issue13034 #13032: h2py.py can fail with UnicodeDecodeError http://bugs.python.org/issue13032 #13030: Be more generic when identifying the Windows main dir in insta http://bugs.python.org/issue13030 #13025: mimetypes should read the rule file using UTF-8, not the local http://bugs.python.org/issue13025 #13024: cgitb uses stdout encoding http://bugs.python.org/issue13024 #13023: argparse should allow displaying argument default values in ad http://bugs.python.org/issue13023 #13019: bytearrayobject.c: refleak http://bugs.python.org/issue13019 #13018: dictobject.c: refleak http://bugs.python.org/issue13018 #13017: pyexpat.c: refleak http://bugs.python.org/issue13017 #13016: selectmodule.c: refleak http://bugs.python.org/issue13016 #13015: _collectionsmodule.c: refleak http://bugs.python.org/issue13015 #13011: Frozen programs require the original build directory in order http://bugs.python.org/issue13011 #12984: XML NamedNodeMap ( attribName in NamedNodeMap fails ) http://bugs.python.org/issue12984 #12983: byte string literals with invalid hex escape codes raise Value http://bugs.python.org/issue12983 Most recent 15 issues waiting for review (15) ============================================= #13032: h2py.py can fail with UnicodeDecodeError http://bugs.python.org/issue13032 #13031: [PATCH] small speed-up for tarfile.py when unzipping tarballs http://bugs.python.org/issue13031 #13025: mimetypes should read the rule file using UTF-8, not the local http://bugs.python.org/issue13025 #13024: cgitb uses stdout encoding http://bugs.python.org/issue13024 #13018: dictobject.c: refleak http://bugs.python.org/issue13018 #13017: pyexpat.c: refleak http://bugs.python.org/issue13017 #13016: selectmodule.c: refleak http://bugs.python.org/issue13016 #13015: _collectionsmodule.c: refleak http://bugs.python.org/issue13015 #13012: Allow keyword argument in str.splitlines() http://bugs.python.org/issue13012 #13001: test_socket.testRecvmsgTrunc failure on FreeBSD 7.2 buildbot http://bugs.python.org/issue13001 #12991: Python 64-bit build on HP Itanium - Executable built successfu http://bugs.python.org/issue12991 #12989: Consistently handle path separator in Py_GetPath on Windows http://bugs.python.org/issue12989 #12986: Using getrandbits() in uuid.uuid4() is faster and more readabl http://bugs.python.org/issue12986 #12985: Check signed arithmetic overflow in ./configure http://bugs.python.org/issue12985 #12981: rewrite multiprocessing (senfd|recvfd) in Python http://bugs.python.org/issue12981 Top 10 most discussed issues (10) ================================= #12981: rewrite multiprocessing (senfd|recvfd) in Python http://bugs.python.org/issue12981 11 msgs #12943: tokenize: add python -m tokenize support back http://bugs.python.org/issue12943 8 msgs #12991: Python 64-bit build on HP Itanium - Executable built successfu http://bugs.python.org/issue12991 8 msgs #12729: Python lib re cannot handle Unicode properly due to narrow/wid http://bugs.python.org/issue12729 7 msgs #12955: urllib.request example should use "with ... as:" http://bugs.python.org/issue12955 7 msgs #13000: unhandled exception at install http://bugs.python.org/issue13000 7 msgs #13012: Allow keyword argument in str.splitlines() http://bugs.python.org/issue13012 6 msgs #12998: Memory leak with CTypes Structure http://bugs.python.org/issue12998 5 msgs #13026: Dis module - documentation of MAKE_FUNCTION http://bugs.python.org/issue13026 5 msgs #11816: Refactor the dis module to provide better building blocks for http://bugs.python.org/issue11816 4 msgs Issues closed (31) ================== #11037: State of PEP 382 or How does distutils2 handle namespaces? http://bugs.python.org/issue11037 closed by eric.araujo #11701: email.parser.BytesParser().parse() closes file argument http://bugs.python.org/issue11701 closed by sdaoden #11913: sdist refuses README.rst http://bugs.python.org/issue11913 closed by eric.araujo #11935: MMDF/MBOX mailbox need utime http://bugs.python.org/issue11935 closed by sdaoden #12145: distutils2 should support README.rst http://bugs.python.org/issue12145 closed by eric.araujo #12395: packaging remove fails under Windows http://bugs.python.org/issue12395 closed by eric.araujo #12678: test_packaging and test_distutils failures under Windows http://bugs.python.org/issue12678 closed by eric.araujo #12785: list_distinfo_file is wrong http://bugs.python.org/issue12785 closed by eric.araujo #12931: xmlrpclib confuses unicode and string http://bugs.python.org/issue12931 closed by haypo #12936: armv5tejl segfaults: sched_setaffinity() vs. pthread_setaffini http://bugs.python.org/issue12936 closed by skrah #12938: html.escape docstring does not mention single quotes (') http://bugs.python.org/issue12938 closed by orsenthil #12958: test_socket failures on Mac OS X http://bugs.python.org/issue12958 closed by python-dev #12960: threading.Condition is not a class http://bugs.python.org/issue12960 closed by haypo #12961: itertools: unlabelled balls in boxes http://bugs.python.org/issue12961 closed by rhettinger #12972: Color prompt + readline http://bugs.python.org/issue12972 closed by terry.reedy #12976: add support for MirBSD platform http://bugs.python.org/issue12976 closed by loewis #12977: socket.socket.setblocking does not raise exception if no data http://bugs.python.org/issue12977 closed by georg.brandl #12994: cx_Oracle failed to load in newly build python 2.7.1 http://bugs.python.org/issue12994 closed by terry.reedy #12995: Different behaviours with <class str> between v3.1.2 and v3.2. http://bugs.python.org/issue12995 closed by benjamin.peterson #12996: multiprocessing.Connection endianness issue http://bugs.python.org/issue12996 closed by neologix #13002: peephole.c: unused parameter http://bugs.python.org/issue13002 closed by skrah #13003: Bug in equivalent code for itertools.izip_longest http://bugs.python.org/issue13003 closed by georg.brandl #13005: operator module docs include repeat http://bugs.python.org/issue13005 closed by python-dev #13006: bug in core python variable binding http://bugs.python.org/issue13006 closed by amaury.forgeotdarc #13007: gdbm 1.9 has new magic that whichdb does not recognize http://bugs.python.org/issue13007 closed by python-dev #13010: devguide doc: ./python.exe on OS X http://bugs.python.org/issue13010 closed by ezio.melotti #13021: Resource is not released before returning from the function http://bugs.python.org/issue13021 closed by barry #13022: _multiprocessing.recvfd() doesn't check that file descriptor w http://bugs.python.org/issue13022 closed by python-dev #13036: time format in logging is wrong http://bugs.python.org/issue13036 closed by vinay.sajip #793069: Add --remove-source option http://bugs.python.org/issue793069 closed by eric.araujo #1172711: long long support for array module http://bugs.python.org/issue1172711 closed by meador.inge

1 0

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #12931: xmlrpclib now encodes Unicode URI to ISO-8859-1, instead of
by Éric Araujo Sept. 23, 2011

Sept. 23, 2011

Hi Victor, > summary: > Issue #12931: xmlrpclib now encodes Unicode URI to ISO-8859-1, instead of > failing with a UnicodeDecodeError. > > diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py > --- a/Lib/test/test_xmlrpc.py > +++ b/Lib/test/test_xmlrpc.py > @@ -472,6 +472,9 @@ > # protocol error; provide additional information in test output > self.fail("%s\n%s" % (e, getattr(e, "headers", ""))) > > + def test_unicode_host(self): > + server = xmlrpclib.ServerProxy(u"http://%s:%d/RPC2"%(ADDR, PORT)) Spaces around the modulo operator would have been nice here. Readability counts :)

1 0

Re: [Python-Dev] cpython: Issue #1172711: Add 'long long' support to the array module.
by Georg Brandl Sept. 21, 2011

Sept. 21, 2011

Am 21.09.2011 15:25, schrieb Ezio Melotti: >> @@ -1205,6 +1214,18 @@ >> minitemsize = 4 >> tests.append(UnsignedLongTest) >> >> +(a)unittest.skipIf(not have_long_long, 'need long long support') > > I think this would read better with skipUnless and s/have/has/: > > @unittest.skipUnless(HAS_LONG_LONG, 'need long long support') I don't think so. "skip if not" reads pretty well for me, while I always have to think twice about "unless" -- may be a non-native- speaker thing. Georg

5 4

Packaging in Python 2 anyone ?
by Tarek Ziadé Sept. 21, 2011

Sept. 21, 2011

Hi all, I am lacking of time right now to finish an important task before 3.2 final is out: we need to release "packaging" as a standalone release under Python 2.x and Python 3.1, to gather as much feedback as we can from more people. Doing an automated conversion turned out to be a nightmare, and I was about to go ahead and maintain a fork of the packaging package, with the few modules that are needed (sysconfig, etc) within a standalone release. I am looking for someone that has some free time and that is willing to lead this work. 3.2 can go out without this work of course, but it would be *much* better to have that feedback If you are interested, please let me know. Cheers Tarek -- Tarek Ziadé | http://ziade.org

11 29

Re: [Python-Dev] [Python-checkins] cpython: Issue #1172711: Add 'long long' support to the array module.
by Stefan Krah Sept. 21, 2011

Sept. 21, 2011

Ezio Melotti <ezio.melotti(a)gmail.com> wrote: >> +(a)unittest.skipIf(not have_long_long, 'need long long support') > > I think this would read better with skipUnless and s/have/has/: > > @unittest.skipUnless(HAS_LONG_LONG, 'need long long support') skipUnless() is perhaps a bit cleaner, but have_long_long is pretty established elsewhere (for example in pyport.h). Stefan Krah

2 1