Mailman 3 October 2011 - Python-Dev

Packaging and binary distributions for Python 3.3
by Paul Moore Oct. 17, 2011

Oct. 17, 2011

I see that the Packaging documentation is now more complete (at least at docs.python.org) - I don't know if it's deemed fully complete yet, but I scanned the documentation and "Installing Python Projects" looks pretty much converted (and very good!!), but "Distributing Python Projects" still has quite a lot of distutils-related text in, and I need to read more deeply to understand if that's because it remains unchanged, or if it is still to be updated. But one thing struck me - the "Installing Python Projects" document talks about source distributions, but not much about binary distributions. On Windows, binary distributions are significantly more important than on Unix, because not all users have easy access to a compiler, and more importantly, C library dependencies can be difficult to build, hard to set up, and generally a pain to deal with. The traditional solution was always bdist_wininst installers, and with the advent of setuptools binary eggs started to become common. I've noticed that since pip became more widely used, with its focus on source builds, binary eggs seemed to fade away somewhat. I don't know what format packaging favours. The problem when Python 3.3 comes out is that bdist_wininst/bdist_msi installers do not interact well with pysetup. And if native virtual environment support becomes part of Python 3.3, they won't work well there either (they don't deal well with today's virtualenv, for that matter). So there will be a need for a pysetup-friendly binary format. I assume that the egg format will fill this role - or is that not the case? What is the current thinking on binary distribution formats for Python 3.3? The main reason I am asking is that I would like to write an article (or maybe a series of articles) for Python Insider, introducing the new packaging facilities from the point of view of an end user with straightforward needs (whether a package user just looking to manage a set of installed packages, or a module author who just wants to publish his code in a form that satisfies as many people as possible). What I'd hope to do is, as well as showing people all the nice things they can expect to see in Python 3.3, to also start package authors thinking about what they need to do to support their users under the new system. If we get the message out early, and make people aware of the benefits of the new end user tools, then I'm hoping more authors will see the advantage of switching to the new format rather than just sticking with bdist_xxx because "it's always worked". I suspect I should (re-)join the distutils SIG and take this discussion there. But honestly, I'm not sure I have the time - the traffic was always fairly high, and the number of relevant posts for a casual observer was quite low. So even if that's the right place to go, some pointers to some "high spots" to get me up to speed on the current state of affairs would help. Thanks, Paul.

10 53

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #13156: revert changeset f6feed6ec3f9, which was only relevant for native
by Victor Stinner Oct. 16, 2011

Oct. 16, 2011

Le mercredi 12 octobre 2011 21:07:33, charles-francois.natali a écrit : > changeset: 72897:ee4fe16d9b48 > branch: 2.7 > parent: 69635:f6feed6ec3f9 > user: Charles-François Natali <neologix(a)free.fr> > date: Wed Oct 12 21:07:54 2011 +0200 > summary: > Issue #13156: revert changeset f6feed6ec3f9, which was only relevant for > native TLS implementations, and fails with the ad-hoc TLS implementation > when a thread doesn't have an auto thread state (e.g. a thread created > outside of Python calling into a subinterpreter). > > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -61,10 +61,6 @@ > Library > ------- > > -- Issue #10517: After fork(), reinitialize the TLS used by the > PyGILState_* - APIs, to avoid a crash with the pthread implementation in > RHEL 5. Patch - by Charles-François Natali. You should restore this NEWS entry and add a new one to say that the patch has been reverted. Victor

4 3

check for PyUnicode_READY look backwards
by Amaury Forgeot d'Arc Oct. 16, 2011

Oct. 16, 2011

Hi, with the new Unicode API, there are many checks like: + if (PyUnicode_READY(*filename)) + goto handle_error; Every time I read it, I get it wrong: "If filename is ready, then fail" then I have to remember that the function returns either 0 or -1. I'd prefer it was written : if (PyUnicode_READY(*filename) < 0) because "< 0" clearly indicates an error condition. That's how all calls to PyType_Ready are written, for example. Am I the only one to be distracted by this idiom? -- Amaury Forgeot d'Arc

11 15

Identifier API
by "Martin v. Löwis" Oct. 15, 2011

Oct. 15, 2011

In benchmarking PEP 393, I noticed that many UTF-8 decode calls originate from C code with static strings, in particular PyObject_CallMethod. Many of such calls already have been optimized to cache a string object, however, PyObject_CallMethod remains unoptimized since it requires a char*. I find the ad-hoc approach of declaring and initializing variables inadequate, in particular since it is difficult to clean up all those string objects at interpreter shutdown. I propose to add an explicit API to deal with such identifiers. With this API, tmp = PyObject_CallMethod(result, "update", "O", other); would be replaced with PyObject *tmp; Py_identifier(update); ... tmp = PyObject_CallMethodId(result, &PyId_update, "O", other); Py_identifier expands to a struct typedef struct Py_Identifier { struct Py_Identifier *next; const char* string; PyObject *object; } Py_Identifier; string will be initialized by the compiler, next and object on first use. The new API for that will be PyObject* PyUnicode_FromId(Py_Identifier*); PyObject* PyObject_CallMethodId(PyObject*, Py_Identifier*, char*, ...); PyObject* PyObject_GetAttrId(PyObject*, Py_Identifier*); int PyObject_SetAttrId(PyObject*, Py_Identifier*, PyObject*); int PyObject_HasAttrId(PyObject*, Py_Identifier*); I have micro-benchmarked this; for import time d={} i=d.items() t=time.time() for _ in range(10**6): i | d print(time.time()-t) I get a speed-up of 30% (notice that "i | d" invokes the above PyObject_CallMethod call). Regards, Martin

10 27

Summary of Python tracker Issues
by Python tracker Oct. 14, 2011

Oct. 14, 2011

ACTIVITY SUMMARY (2011-10-07 - 2011-10-14) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3077 (+25) closed 21884 (+31) total 24961 (+56) Open issues with patches: 1315 Issues opened (42) ================== #11085: expose _abcoll as collections.abc http://bugs.python.org/issue11085 reopened by georg.brandl #12967: IDLE RPC Proxy for standard IO streams lacks 'errors' attribut http://bugs.python.org/issue12967 reopened by ned.deily #13124: Add "Running a Build Slave" page to the devguide http://bugs.python.org/issue13124 opened by eric.snow #13125: test_all_project_files() expected failure http://bugs.python.org/issue13125 opened by barry #13126: find() slower than rfind() http://bugs.python.org/issue13126 opened by pitrou #13127: xml.dom.Attr.name is not labeled as read-only http://bugs.python.org/issue13127 opened by dillona #13128: httplib debuglevel on CONNECT doesn't print response headers http://bugs.python.org/issue13128 opened by Matt.Spear #13131: FD leak in urllib2 http://bugs.python.org/issue13131 opened by Valery.Khamenya #13132: distutils sends non-RFC compliant HTTP request http://bugs.python.org/issue13132 opened by mitchellh #13133: FD leaks in ZipFile.read(), ZipFile.extract() and also using e http://bugs.python.org/issue13133 opened by Valery.Khamenya #13139: multiprocessing.map skips finally blocks http://bugs.python.org/issue13139 opened by illicitonion #13140: ThreadingMixIn.daemon_threads is not honored when parent is da http://bugs.python.org/issue13140 opened by flox #13141: get rid of old threading API in the examples http://bugs.python.org/issue13141 opened by flox #13143: os.path.islink documentation is ambiguous http://bugs.python.org/issue13143 opened by Garen #13144: Global Module Index link in the offline documentation is incor http://bugs.python.org/issue13144 opened by graemeglass #13146: Writing a pyc file is not atomic http://bugs.python.org/issue13146 opened by pitrou #13147: Multiprocessing Pool.map_async() does not have an error_callba http://bugs.python.org/issue13147 opened by Jakub.Gedeon #13149: optimization for append-only StringIO http://bugs.python.org/issue13149 opened by pitrou #13150: Most of Python's startup time is sysconfig http://bugs.python.org/issue13150 opened by pitrou #13151: pysetup3 run bdist_wininst fails http://bugs.python.org/issue13151 opened by vinay.sajip #13152: textwrap: support custom tabsize http://bugs.python.org/issue13152 opened by jfeuerstein #13153: IDLE crash with unicode bigger than 0xFFFF http://bugs.python.org/issue13153 opened by JBernardo #13156: _PyGILState_Reinit assumes auto thread state will always exist http://bugs.python.org/issue13156 opened by grahamd #13157: Build Python outside the source directory http://bugs.python.org/issue13157 opened by haypo #13160: Rename install_dist to install http://bugs.python.org/issue13160 opened by eric.araujo #13161: problems with help() documentation of __i*__ operators http://bugs.python.org/issue13161 opened by eli.bendersky #13163: `port` and `host` are confused in `_get_socket http://bugs.python.org/issue13163 opened by cool-RR #13164: importing rlcompleter module writes a control sequence in stdo http://bugs.python.org/issue13164 opened by valva #13165: Integrate stringbench in the Tools directory http://bugs.python.org/issue13165 opened by pitrou #13166: Implement packaging.database.Distribution.__str__ http://bugs.python.org/issue13166 opened by eric.araujo #13167: Add get_metadata to packaging http://bugs.python.org/issue13167 opened by eric.araujo #13168: Python 2.6 having trouble finding modules when invoked via a s http://bugs.python.org/issue13168 opened by RandyGalbraith #13169: Regular expressions with 0 to 65536 repetitions raises Overflo http://bugs.python.org/issue13169 opened by techmaurice #13170: distutils2 test failures http://bugs.python.org/issue13170 opened by eric.araujo #13171: Bug in file.read(), can access unknown data. http://bugs.python.org/issue13171 opened by Alexander.Steppke #13172: pysetup run --list-commands fails with a traceback http://bugs.python.org/issue13172 opened by pmoore #13173: Default values for string.Template http://bugs.python.org/issue13173 opened by nitupho #13174: test_os failures on Fedora 15: listxattr() returns ['security. http://bugs.python.org/issue13174 opened by haypo #13175: packaging uses wrong line endings in RECORD files on Windows http://bugs.python.org/issue13175 opened by pmoore #13177: Avoid chained exceptions in lru_cache http://bugs.python.org/issue13177 opened by ezio.melotti #13178: Need tests for Unicode handling in install_distinfo and instal http://bugs.python.org/issue13178 opened by eric.araujo #13179: IDLE uses common tkinter variables across all editor windows http://bugs.python.org/issue13179 opened by serwy Most recent 15 issues with no replies (15) ========================================== #13179: IDLE uses common tkinter variables across all editor windows http://bugs.python.org/issue13179 #13178: Need tests for Unicode handling in install_distinfo and instal http://bugs.python.org/issue13178 #13170: distutils2 test failures http://bugs.python.org/issue13170 #13168: Python 2.6 having trouble finding modules when invoked via a s http://bugs.python.org/issue13168 #13166: Implement packaging.database.Distribution.__str__ http://bugs.python.org/issue13166 #13164: importing rlcompleter module writes a control sequence in stdo http://bugs.python.org/issue13164 #13161: problems with help() documentation of __i*__ operators http://bugs.python.org/issue13161 #13160: Rename install_dist to install http://bugs.python.org/issue13160 #13152: textwrap: support custom tabsize http://bugs.python.org/issue13152 #13147: Multiprocessing Pool.map_async() does not have an error_callba http://bugs.python.org/issue13147 #13144: Global Module Index link in the offline documentation is incor http://bugs.python.org/issue13144 #13141: get rid of old threading API in the examples http://bugs.python.org/issue13141 #13140: ThreadingMixIn.daemon_threads is not honored when parent is da http://bugs.python.org/issue13140 #13128: httplib debuglevel on CONNECT doesn't print response headers http://bugs.python.org/issue13128 #13126: find() slower than rfind() http://bugs.python.org/issue13126 Most recent 15 issues waiting for review (15) ============================================= #13179: IDLE uses common tkinter variables across all editor windows http://bugs.python.org/issue13179 #13177: Avoid chained exceptions in lru_cache http://bugs.python.org/issue13177 #13174: test_os failures on Fedora 15: listxattr() returns ['security. http://bugs.python.org/issue13174 #13167: Add get_metadata to packaging http://bugs.python.org/issue13167 #13163: `port` and `host` are confused in `_get_socket http://bugs.python.org/issue13163 #13157: Build Python outside the source directory http://bugs.python.org/issue13157 #13156: _PyGILState_Reinit assumes auto thread state will always exist http://bugs.python.org/issue13156 #13152: textwrap: support custom tabsize http://bugs.python.org/issue13152 #13150: Most of Python's startup time is sysconfig http://bugs.python.org/issue13150 #13149: optimization for append-only StringIO http://bugs.python.org/issue13149 #13146: Writing a pyc file is not atomic http://bugs.python.org/issue13146 #13133: FD leaks in ZipFile.read(), ZipFile.extract() and also using e http://bugs.python.org/issue13133 #13132: distutils sends non-RFC compliant HTTP request http://bugs.python.org/issue13132 #13131: FD leak in urllib2 http://bugs.python.org/issue13131 #13128: httplib debuglevel on CONNECT doesn't print response headers http://bugs.python.org/issue13128 Top 10 most discussed issues (10) ================================= #13150: Most of Python's startup time is sysconfig http://bugs.python.org/issue13150 14 msgs #12602: Missing cross-references in Doc/using http://bugs.python.org/issue12602 12 msgs #12436: Missing items in installation/setup instructions http://bugs.python.org/issue12436 11 msgs #13156: _PyGILState_Reinit assumes auto thread state will always exist http://bugs.python.org/issue13156 10 msgs #13146: Writing a pyc file is not atomic http://bugs.python.org/issue13146 9 msgs #6715: xz compressor support http://bugs.python.org/issue6715 8 msgs #7833: bdist_wininst installers fail to load extensions built with Is http://bugs.python.org/issue7833 8 msgs #3902: Packages containing only extension modules have to contain __i http://bugs.python.org/issue3902 7 msgs #6164: [AIX] Patch to correct the AIX C/C++ linker argument used for http://bugs.python.org/issue6164 7 msgs #1673007: urllib2 requests history + HEAD support http://bugs.python.org/issue1673007 7 msgs Issues closed (27) ================== #9100: test_sysconfig fails (test_user_similar) http://bugs.python.org/issue9100 closed by eric.araujo #10653: test_time test_strptime fails on windows http://bugs.python.org/issue10653 closed by haypo #11254: distutils doesn't byte-compile .py files to __pycache__ during http://bugs.python.org/issue11254 closed by eric.araujo #12192: Doc that collection mutation methods return item or None http://bugs.python.org/issue12192 closed by python-dev #12386: packaging fails in install_distinfo when writing RESOURCES http://bugs.python.org/issue12386 closed by eric.araujo #12427: packaging register fails because "POST data should be bytes" http://bugs.python.org/issue12427 closed by eric.araujo #12555: PEP 3151 implementation http://bugs.python.org/issue12555 closed by pitrou #13025: mimetypes should read the rule file using UTF-8, not the local http://bugs.python.org/issue13025 closed by haypo #13029: test_strptime fails on Windows 7 french http://bugs.python.org/issue13029 closed by haypo #13075: PEP-0001 contains dead links http://bugs.python.org/issue13075 closed by ezio.melotti #13114: check -r fails with non-ASCII unicode long_description http://bugs.python.org/issue13114 closed by eric.araujo #13129: bad argument exceptions observed in AST http://bugs.python.org/issue13129 closed by rmtew #13130: test_gdb: attempt to dereference a generic pointer http://bugs.python.org/issue13130 closed by pitrou #13134: speed up finding of one-character strings http://bugs.python.org/issue13134 closed by pitrou #13135: Using type() as a constructor doesn't support new class keywor http://bugs.python.org/issue13135 closed by benjamin.peterson #13136: speed-up conversion between unicode widths http://bugs.python.org/issue13136 closed by pitrou #13137: from __future__ import division breaks ad hoc numeric types http://bugs.python.org/issue13137 closed by benjamin.peterson #13138: ElementTree's Element.iter() lacks versionadded http://bugs.python.org/issue13138 closed by ezio.melotti #13142: Add support for other HTTP methods in urllib.request http://bugs.python.org/issue13142 closed by ezio.melotti #13145: Documentation of PyNumber_ToBase() wrong http://bugs.python.org/issue13145 closed by mark.dickinson #13148: simple bug in mmap size check http://bugs.python.org/issue13148 closed by pitrou #13154: pep-0000.txt doesn't build anymore http://bugs.python.org/issue13154 closed by orsenthil #13155: Optimize finding the max character width http://bugs.python.org/issue13155 closed by pitrou #13158: tarfile.TarFile.getmembers misses some entries http://bugs.python.org/issue13158 closed by lars.gustaebel #13159: _io.FileIO uses a quadratic-time buffer growth algorithm http://bugs.python.org/issue13159 closed by nadeem.vawda #13162: Trying to install a binary extension as a resource file causes http://bugs.python.org/issue13162 closed by eric.araujo #13176: Broken link in bugs.rst http://bugs.python.org/issue13176 closed by eric.araujo

1 0

Re: [Python-Dev] [Python-checkins] cpython: Add a comment explaining this heuristic.
by Éric Araujo Oct. 14, 2011

Oct. 14, 2011

Hi Antoine, > changeset: 701b2e0e6f3f > user: Antoine Pitrou <solipsis(a)pitrou.net> > date: Thu Oct 13 18:07:37 2011 +0200 > summary: > Add a comment explaining this heuristic. > > diff --git a/Objects/stringlib/fastsearch.h b/Objects/stringlib/fastsearch.h > --- a/Objects/stringlib/fastsearch.h > +++ b/Objects/stringlib/fastsearch.h > @@ -115,6 +115,9 @@ > unsigned char needle; > needle = p[0] & 0xff; > #if STRINGLIB_SIZEOF_CHAR > 1 > + /* If looking for a multiple of 256, we'd have two > + many false positives looking for the '\0' byte in UCS2 > + and UCS4 representations. */ I guess this should read “too many”, not “two”. Cheers

1 0

Re: [Python-Dev] cpython: Use identifier API for PyObject_GetAttrString.
by Antoine Pitrou Oct. 13, 2011

Oct. 13, 2011

On Thu, 13 Oct 2011 20:05:06 +0200 "Martin v. Löwis" <martin(a)v.loewis.de> wrote: > > - In Modules/_json.c, line 1126, _Py_identifier(strict) is > > declared but not used, and there are 5 other possible replacements. > > Antoine reverted this in 8ed6a627a834. I think I started doing them, > then noticed that this is an initializer, so it's likely not called > that often. That's what I thought too. There didn't seem to be much point in optimizing that code. Regards Antoine.

1 0

Re: [Python-Dev] cpython: Optimize findchar() for PyUnicode_1BYTE_KIND: use memchr and memrchr
by Antoine Pitrou Oct. 12, 2011

Oct. 12, 2011

On Thu, 13 Oct 2011 01:17:29 +0200 victor.stinner <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/e5bd48b43a58 > changeset: 72903:e5bd48b43a58 > user: Victor Stinner <victor.stinner(a)haypocalc.com> > date: Thu Oct 13 00:18:12 2011 +0200 > summary: > Optimize findchar() for PyUnicode_1BYTE_KIND: use memchr and memrchr Can't we simply reuse the stringlib here?

2 1

Re: [Python-Dev] [Python-checkins] peps: Mark PEP accepted.
by Benjamin Peterson Oct. 12, 2011

Oct. 12, 2011

Isn't now final? 2011/10/12 antoine.pitrou <python-checkins(a)python.org>: > http://hg.python.org/peps/rev/f50f0e14c774 > changeset: 3962:f50f0e14c774 > parent: 3959:2e1f0462a847 > user: Antoine Pitrou <solipsis(a)pitrou.net> > date: Thu Oct 13 02:01:21 2011 +0200 > summary: > Mark PEP accepted. > > files: > pep-3151.txt | 10 +++++----- > 1 files changed, 5 insertions(+), 5 deletions(-) > > > diff --git a/pep-3151.txt b/pep-3151.txt > --- a/pep-3151.txt > +++ b/pep-3151.txt > @@ -3,13 +3,12 @@ > Version: $Revision$ > Last-Modified: $Date$ > Author: Antoine Pitrou <solipsis(a)pitrou.net> > -Status: Draft > +Status: Accepted > Type: Standards Track > Content-Type: text/x-rst > Created: 2010-07-21 > Python-Version: 3.3 > Post-History: > -Resolution: TBD > > > Abstract > @@ -507,9 +506,10 @@ > Implementation > ============== > > -A reference implementation is available in > -http://hg.python.org/features/pep-3151/ in branch ``pep-3151``. It is > -also tracked on the bug tracker at http://bugs.python.org/issue12555. > +The reference implementation has been integrated into Python 3.3. > +It was formerly developed in http://hg.python.org/features/pep-3151/ in > +branch ``pep-3151``, and also tracked on the bug tracker at > +http://bugs.python.org/issue12555. > It has been successfully tested on a variety of systems: Linux, Windows, > OpenIndiana and FreeBSD buildbots. > > > -- > Repository URL: http://hg.python.org/peps > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -- Regards, Benjamin

1 0

Documentation strategy for PEP 3151
by Antoine Pitrou Oct. 12, 2011

Oct. 12, 2011

Hello, I'd like some advice on what the best path is in cases such as: A :exc:`socket.error` is raised for errors from the call to :func:`inet_ntop`. Should I replace "socket.error" with "OSError" (knowing that the former is now an alias of the latter), or leave "socket.error" so that people have less surprises when running their code with a previous Python version? Regards Antoine.

5 4