Mailman 3 April 2011 - Python-Dev

Pass possibly imcompatible options to distutil's ccompiler
by Lukas Lueg 14 Apr '11

14 Apr '11

Hi, I'm the maintainer of Pyrit (http://pyrit.googlecode.com) and recently checked in some code that uses the AES-NI intrinsics found in GCC 4.4+. I'm looking for a way how to build the python-extension using distutils in a sane way and could not get an answer from the distutils-people about that. The enable the intrinsics, one must pass '-maes' and '-mpclmul' as commandline-arguments to gcc, e.g. through extra_compile_args. This is not always safe to do as previous versions of GCC do not support these options and cause cc to fail with an error. Such platforms are not uncommon, e.g. XCode 3.2 on MacOS is shipped with gcc 4.2. I fail to see how to determine in advance what compiler distutils will use and what version that compiler has. Therefor I see two options: - Try to build a small pseudo-extension with the flags enabled, watch for exceptions and only enable the extra_compile_args on the real extension if the build succeeds - Override the build_ext-command with another class and override build_extension. Try to build the extension and, if a CompilerError is thrown, remove '-maes' and '-mpclmul' from extra_compile_args. Try again and re-raise possible CompilerErrors now. The first option seems rather bogus so I'm currently going with the second option. After all, this leaves me with the best chance of enabling the AES-NI-code on compatible machines (no false-negatives with some kind of auto-detection) and not having people being unable to compile it at all (false-positives, resulting in final compiler errors). The downside is that visible error messages are printed to stderr from the first call to build_ext.build_extension if AES-NI is actually not supported. Any other ideas on how to solve this in a better way? Best regards Lukas

5 6

Re: [Python-Dev] cpython (merge default -> default): merge from push conflict.
by Georg Brandl 13 Apr '11

13 Apr '11

On 13.04.2011 03:43, senthil.kumaran wrote: > http://hg.python.org/cpython/rev/a4d1a3e0f7bd > changeset: 69306:a4d1a3e0f7bd > parent: 69305:35b16d49c0b1 > parent: 69299:c8d075051e88 > user: Senthil Kumaran <orsenthil(a)gmail.com> > date: Wed Apr 13 09:38:51 2011 +0800 > summary: > merge from push conflict. Hi, this message is not quite correct -- there is no conflict involved. You're just merging two heads on the same branch in order to have only one head in the master repo. Georg

2 1

Re: [Python-Dev] cpython: Fix 64-bit safety issue in BZ2Compressor and BZ2Decompressor.
by Georg Brandl 13 Apr '11

13 Apr '11

On 12.04.2011 23:05, nadeem.vawda wrote: > http://hg.python.org/cpython/rev/0010cc5f22d4 > changeset: 69275:0010cc5f22d4 > user: Nadeem Vawda <nadeem.vawda(a)gmail.com> > date: Tue Apr 12 23:02:42 2011 +0200 > summary: > Fix 64-bit safety issue in BZ2Compressor and BZ2Decompressor. > > files: > Lib/test/test_bz2.py | 36 +++++++++++++++++++++++++++++++- > Modules/_bz2module.c | 33 +++++++++++++++++++++-------- > 2 files changed, 59 insertions(+), 10 deletions(-) > > > diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py > --- a/Lib/test/test_bz2.py > +++ b/Lib/test/test_bz2.py Hi, for reviewing and "hg log" purposes it would be good to include a bit more information in the commit message, like where the safety issue was and what its potential implications were. (Of course that is different if there is an issue on the tracker to refer to; usually the issue is explained clearly in the issue.) Georg

2 1

Re: [Python-Dev] peps: Update PEP 399 to include comments from python-dev.
by Antoine Pitrou 13 Apr '11

13 Apr '11

On Tue, 12 Apr 2011 23:59:53 +0200 brett.cannon <python-checkins(a)python.org> wrote: > Technical details of > +the VM providing the accelerated code are allowed to differ as > +necessary, e.g., a class being a ``type`` when implemented in C. I don't understand what this means ("a class being a ``type`` when implemented in C"). > +If this test were to provide 100% branch coverage for > +``heapq.heappop()`` in the pure Python implementation then the > +accelerated C code would be allowed to be added to CPython's standard > +library. If it did not, then the test suite would need to be updated > +until 100% branch coverage was provided before the accelerated C code > +could be added. I really think that's a too strong requirement. We don't want to paralyze development until the stdlib gets 100% coverage in the tests suite. Regards Antoine.

9 12

Re: [Python-Dev] Mentor for Py3 benchmarking
by Senthil Kumaran 13 Apr '11

13 Apr '11

Hi Arc, I think you should forward this to python-dev. (CCed) There was a discussion on this over there, so someone should be definitely interested. On Tue, Apr 12, 2011 at 11:33:55AM -0400, Arc Riley wrote: > We have a number of students who proposed to port PyPy's benchmarking suite to > Python3 to run on speed.python.org, we don't have a mentor for these at the > moment. > > Would anyone here (pref previous GSoC mentor/student) like to take one of these > on? > > We have until Monday (4/18) to evaluate students, get patches/blogs/etc taken > care of, and all mentors assigned. If there are people here who want to mentor > talk to either Tarek (for packaging) or Martin v. Löwis (for python-core). If > you're an existing python-dev contributor we could especially use your help. -- Senthil

1 0

Hg question
by Alexander Belopolsky 13 Apr '11

13 Apr '11

I was preparing a commit to 3.2 and default branches and mistakenly used -m insread of -l commit option. As a result, I have $ hg out comparing with ssh://hg@hg.python.org/cpython searching for changes changeset: 69272:0bf1354fab6b branch: 3.2 parent: 69268:bfc586c558ed user: Alexander Belopolsky <alexander.belopolsky(a)gmail.com> date: Tue Apr 12 14:00:43 2011 -0400 summary: m.txt changeset: 69273:516ed700ce22 tag: tip parent: 69270:c26d015cbde8 parent: 69272:0bf1354fab6b user: Alexander Belopolsky <alexander.belopolsky(a)gmail.com> date: Tue Apr 12 14:02:22 2011 -0400 summary: m.txt I would like to replace m.txt in the summary with the content of the file m.txt. I tried to use the recipe [1], but qimport fails: $ hg qimport -r 69272:tip abort: cannot import merge revision 69273 [1] http://stackoverflow.com/questions/623052/how-to-edit-incorrect-commit-mess… PS: This scenario seems to be a usability regression compared to SVN. SVN would actually warn me if I tried to use -m with a file name instead of a message and editing the commit log in SVN is fairly straightforward.

5 4

Re: [Python-Dev] cpython (3.1): Fix Issue11703 - urllib2.geturl() does not return correct url when the original
by Antoine Pitrou 13 Apr '11

13 Apr '11

On Wed, 13 Apr 2011 01:26:12 +0200 senthil.kumaran <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/3f240a1cd245 > changeset: 69284:3f240a1cd245 > branch: 3.1 > parent: 69277:707078ca0a77 > user: Senthil Kumaran <orsenthil(a)gmail.com> > date: Wed Apr 13 07:01:19 2011 +0800 > summary: > Fix Issue11703 - urllib2.geturl() does not return correct url when the original url contains #fragment. Patch Contribution by Santoso Wijaya. > > files: > Lib/test/test_urllib.py | 10 ++++++++++ > Lib/test/test_urllib2.py | 15 ++++++++++++++- > Lib/test/test_urllib2net.py | 2 +- > Lib/urllib/request.py | 9 ++++++--- > 4 files changed, 31 insertions(+), 5 deletions(-) Can you add a Misc/NEWS entry? Thank you Antoine.

2 1

Re: [Python-Dev] cpython: Fix 64-bit safety issue in BZ2Compressor and BZ2Decompressor.
by Antoine Pitrou 12 Apr '11

12 Apr '11

On Tue, 12 Apr 2011 23:05:40 +0200 nadeem.vawda <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/0010cc5f22d4 > changeset: 69275:0010cc5f22d4 > user: Nadeem Vawda <nadeem.vawda(a)gmail.com> > date: Tue Apr 12 23:02:42 2011 +0200 > summary: > Fix 64-bit safety issue in BZ2Compressor and BZ2Decompressor. > > files: > Lib/test/test_bz2.py | 36 +++++++++++++++++++++++++++++++- > Modules/_bz2module.c | 33 +++++++++++++++++++++-------- > 2 files changed, 59 insertions(+), 10 deletions(-) Can you add a Misc/NEWS entry? Thank you Antoine.

1 0

Re: [Python-Dev] PEP 396, Module Version Numbers
by Laura Creighton 12 Apr '11

12 Apr '11

In a message of Tue, 12 Apr 2011 15:56:32 EDT, Barry Warsaw writes: <snip> >The Deriving section of the PEP is not the most important part of it, and > is >not making specific recommendations. If it's not clear that it's only >providing examples, or it's distracting, then maybe it's better off being >removed, cut down or rewritten. To me, at any rate, it read as a pretty important part. But the version I just read from http://www.python.org/dev/peps/pep-0396/ still has the re in it as well. Laura

2 1

PEP 396, Module Version Numbers
by Barry Warsaw 12 Apr '11

12 Apr '11

I just checked in PEP 396, Module Version Numbers. This is an informational PEP describing how to specify version numbers using the __version__ attribute. This has already made one round through distutils-sig so it's time to post it here. Comments welcome of course! Cheers, -Barry P.S. Yeah, I know the $Keyword$ strings here are wrong. I'm not sure what to do about them though. PEP: 396 Title: Module Version Numbers Version: $Revision: 65628 $ Last-Modified: $Date: 2008-08-10 09:59:20 -0400 (Sun, 10 Aug 2008) $ Author: Barry Warsaw <barry(a)python.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 2011-03-16 Post-History: 2011-04-05 Abstract ======== Given that it is useful and common to specify version numbers for Python modules, and given that different ways of doing this have grown organically within the Python community, it is useful to establish standard conventions for module authors to adhere to and reference. This informational PEP describes best practices for Python module authors who want to define the version number of their Python module. Conformance with this PEP is optional, however other Python tools (such as ``distutils2`` [1]_) may be adapted to use the conventions defined here. User Stories ============ Alice is writing a new module, called ``alice``, which she wants to share with other Python developers. ``alice`` is a simple module and lives in one file, ``alice.py``. Alice wants to specify a version number so that her users can tell which version they are using. Because her module lives entirely in one file, she wants to add the version number to that file. Bob has written a module called ``bob`` which he has shared with many users. ``bob.py`` contains a version number for the convenience of his users. Bob learns about the Cheeseshop [2]_, and adds some simple packaging using classic distutils so that he can upload *The Bob Bundle* to the Cheeseshop. Because ``bob.py`` already specifies a version number which his users can access programmatically, he wants the same API to continue to work even though his users now get it from the Cheeseshop. Carole maintains several namespace packages, each of which are independently developed and distributed. In order for her users to properly specify dependencies on the right versions of her packages, she specifies the version numbers in the namespace package's ``setup.py`` file. Because Carol wants to have to update one version number per package, she specifies the version number in her module and has the ``setup.py`` extract the module version number when she builds the *sdist* archive. David maintains a package in the standard library, and also produces standalone versions for other versions of Python. The standard library copy defines the version number in the module, and this same version number is used for the standalone distributions as well. Rationale ========= Python modules, both in the standard library and available from third parties, have long included version numbers. There are established de-facto standards for describing version numbers, and many ad-hoc ways have grown organically over the years. Often, version numbers can be retrieved from a module programmatically, by importing the module and inspecting an attribute. Classic Python distutils ``setup()`` functions [3]_ describe a ``version`` argument where the release's version number can be specified. PEP 8 [4]_ describes the use of a module attribute called ``__version__`` for recording "Subversion, CVS, or RCS" version strings using keyword expansion. In the PEP author's own email archives, the earliest example of the use of an ``__version__`` module attribute by independent module developers dates back to 1995. Another example of version information is the sqlite3 [5]_ library with its ``sqlite_version_info``, ``version``, and ``version_info`` attributes. It may not be immediately obvious which attribute contains a version number for the module, and which contains a version number for the underlying SQLite3 library. This informational PEP codifies established practice, and recommends standard ways of describing module version numbers, along with some use cases for when -- and when *not* -- to include them. Its adoption by module authors is purely voluntary; packaging tools in the standard library will provide optional support for the standards defined herein, and other tools in the Python universe may comply as well. Specification ============= #. In general, modules in the standard library SHOULD NOT have version numbers. They implicitly carry the version number of the Python release they are included in. #. On a case-by-case basis, standard library modules which are also released in standalone form for other Python versions MAY include a module version number when included in the standard library, and SHOULD include a version number when packaged separately. #. When a module includes a version number, it SHOULD be available in the ``__version__`` attribute on that module. #. For modules which are also packages, the module namespace SHOULD include the ``__version__`` attribute. #. For modules which live inside a namespace package, the sub-package name SHOULD include the ``__version__`` attribute. The namespace module itself SHOULD NOT include its own ``__version__`` attribute. #. The ``__version__`` attribute's value SHOULD be a string. #. Module version numbers SHOULD conform to the normalized version format specified in PEP 386 [6]_. #. Module version numbers SHOULD NOT contain version control system supplied revision numbers, or any other semantically different version numbers (e.g. underlying library version number). #. Wherever a ``__version__`` attribute exists, a module MAY also include a ``__version_info__`` attribute, containing a tuple representation of the module version number, for easy comparisons. #. ``__version_info__`` SHOULD be of the format returned by PEP 386's ``parse_version()`` function. #. The ``version`` attribute in a classic distutils ``setup.py`` file, or the PEP 345 [7]_ ``Version`` metadata field SHOULD be derived from the ``__version__`` field, or vice versa. Examples ======== Retrieving the version number from a third party package:: >>> import bzrlib >>> bzrlib.__version__ '2.3.0' Retrieving the version number from a standard library package that is also distributed as a standalone module:: >>> import email >>> email.__version__ '5.1.0' Version numbers for namespace packages:: >>> import flufl.i18n >>> import flufl.enum >>> import flufl.lock >>> print flufl.i18n.__version__ 1.0.4 >>> print flufl.enum.__version__ 3.1 >>> print flufl.lock.__version__ 2.1 >>> import flufl >>> flufl.__version__ Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute '__version__' >>> Deriving ======== Module version numbers can appear in at least two places, and sometimes more. For example, in accordance with this PEP, they are available programmatically on the module's ``__version__`` attribute. In a classic distutils ``setup.py`` file, the ``setup()`` function takes a ``version`` argument, while the distutils2 ``setup.cfg`` file has a ``version`` key. The version number must also get into the PEP 345 metadata, preferably when the *sdist* archive is built. It's desirable for module authors to only have to specify the version number once, and have all the other uses derive from this single definition. While there are any number of ways this could be done, this section describes one possible approach, for each scenario. Let's say Elle adds this attribute to her module file ``elle.py``:: __version__ = '3.1.1' Classic distutils ----------------- In classic distutils, the simplest way to add the version string to the ``setup()`` function in ``setup.py`` is to do something like this:: from elle import __version__ setup(name='elle', version=__version__) In the PEP author's experience however, this can fail in some cases, such as when the module uses automatic Python 3 conversion via the ``2to3`` program (because ``setup.py`` is executed by Python 3 before the ``elle`` module has been converted). In that case, it's not much more difficult to write a little code to parse the ``__version__`` from the file rather than importing it:: import re DEFAULT_VERSION_RE = re.compile(r'(?P<version>\d+\.\d(?:\.\d+)?)') def get_version(filename, pattern=None): if pattern is None: cre = DEFAULT_VERSION_RE else: cre = re.compile(pattern) with open(filename) as fp: for line in fp: if line.startswith('__version__'): mo = cre.search(line) assert mo, 'No valid __version__ string found' return mo.group('version') raise AssertionError('No __version__ assignment found') setup(name='elle', version=get_version('elle.py')) Distutils2 ---------- Because the distutils2 style ``setup.cfg`` is declarative, we can't run any code to extract the ``__version__`` attribute, either via import or via parsing. This PEP suggests a special key be added to the ``[metadata]`` section of the ``setup.cfg`` file to indicate "get the version from this file". Something like this might work:: [metadata] version-from-file: elle.py where ``parse`` means to use a parsing method similar to the above, on the file named after the colon. The exact recipe for doing this will be discussed in the appropriate distutils2 development forum. An alternative is to only define the version number in ``setup.cfg`` and use the ``pkgutil`` module [8]_ to make it available programmatically. E.g. in ``elle.py``:: from distutils2._backport import pkgutil __version__ = pkgutil.get_distribution('elle').metadata['version'] PEP 376 metadata ================ PEP 376 [9]_ defines a standard for static metadata, but doesn't describe the process by which this metadata gets created. It is highly desirable for the derived version information to be placed into the PEP 376 ``.dist-info`` metadata at build-time rather than install-time. This way, the metadata will be available for introspection even when the code is not installed. References ========== .. [1] Distutils2 documentation (http://distutils2.notmyidea.org/) .. [2] The Cheeseshop (Python Package Index) (http://pypi.python.org) .. [3] http://docs.python.org/distutils/setupscript.html .. [4] PEP 8, Style Guide for Python Code (http://www.python.org/dev/peps/pep-0008) .. [5] sqlite3 module documentation (http://docs.python.org/library/sqlite3.html) .. [6] PEP 386, Changing the version comparison module in Distutils (http://www.python.org/dev/peps/pep-0386/) .. [7] PEP 345, Metadata for Python Software Packages 1.2 (http://www.python.org/dev/peps/pep-0345/#version) .. [8] pkgutil - Package utilities (http://distutils2.notmyidea.org/library/pkgutil.html) .. [9] PEP 376, Database of Installed Python Distributions (http://www.python.org/dev/peps/pep-0376/) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

11 33