Mailman 3 January 2010 - Python-checkins

Daily py3k reference leaks (r77825): sum=0
by solipsis＠pitrou.net 30 Jan '10

30 Jan '10

py3k results for svn r77825 (hg cset c4bc5f8dc883) -------------------------------------------------- test_os leaked [-23, 23, 0] references, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/py3k/refleaks/reflogzcL5gQ', '-x', 'test_httpservers']

1 0

r77833 - python/branches/release31-maint
by r.david.murray 29 Jan '10

29 Jan '10

Author: r.david.murray Date: Fri Jan 29 23:29:35 2010 New Revision: 77833 Log: Recorded merge of revisions 77831 via svnmerge from svn+ssh://pythondev@svn.python.org/python/branches/py3k ................ r77831 | r.david.murray | 2010-01-29 17:17:58 -0500 (Fri, 29 Jan 2010) | 9 lines Merged revisions 77828 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r77828 | r.david.murray | 2010-01-29 14:35:39 -0500 (Fri, 29 Jan 2010) | 2 lines Fix typo in assertSequenceEqual docstring. ........ ................ Modified: python/branches/release31-maint/ (props changed)

1 0

r77832 - python/branches/release31-maint/Lib/unittest.py
by r.david.murray 29 Jan '10

29 Jan '10

Author: r.david.murray Date: Fri Jan 29 23:26:45 2010 New Revision: 77832 Log: Fix typo in assertSequenceEqual docstring. Modified: python/branches/release31-maint/Lib/unittest.py Modified: python/branches/release31-maint/Lib/unittest.py ============================================================================== --- python/branches/release31-maint/Lib/unittest.py (original) +++ python/branches/release31-maint/Lib/unittest.py Fri Jan 29 23:26:45 2010 @@ -692,7 +692,7 @@ def assertSequenceEqual(self, seq1, seq2, msg=None, seq_type=None): """An equality assertion for ordered sequences (like lists and tuples). - For the purposes of this function, a valid orderd sequence type is one + For the purposes of this function, a valid ordered sequence type is one which can be indexed, has a length, and has an equality operator. Args:

1 0

r77831 - in python/branches/py3k: Lib/unittest/case.py
by r.david.murray 29 Jan '10

29 Jan '10

Author: r.david.murray Date: Fri Jan 29 23:17:58 2010 New Revision: 77831 Log: Merged revisions 77828 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r77828 | r.david.murray | 2010-01-29 14:35:39 -0500 (Fri, 29 Jan 2010) | 2 lines Fix typo in assertSequenceEqual docstring. ........ Modified: python/branches/py3k/ (props changed) python/branches/py3k/Lib/unittest/case.py Modified: python/branches/py3k/Lib/unittest/case.py ============================================================================== --- python/branches/py3k/Lib/unittest/case.py (original) +++ python/branches/py3k/Lib/unittest/case.py Fri Jan 29 23:17:58 2010 @@ -518,7 +518,7 @@ def assertSequenceEqual(self, seq1, seq2, msg=None, seq_type=None): """An equality assertion for ordered sequences (like lists and tuples). - For the purposes of this function, a valid orderd sequence type is one + For the purposes of this function, a valid ordered sequence type is one which can be indexed, has a length, and has an equality operator. Args:

1 0

r77830 - python/branches/release26-maint
by r.david.murray 29 Jan '10

29 Jan '10

Author: r.david.murray Date: Fri Jan 29 23:12:01 2010 New Revision: 77830 Log: Blocked revisions 77828 via svnmerge ........ r77828 | r.david.murray | 2010-01-29 14:35:39 -0500 (Fri, 29 Jan 2010) | 2 lines Fix typo in assertSequenceEqual docstring. ........ Modified: python/branches/release26-maint/ (props changed)

1 0

r77829 - in peps/trunk: pep-3147-1.dia pep-3147-1.png pep-3147.txt
by barry.warsaw 29 Jan '10

29 Jan '10

Author: barry.warsaw Date: Fri Jan 29 21:37:07 2010 New Revision: 77829 Log: PEP 3147, PYC Repository Directories, Warsaw Added: peps/trunk/pep-3147-1.dia (contents, props changed) peps/trunk/pep-3147-1.png (contents, props changed) peps/trunk/pep-3147.txt Added: peps/trunk/pep-3147-1.dia ============================================================================== Binary file. No diff available. Added: peps/trunk/pep-3147-1.png ============================================================================== Binary file. No diff available. Added: peps/trunk/pep-3147.txt ============================================================================== --- (empty file) +++ peps/trunk/pep-3147.txt Fri Jan 29 21:37:07 2010 @@ -0,0 +1,379 @@ +PEP: 3147 +Title: PYC Repository Directories +Version: $Revision$ +Last-Modified: $Date$ +Author: Barry Warsaw <barry(a)python.org> +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 2009-12-16 +Python-Version: 3.2 +Post-History: + + +Abstract +======== + +This PEP describes an extension to Python's import mechanism which +improves sharing of Python source code files among multiple installed +different versions of the Python interpreter. It does this by +allowing many different byte compilation files (.pyc files) to be +co-located with the Python source file (.py file). The extension +described here can also be used to support different Python +compilation caches, such as JIT output that may be produced by an +Unladen Swallow [1]_ enabled C Python. + + +Rationale +========= + +Linux distributions such as Ubuntu [2]_ and Debian [3]_ provide more +than one Python version at the same time to their users. For example, +Ubuntu 9.10 Karmic Koala can install Python 2.5, 2.6, and 3.1, with +Python 2.6 being the default. + +In order to ease the burden on operating system packagers for these +distributions, the distribution packages do not contain Python version +numbers [4]_; they are shared across all Python versions installed on +the system. Putting Python version numbers in the packages would be a +maintenance nightmare, since all the packages - *and their +dependencies* - would have to be updated every time a new Python +release was added or removed from the distribution. Because of the +sheer number of packages available, this amount of work is infeasible. + +For pure Python modules, sharing is possible because upstream +maintainers typically support multiple versions of Python in a source +compatible way. In practice though, it is well known that pyc files +are not compatible across Python major releases. A reading of +import.c [5]_ in the Python source code proves that within recent +memory, every new CPython major release has bumped the pyc magic +number. + +Even C extensions can be source compatible across multiple versions of +Python. Compiled extension modules are usually not compatible though, +and PEP 384 [6]_ has been proposed to address this by defining a +stable ABI for extension modules. + +Because the distributions cannot share pyc files, elaborate mechanisms +have been developed to put the resulting pyc files in non-shared +locations while the source code is still shared. Examples include the +symlink-based Debian regimes python-support [7]_ and python-central +[8]_. These approaches make for much more complicated, fragile, +inscrutable, and fragmented policies for delivering Python +applications to a wide range of users. Arguably more users get Python +from their operating system vendor than from upstream tarballs. Thus, +solving this pyc sharing problem for CPython is a high priority for +such vendors. + +This PEP proposes a solution to this problem. + + +Proposal +======== + +Python's import machinery is extended to search for byte code cache +files in a directory co-located with the source file, but with an +extension 'pyr'. The pyr directory contains individual files with the +cached byte compilation of the source code, identical to current pyc +and pyo files. The files inside the pyr directory retain their file +extensions, but the base name is replaced by the hexlified [10]_ magic +number of the Python version the byte code is compatible with. + +The file extension pyr was chosen because 'r' is a mnemonic for +'repository', and there appears to be no prior uses of the extension +[9]_. + +For example, a module `foo` with source code in `foo.py` and byte +compiled with Python 2.5, Python 2.6, Python 2.6 `-O`, Python 2.6 +`-U`, and Python 3.1 would have the following file system layout:: + + foo.py + foo.pyr/ + f2b30a0d.pyc # Python 2.5 + f2d10a0d.pyc # Python 2.6 + f2d10a0d.pyo # Python 2.6 -O + f2d20a0d.pyc # Python 2.6 -U + 0c4f0a0d.pyc # Python 3.1 + + +Python behavior +=============== + +When Python searches for a module to import (say `foo`), it may find +one of several situations. As per current Python rules, the term +"matching pyc" means that the magic number matches the current +interpreter's magic number, and the source file is not newer than the +`pyc` file. + +When Python finds a `foo.py` file for which no `foo.pyc` file or +`foo.pyr` directory exists, Python will by default load the `foo.py` +file and write a `foo.pyc` file next to the source file. This is +unchanged from current behavior. + +When the Python executable is given a `-R` flag, or the environment +variable `$PYTHONPYR` is set, then Python will create a `foo.pyr` +directory and write a `pyc` file to that directory with the hexlified +magic number as the base name. + +If during import, Python finds an existing `pyc` file but no `pyr` +directory, and the `$PYTHONPYR` environment variable is not set, then +the `pyc` file is loaded as normal and no `pyr` directory is created. + +If during import, Python finds a `pyr` directory with a matching `pyc` +file, *regardless of whether `$PYTHONPYR` is set or not*, then +`foo.pyr/<magic>.pyc` is loaded and import completes successfully. +Thus a matching `pyc` file inside a `pyr` directory always takes +precedence over a sibling `pyc` file. + +If during import, Python finds a `pyr` directory that does not contain +a matching `pyc` file, and no sibling `foo.pyc` file exists, Python +will load the source file and write a sibling `foo.pyc` file, unless +the `-R` flag is given in which case a `foo.pyr/<magic>.pyc` file will +be written. + +Here is a flowchart illustrating the rules. + +.. image:: pep-3147-1.png + :scale: 75 + + +Effects on non-conforming Python versions +========================================= + +Python implementations which don't know anything about `pyr` +directories will ignore them. This means that they will read and +write `pyc` files as usual. A conforming implementation will still +prefer any existing `foo.pyr/<magic>.pyc` file over an existing +sibling `pyc` file. + +The one possible conflicting state is where a sibling `pyc` file +exists, but its magic number does not match. + +In the default case, when Python finds a `pyc` file with a +non-matching magic number, it simply overwrites the `pyc` file with +the new byte code and magic number. In the absence of the `-R` flag, +this remains unchanged. When the `-R` flag was given, the +non-matching sibling `pyc` file is ignored - it is neither removed nor +overwritten - and a `foo.pyr/<magic>.pyc` file is written instead. + + +Implementation strategy +======================= + +This feature is targeted for Python 3.2, solving the problem for those +and all future versions. It may be back-ported to Python 2.7. +Vendors are free to backport the changes to earlier distributions as +they see fit. + + +Alternatives +============ + +PEP 304 +------- + +There is some overlap between the goals of this PEP and PEP 304 [12]_, +which has been withdrawn. However PEP 304 would allow a user to +create a shadow file system hierarchy in which to store `pyc` files. +This concept of a shadow hierarchy for `pyc` files could be used to +satisfy the aims of this PEP. Although the PEP 304 does not indicate +why it was withdrawn, shadow directories have a number of problems. +The location of the shadow `pyc` files would not be easily discovered +and would depend on the proper and consistent use of the +`$PYTHONBYTECODE` environment variable both by the system and by end +users. There are also global implications, meaning that while the +system might want to shadow `pyc` files, users might not want to, but +the PEP defines only an all-or-nothing approach. + +As an example of the problem, a common (though fragile) Python idiom +for locating data files is to do something like this:: + + from os import dirname, join + import foo.bar + data_file = join(dirname(foo.bar.__file__), 'my.dat') + +This would be problematic since `foo.bar.__file__` will give the +location of the `pyc` file in the shadow directory, and it may not be +possible to find the `my.dat` file relative to the source directory +from there. + +On the other hand, PEP 999 keeps all byte code artifacts co-located +with the source file. Some adjustment will have to be made for the +fact that the `pyc` file lives in a subdirectory. For example, in +current Python, when you import a module, its `__file__` attribute +points to its `pyc` file. A package's `__file__` points to the `pyc` +file for its `__init__.py`. E.g.:: + + >>> import foo + >>> foo.__file__ + 'foo.pyc' + # baz is a package + >>> import baz + >>> baz.__file__ + 'baz/__init__.pyc' + +The implementation of PEP 999 would have to ensure that the same +directory level is returned from `__file__` as it does without the +`pyr` directory, so that the common idiom above continues to work:: + + >>> import foo + >>> foo.__file__ + 'foo.pyr' + # baz is a package + >>> import baz + >>> baz.__file__ + 'baz/__init__.pyr' + +Note that some existing Python code only checks for `.py` and `.pyc` +file extensions (and possibly `.pyo`). These would have to be +extended to also check for `.pyr` extensions. + + +Fat byte compilation files +-------------------------- + +An earlier version of this PEP described "fat" Python byte code files. +These files would contain the equivalent of multiple `pyc` files in a +single `pyf` file, with a lookup table keyed off the appropriate magic +number. This was an extensible file format so that the first 5 +parallel Python implementations could be supported fairly efficiently, +but with extension lookup tables available to scale `pyf` byte code +objects as large as necessary. + +The fat byte compilation files were fairly complex, so the current +simplification of using directories was suggested. + + +Multiple file extensions +------------------------ + +The PEP author also considered an approach where multiple thin byte +compiled files lived in the same place, but used different file +extensions to designate the Python version. E.g. foo.pyc25, +foo.pyc26, foo.pyc31 etc. This was rejected because of the clutter +involved in writing so many different files. The multiple extension +approach makes it more difficult (and an ongoing task) to update any +tools that are dependent on the file extension. + + +Open questions +============== + +* Are there any concurrency issues added by this PEP, above those that + already exist? For example, what if two Python processes attempt to + write the same `<magic>.pyc` file? Is that any different than two + Python processes trying to write to the same `foo.pyc` file? + Current thinking is that there isn't since the exclusive open + mechanism currently used, will still be used to open `pyc` files + inside a `pyr` directory. + +* How do the imp [13]_ and importlib [14]_ modules need to be updated + to conform to the `pyr` directories? + +* What about `py` source files that are compatible with most but not + all installed Python versions. We might need a way to say "this py + file should be hidden from Python versions X.Y or earlier". There + are three options: + + - Use file system tricks to only share py files that are actually + sharable in all installed Python versions (e.g. different search + directories for Python X.Y and Python X.Z). + - Introduce Python syntax that is legal before __future__ imports + and is evaluated to determine if the py file is compatible, + raising an `ImportError('no module named foo')` if not. + - Add an optional metadata file co-located with the py file that + declares which Python versions it is compatible with. + + How does this requirement interact with PEP 382 namespace packages [15]_? + +* Are there any opportunities for also sharing extension modules + (.so/.dll files) in a `pyr` directory? + +* Would a moratorium on byte code changes, similar to the language + moratorium described in PEP 3003 [16]_ be a better approach to + pursue, and would that solve the problem for vendors? At the time + of this writing, PEP 3003 is silent on the issue. + + +Reference implementation +======================== + +TBD + + +References +========== + +.. [1] PEP 3146 + +.. [2] Ubuntu: <http://www.ubuntu.com> + +.. [3] Debian: <http://www.debian.org> + +.. [4] Debian Python Policy: + http://www.debian.org/doc/packaging-manuals/python-policy/ + +.. [5] import.c: + http://svn.python.org/view/python/branches/py3k/Python/import.c?view=markup + +.. [6] PEP 384 + +.. [7] python-support: + http://wiki.debian.org/DebianPythonFAQ#Whatispython-support.3F + +.. [8] python-central: + http://wiki.debian.org/DebianPythonFAQ#Whatispython-central.3F + +.. [9] http://www.filesuffix.com/?m=search&e=pyr&submit=Search + +.. [10] binascii.hexlify(): + http://www.python.org/doc/current/library/binascii.html#binascii.hexlify + +.. [11] The marshal module: + http://www.python.org/doc/current/library/marshal.html + +.. [12] PEP 304: + +.. [13] imp: http://www.python.org/doc/current/library/imp.html + +.. [14] importlib: http://docs.python.org/3.1/library/importlib.html + +.. [15] PEP 382 + +.. [16] PEP 3003 + + +ACKNOWLEDGMENTS +=============== + +Barry Warsaw's original idea was for fat Python byte code files. +Martin von Loewis reviewed an early draft of the PEP and suggested the +simplification to store traditional `pyc` and `pyo` files in a +directory. Many other people reviewed early versions of this PEP and +provided useful feedback including: + +* David Malcolm +* Josselin Mouette +* Matthias Klose +* Michael Hudson +* Michael Vogt +* Piotr Ożarowski +* Scott Kitterman +* Toshio Kuratomi + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End:

1 0

r77828 - python/trunk/Lib/unittest/case.py
by r.david.murray 29 Jan '10

29 Jan '10

Author: r.david.murray Date: Fri Jan 29 20:35:39 2010 New Revision: 77828 Log: Fix typo in assertSequenceEqual docstring. Modified: python/trunk/Lib/unittest/case.py Modified: python/trunk/Lib/unittest/case.py ============================================================================== --- python/trunk/Lib/unittest/case.py (original) +++ python/trunk/Lib/unittest/case.py Fri Jan 29 20:35:39 2010 @@ -505,7 +505,7 @@ def assertSequenceEqual(self, seq1, seq2, msg=None, seq_type=None): """An equality assertion for ordered sequences (like lists and tuples). - For the purposes of this function, a valid orderd sequence type is one + For the purposes of this function, a valid ordered sequence type is one which can be indexed, has a length, and has an equality operator. Args:

1 0

r77827 - python/branches/release31-maint
by r.david.murray 29 Jan '10

29 Jan '10

Author: r.david.murray Date: Fri Jan 29 19:01:56 2010 New Revision: 77827 Log: Blocked revisions 77825 via svnmerge ................ r77825 | r.david.murray | 2010-01-29 12:55:47 -0500 (Fri, 29 Jan 2010) | 10 lines Merged revisions 77815 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r77815 | r.david.murray | 2010-01-28 16:16:33 -0500 (Thu, 28 Jan 2010) | 3 lines Change error report when the object passed to suite.addTest is not callable to include the repr of the invalid object. ........ ................ Modified: python/branches/release31-maint/ (props changed)

1 0

r77826 - python/branches/release26-maint
by r.david.murray 29 Jan '10

29 Jan '10

Author: r.david.murray Date: Fri Jan 29 18:58:58 2010 New Revision: 77826 Log: Blocked revisions 77815 via svnmerge ........ r77815 | r.david.murray | 2010-01-28 16:16:33 -0500 (Thu, 28 Jan 2010) | 3 lines Change error report when the object passed to suite.addTest is not callable to include the repr of the invalid object. ........ Modified: python/branches/release26-maint/ (props changed)

1 0

r77825 - in python/branches/py3k: Lib/unittest/suite.py
by r.david.murray 29 Jan '10

29 Jan '10

Author: r.david.murray Date: Fri Jan 29 18:55:47 2010 New Revision: 77825 Log: Merged revisions 77815 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r77815 | r.david.murray | 2010-01-28 16:16:33 -0500 (Thu, 28 Jan 2010) | 3 lines Change error report when the object passed to suite.addTest is not callable to include the repr of the invalid object. ........ Modified: python/branches/py3k/ (props changed) python/branches/py3k/Lib/unittest/suite.py Modified: python/branches/py3k/Lib/unittest/suite.py ============================================================================== --- python/branches/py3k/Lib/unittest/suite.py (original) +++ python/branches/py3k/Lib/unittest/suite.py Fri Jan 29 18:55:47 2010 @@ -40,7 +40,7 @@ def addTest(self, test): # sanity checks if not hasattr(test, '__call__'): - raise TypeError("the test to add must be callable") + raise TypeError("{} is not callable".format(repr(test))) if isinstance(test, type) and issubclass(test, (case.TestCase, TestSuite)): raise TypeError("TestCases and TestSuites must be instantiated "

1 0