Mailman 3 August 2011 - Python-Dev

Re: [Python-Dev] [Python-checkins] peps: Add Alexandre's suggestions
by Nick Coghlan Aug. 16, 2011

Aug. 16, 2011

On Tue, Aug 16, 2011 at 11:30 AM, antoine.pitrou <python-checkins(a)python.org> wrote: > +Serializing "pseudo-global" objects > +----------------------------------- > + > +Objects which are not module-global, but should be treated in a similar > +fashion -- such as methods [4]_ or nested classes -- cannot currently be > +pickled (or, rather, unpickled) because the pickle protocol does not > +correctly specify how to retrieve them. One solution would be through the > +adjunction of a ``__namespace__`` (or ``__qualname__``) to all class and > +function objects, specifying the full "path" by which they can be retrieved. > +For globals, this would generally be ``"{}.{}".format(obj.__module__, obj.__name__)``. > +Then a new opcode can resolve that path and push the object on the stack, > +similarly to the GLOBAL opcode. > + I think this is the part that ties in with the pickle-related aspects for PEP 395 - using '__qualname__' would be one way to align a module's real name with where it should be retrieved from and where it's documentation lives (I like 'qualified name' as a term, too). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 4

Re: [Python-Dev] cpython: Add Py_RETURN_NOTIMPLEMENTED macro. Fixes #12724.
by Antoine Pitrou Aug. 15, 2011

Aug. 15, 2011

Le Thu, 11 Aug 2011 03:34:37 +0200, brian.curtin <python-checkins(a)python.org> a écrit : > http://hg.python.org/cpython/rev/77a65b078852 > changeset: 71809:77a65b078852 > parent: 71803:1b4fae183da3 > user: Brian Curtin <brian(a)python.org> > date: Wed Aug 10 20:05:21 2011 -0500 > summary: > Add Py_RETURN_NOTIMPLEMENTED macro. Fixes #12724. It would sound more useful to have a generic Py_RETURN() macro rather than some specific forms for each and every common object. Regards Antoine.

11 21

PEP 3154 - pickle protocol 4
by Antoine Pitrou Aug. 15, 2011

Aug. 15, 2011

Hello, This PEP is an attempt to foster a number of small incremental improvements in a future pickle protocol version. The PEP process is used in order to gather as many improvements as possible, because the introduction of a new protocol version should be a rare occurrence. Feel free to suggest any additions. Regards Antoine. http://www.python.org/dev/peps/pep-3154/ PEP: 3154 Title: Pickle protocol version 4 Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou <solipsis(a)pitrou.net> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2011-08-11 Python-Version: 3.3 Post-History: Resolution: TBD Abstract ======== Data serialized using the pickle module must be portable accross Python versions. It should also support the latest language features as well as implementation-specific features. For this reason, the pickle module knows about several protocols (currently numbered from 0 to 3), each of which appeared in a different Python version. Using a low-numbered protocol version allows to exchange data with old Python versions, while using a high-numbered protocol allows access to newer features and sometimes more efficient resource use (both CPU time required for (de)serializing, and disk size / network bandwidth required for data transfer). Rationale ========= The latest current protocol, coincidentally named protocol 3, appeared with Python 3.0 and supports the new incompatible features in the language (mainly, unicode strings by default and the new bytes object). The opportunity was not taken at the time to improve the protocol in other ways. This PEP is an attempt to foster a number of small incremental improvements in a future new protocol version. The PEP process is used in order to gather as many improvements as possible, because the introduction of a new protocol version should be a rare occurrence. Improvements in discussion ========================== 64-bit compatibility for large objects -------------------------------------- Current protocol versions export object sizes for various built-in types (str, bytes) as 32-bit ints. This forbids serialization of large data [1]_. New opcodes are required to support very large bytes and str objects. Native opcodes for sets and frozensets -------------------------------------- Many common built-in types (such as str, bytes, dict, list, tuple) have dedicated opcodes to improve resource consumption when serializing and deserializing them; however, sets and frozensets don't. Adding such opcodes would be an obvious improvement. Also, dedicated set support could help remove the current impossibility of pickling self-referential sets [2]_. Binary encoding for all opcodes ------------------------------- The GLOBAL opcode, which is still used in protocol 3, uses the so-called "text" mode of the pickle protocol, which involves looking for newlines in the pickle stream. Looking for newlines is difficult to optimize on a non-seekable stream, and therefore a new version of GLOBAL (BINGLOBAL?) could use a binary encoding instead. It seems that all other opcodes emitted when using protocol 3 already use binary encoding. Acknowledgments =============== (...) References ========== .. [1] "pickle not 64-bit ready": http://bugs.python.org/issue11564 .. [2] "Cannot pickle self-referencing sets": http://bugs.python.org/issue9269 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

5 5

Fwd: Mirroring Python repos to Bitbucket
by Doug Hellmann Aug. 15, 2011

Aug. 15, 2011

Charles McLaughlin of Atlassian has set up mirrors of the Mercurial repositories hosted on python.org as part of the ongoing infrastructure improvement work. These mirrors will give us a public fail-over repository in the event that hg.python.org goes offline unexpectedly, and also provide features such as RSS feeds of changes for users interested in monitoring the repository passively. Thank you, Charles for setting this up and Atlassian for hosting it! Doug Begin forwarded message: > From: Charles McLaughlin <cmclaughlin(a)atlassian.com> > Date: August 10, 2011 3:43:06 PM EDT > To: Jesse Noller <jnoller(a)gmail.com>, Doug Hellmann <doug.hellmann(a)gmail.com> > Subject: Mirroring Python repos to Bitbucket > > Hey, > > You guys expressed some interest in mirroring repos to Bitbucket a > couple weeks ago. I mentioned we mirror a few Python repos here: > > https://bitbucket.org/mirror/ > > But that doesn't cover everything from hg.python.org. So, I wrote a > little script that scrapes the Python HgWeb and mirrors everything to > Bitbucket. Here's the script in case you're curious: > > https://bitbucket.org/cmclaughlin/mirror-python-repos/ > > We're running the script hourly to keep the mirrors up to date. The > mirrored repos live under this URL: > > https://bitbucket.org/python_mirrors > > A few people here have mentioned "python_mirrors" is a strange name. I > can change that if you'd like. I don't have any better ideas though. > Also, anyone can fork my script if they see any room for improvement :) > > Please feel free to forward this email to mailing lists, etc to get the > word out. > > Regards, > Charles

4 4

Latest draft of PEP 399 (Pure Python/C Accelerator Module Compatibility Requirements)
by Brett Cannon Aug. 15, 2011

Aug. 15, 2011

While at a mini-PyPy sprint w/ Alex Gaynor of PyPy and Phil Jenvey of Jython, I decided to finally put the time in to update this PEP yet again. The biggest changes is that the 100% branch coverage requirement has been replaced with "comprehensive" coverage from the tests. I think we are all enough grown-ups to not have to specify anything tighter than this. I also added a paragraph in the Details section about using the abstract C APIs (e.g., PyObject_GetItem) over type-specific ones (e.g., PyList_GetItem) in order to be more supportive of duck typing from the Python code. I figure the "be API compatible" assumes this, but mentioning it doesn't hurt (and should help make Raymond less angry =). PEP: 399 Title: Pure Python/C Accelerator Module Compatibility Requirements Version: $Revision: 88219 $ Last-Modified: $Date: 2011-01-27 13:47:00 -0800 (Thu, 27 Jan 2011) $ Author: Brett Cannon <brett(a)python.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 04-Apr-2011 Python-Version: 3.3 Post-History: 04-Apr-2011, 12-Apr-2011, 17-Jul-2011 Abstract ======== The Python standard library under CPython contains various instances of modules implemented in both pure Python and C (either entirely or partially). This PEP requires that in these instances that the C code *must* pass the test suite used for the pure Python code so as to act as much as a drop-in replacement as reasonably possible (C- and VM-specific tests are exempt). It is also required that new C-based modules lacking a pure Python equivalent implementation get special permission to be added to the standard library. Rationale ========= Python has grown beyond the CPython virtual machine (VM). IronPython_, Jython_, and PyPy_ are all currently viable alternatives to the CPython VM. The VM ecosystem that has sprung up around the Python programming language has led to Python being used in many different areas where CPython cannot be used, e.g., Jython allowing Python to be used in Java applications. A problem all of the VMs other than CPython face is handling modules from the standard library that are implemented (to some extent) in C. Since other VMs do not typically support the entire `C API of CPython`_ they are unable to use the code used to create the module. Often times this leads these other VMs to either re-implement the modules in pure Python or in the programming language used to implement the VM itself (e.g., in C# for IronPython). This duplication of effort between CPython, PyPy, Jython, and IronPython is extremely unfortunate as implementing a module *at least* in pure Python would help mitigate this duplicate effort. The purpose of this PEP is to minimize this duplicate effort by mandating that all new modules added to Python's standard library *must* have a pure Python implementation _unless_ special dispensation is given. This makes sure that a module in the stdlib is available to all VMs and not just to CPython (pre-existing modules that do not meet this requirement are exempt, although there is nothing preventing someone from adding in a pure Python implementation retroactively). Re-implementing parts (or all) of a module in C (in the case of CPython) is still allowed for performance reasons, but any such accelerated code must pass the same test suite (sans VM- or C-specific tests) to verify semantics and prevent divergence. To accomplish this, the test suite for the module must have comprehensive coverage of the pure Python implementation before the acceleration code may be added. Details ======= Starting in Python 3.3, any modules added to the standard library must have a pure Python implementation. This rule can only be ignored if the Python development team grants a special exemption for the module. Typically the exemption will be granted only when a module wraps a specific C-based library (e.g., sqlite3_). In granting an exemption it will be recognized that the module will be considered exclusive to CPython and not part of Python's standard library that other VMs are expected to support. Usage of ``ctypes`` to provide an API for a C library will continue to be frowned upon as ``ctypes`` lacks compiler guarantees that C code typically relies upon to prevent certain errors from occurring (e.g., API changes). Even though a pure Python implementation is mandated by this PEP, it does not preclude the use of a companion acceleration module. If an acceleration module is provided it is to be named the same as the module it is accelerating with an underscore attached as a prefix, e.g., ``_warnings`` for ``warnings``. The common pattern to access the accelerated code from the pure Python implementation is to import it with an ``import *``, e.g., ``from _warnings import *``. This is typically done at the end of the module to allow it to overwrite specific Python objects with their accelerated equivalents. This kind of import can also be done before the end of the module when needed, e.g., an accelerated base class is provided but is then subclassed by Python code. This PEP does not mandate that pre-existing modules in the stdlib that lack a pure Python equivalent gain such a module. But if people do volunteer to provide and maintain a pure Python equivalent (e.g., the PyPy team volunteering their pure Python implementation of the ``csv`` module and maintaining it) then such code will be accepted. In those instances the C version is considered the reference implementation in terms of expected semantics. Any new accelerated code must act as a drop-in replacement as close to the pure Python implementation as reasonable. Technical details of the VM providing the accelerated code are allowed to differ as necessary, e.g., a class being a ``type`` when implemented in C. To verify that the Python and equivalent C code operate as similarly as possible, both code bases must be tested using the same tests which apply to the pure Python code (tests specific to the C code or any VM do not follow under this requirement). The test suite is expected to be extensive in order to verify expected semantics. Acting as a drop-in replacement also dictates that no public API be provided in accelerated code that does not exist in the pure Python code. Without this requirement people could accidentally come to rely on a detail in the accelerated code which is not made available to other VMs that use the pure Python implementation. To help verify that the contract of semantic equivalence is being met, a module must be tested both with and without its accelerated code as thoroughly as possible. As an example, to write tests which exercise both the pure Python and C accelerated versions of a module, a basic idiom can be followed:: import collections.abc from test.support import import_fresh_module, run_unittest import unittest c_heapq = import_fresh_module('heapq', fresh=['_heapq']) py_heapq = import_fresh_module('heapq', blocked=['_heapq']) class ExampleTest(unittest.TestCase): def test_heappop_exc_for_non_MutableSequence(self): # Raise TypeError when heap is not a # collections.abc.MutableSequence. class Spam: """Test class lacking many ABC-required methods (e.g., pop()).""" def __len__(self): return 0 heap = Spam() self.assertFalse(isinstance(heap, collections.abc.MutableSequence)) with self.assertRaises(TypeError): self.heapq.heappop(heap) class AcceleratedExampleTest(ExampleTest): """Test using the accelerated code.""" heapq = c_heapq class PyExampleTest(ExampleTest): """Test with just the pure Python code.""" heapq = py_heapq def test_main(): run_unittest(AcceleratedExampleTest, PyExampleTest) if __name__ == '__main__': test_main() If this test were to provide extensive coverage for ``heapq.heappop()`` in the pure Python implementation then the accelerated C code would be allowed to be added to CPython's standard library. If it did not, then the test suite would need to be updated until proper coverage was provided before the accelerated C code could be added. To also help with compatibility, C code should use abstract APIs on objects to prevent accidental dependence on specific types. For instance, if a function accepts a sequence then the C code should default to using `PyObject_GetItem()` instead of something like `PyList_GetItem()`. C code is allowed to have a fast path if the proper `PyList_Check()` is used, but otherwise APIs should work with any object that duck types to the proper interface instead of a specific type. Copyright ========= This document has been placed in the public domain. .. _IronPython: http://ironpython.net/ .. _Jython: http://www.jython.org/ .. _PyPy: http://pypy.org/ .. _C API of CPython: http://docs.python.org/py3k/c-api/index.html .. _sqlite3: http://docs.python.org/py3k/library/sqlite3.html

2 2

Backporting howto/pyporting to 2.7
by Éric Araujo Aug. 15, 2011

Aug. 15, 2011

Hi everyone, I think it would be useful to have the “Porting Python 2 Code to Python 3” HOWTO in the 2.7 docs, as I think that a lot of users consult the 2.7 docs. Is there any reason not to do it? Regards

2 1

Python 3.2.2rc1
by "Martin v. Löwis" Aug. 15, 2011

Aug. 15, 2011

On behalf of the Python development team, I'm happy to announce the first release candidate of the Python 3.2.2 maintenance release (3.2.2rc1). Python 3.2.2 fixes `a regression <http://bugs.python.org/12576>`_ in the ``urllib.request`` module that prevented opening many HTTP resources correctly with Python 3.2.1. Python 3.2 is a continuation of the efforts to improve and stabilize the Python 3.x line. Since the final release of Python 2.7, the 2.x line will only receive bugfixes, and new features are developed for 3.x only. Since PEP 3003, the Moratorium on Language Changes, is in effect, there are no changes in Python's syntax and built-in types in Python 3.2. Development efforts concentrated on the standard library and support for porting code to Python 3. Highlights are: * numerous improvements to the unittest module * PEP 3147, support for .pyc repository directories * PEP 3149, support for version tagged dynamic libraries * PEP 3148, a new futures library for concurrent programming * PEP 384, a stable ABI for extension modules * PEP 391, dictionary-based logging configuration * an overhauled GIL implementation that reduces contention * an extended email package that handles bytes messages * a much improved ssl module with support for SSL contexts and certificate hostname matching * a sysconfig module to access configuration information * additions to the shutil module, among them archive file support * many enhancements to configparser, among them mapping protocol support * improvements to pdb, the Python debugger * countless fixes regarding bytes/string issues; among them full support for a bytes environment (filenames, environment variables) * many consistency and behavior fixes for numeric operations For a more extensive list of changes in 3.2, see http://docs.python.org/3.2/whatsnew/3.2.html To download Python 3.2 visit: http://www.python.org/download/releases/3.2/ Please consider trying Python 3.2 with your code and reporting any bugs you may notice to: http://bugs.python.org/ Enjoy! -- Martin v. Löwis (on behalf of the entire python-dev team and 3.2's contributors)

2 1

GIL removal question
by Марк Коренберг Aug. 14, 2011

Aug. 14, 2011

Probably I want to re-invent a bicycle. I want developers to say me why we can not remove GIL in that way: 1. Remove GIL completely with all current logick. 2. Add it's own RW-locking to all mutable objects (like list or dict) 3. Add RW-locks to every context instance 4. use RW-locks when accessing members of object instances Only one reason, I see, not do that -- is performance of singlethreaded applications. Why not to fix locking functions for this 4 cases to stubs when only one thread present? For atomicity, locks may be implemented as this: For example for this source: -------------------------------- import threading def x(): i=1000 while i: i-- a = threading.Thread(target=x) b = threading.Thread(target=x) a.start() b.start() a.join() b.join() -------------------------------- in my case it will be fully parallel, as common object is not locked much (only global context when a.xxxx = yyyy executed). I think, performance of such code will be higher that using GIL. Other significant reason of not using my case, as I think, is a plenty of atomic processor instructions in each thread, which affect kernel performance. Also, I know about incompatibility my variant with existing code. In a summary: Please say clearly why, actually, my variant is not still implemented. Thanks. -- Segmentation fault

11 18

Re: [Python-Dev] [Python-checkins] cpython: Monotonic, not monotonous
by Nick Coghlan Aug. 14, 2011

Aug. 14, 2011

On Sun, Aug 14, 2011 at 9:53 AM, antoine.pitrou <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/0273d0734593 > changeset: 71862:0273d0734593 > user: Antoine Pitrou <solipsis(a)pitrou.net> > date: Sun Aug 14 01:51:52 2011 +0200 > summary: > Monotonic, not monotonous > > files: > Lib/test/pickletester.py | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) I dunno, I reckon systematically testing pickles could get pretty monotonous, too ;) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Python-Dev] cpython (3.2): Fix find command in makefile “funny” target
by Georg Brandl Aug. 13, 2011

Aug. 13, 2011

On 08/12/11 18:03, eric.araujo wrote: > http://hg.python.org/cpython/rev/1b818f3639ef > changeset: 71826:1b818f3639ef > branch: 3.2 > parent: 71823:8032ea4c3619 > user: Éric Araujo <merwok(a)netwok.org> > date: Wed Aug 10 02:01:32 2011 +0200 > summary: > Fix find command in makefile “funny” target > > files: > Makefile.pre.in | 4 ++-- > 1 files changed, 2 insertions(+), 2 deletions(-) > > > diff --git a/Makefile.pre.in b/Makefile.pre.in > --- a/Makefile.pre.in > +++ b/Makefile.pre.in > @@ -1283,7 +1283,7 @@ > > # Find files with funny names > funny: > - find $(DISTDIRS) \ > + find $(SUBDIRS) $(SUBDIRSTOO) \ > -name .svn -prune \ > -o -type d \ > -o -name '*.[chs]' \ > @@ -1313,7 +1313,7 @@ > -o -name .hgignore \ > -o -name .bzrignore \ > -o -name MANIFEST \ > - -o -print > + -print This actually broke the command; it only outputs "MANIFEST" now if present. The previous version is correct; please revert this part. Georg

1 0