Mailman 3 April 2011 - Python-Dev

Buildbots and faulthandler
by Victor Stinner April 20, 2011

April 20, 2011

Hi, The new faulthandler module is now fully functional and it has no more known issue. Its timeout feature is used on regrtest to dump the Python backtrace and exit if a test takes more than 1 hour. Using the regrtest timeout and faulthandler signal handlers (enable in regrtest), I started to collect tracebacks of all timeouts. Open issues: * test_threading.test_notify() on Windows http://bugs.python.org/issue11769 Not analyzed yet. I am unable to reproduce it in my VM. * … [View More]test_mmap.test_large_offset() on Mac OS X http://bugs.python.org/issue11779 May be related (and fixed) by issue #11277 which has a patch. * test_threading.test_3_join_in_forked_from_thread() on Ubuntu http://bugs.python.org/issue11870 Only seen once. * test_mmap.test_big_buffer() on Mac OS X (it's a crash, bus error) http://bugs.python.org/issue11277 The origin of the problem was already identified, but the trace proves that faulthandler is able to catch correctly SIGBUS ;-) * test_ttk_guionly on Mac OS X (bus error) http://bugs.python.org/issue5120 Same as #11277 (the origin of the problem was already identified) Closed issues: * test_io.test_interrupted_write_text() on FreeBSD http://bugs.python.org/issue11859 (there was already enough information without faulthandler) * test_threadsignals.test_signals() on Mac OS X http://bugs.python.org/issue11768 Race condition (deadlock). * test_multiprocessing.test_async_error_callback() on many OSes http://bugs.python.org/issue8428 Race condition. I'm proud of #11768 (because I fixed it). The bug was a deadlock. It is usually very hard to reproduce such issue (a deadlock) and without faulthandler, the only available information was the name of the file. With faulthandler, we have not only the name of the test function, but also the full traceback of the hang, but also the traceback of all other threads. Thanks to the faulthandler trace of #8428, with the traceback of all threads, Charles-Francois Natali was able to understand and fix another complex race condition in multiprocessing (at shutdown). I also fixed other issues (not using faulthandler) and so ALL OUR 3.X BUILDBOTS ARE GREEN! ... ok ok, except: - sparc Debian 3.x: offline since 21 days - PPC Leopard 3.x : "hg clean" fails with twisted.internet.error.ProcessExitedAlready, but I think that except this buildbot specific issue, it must be green - x86 Windows7 3.x: the master lost the connection with the slave on test_cmd_line, but it should be a sporadic problem Anyway, if you see a "Timeout (1:00:00)!" or "Fatal error" (with a traceback) issue on a buildbot, please open a new issue (if it doesn't exist, search a least the name of the test file). If you have other problems related to regrtest timeout or faulthandler, contact me or open an issue. Finally, I'm very happy to see that my faulthandler module was as useful as I expected: with more informations, we are now able to identify race conditions. I hope that we will fix all remaining threading, signal and subprocess race conditions! Victor [View Less]

4 3

Re: [Python-Dev] cpython: os.sendfile(): on Linux if offset parameter is passed as NULL we were
by Antoine Pitrou April 20, 2011

April 20, 2011

On Tue, 19 Apr 2011 09:47:21 +0200 giampaolo.rodola <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/8c49f7fbba1d > changeset: 69437:8c49f7fbba1d > user: Giampaolo Rodola' <g.rodola(a)gmail.com> > date: Tue Apr 19 09:47:16 2011 +0200 > summary: > os.sendfile(): on Linux if offset parameter is passed as NULL we were erroneously returning a (bytes_sent, None) tuple instead of bytes_sent Do we have tests for this? Regards Antoine.

3 2

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix a few hyphens in argparse.rst.
by Éric Araujo April 19, 2011

April 19, 2011

Hi, > summary: > Fix a few hyphens in argparse.rst. > - :synopsis: Command-line option and argument parsing library. > + :synopsis: Command-line option and argument-parsing library. I believe that change should be reverted. “argument parsing library” is a noun determined (qualified) by another noun itself determined by a noun, not by an adjective. > - followed by zero or one command-line args. When parsing the > command-line, if > + followed by zero or … [View More]

1 0

PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
by Brett Cannon April 19, 2011

April 19, 2011

At both the VM and language summits at PyCon this year, the issue of compatibility of the stdlib amongst the various VMs came up. Two issues came about in regards to modules that use C code. One is that code that comes in only as C code sucks for all other VMs that are not CPython since they all end up having to re-implement that module themselves. Two is that modules that have an accelerator module (e.g., heapq, warnings, etc.) can end up with compatibility options (sorry, Raymond, for picking … [View More]on heapq, but is was what bit the PyPy people most recently =). In lieu of all of this, here is a draft PEP to more clearly state the policy for the stdlib when it comes to C code. Since this has come up before and this was discussed so much at the summits I have gone ahead and checked this in so that even if this PEP gets rejected there can be a written record as to why. And before anyone asks, I have already run this past the lead devs of PyPy, Jython, and IronPython and they all support what this PEP proposes. And with the devs of the other VMs gaining push privileges there shouldn't be an added developer burden on everyone to make this PEP happen. ========================================================== PEP: 399 Title: Pure Python/C Accelerator Module Compatibiilty Requirements Version: $Revision: 88219 $ Last-Modified: $Date: 2011-01-27 13:47:00 -0800 (Thu, 27 Jan 2011) $ Author: Brett Cannon <brett(a)python.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 04-Apr-2011 Python-Version: 3.3 Post-History: Abstract ======== The Python standard library under CPython contains various instances of modules implemented in both pure Python and C. This PEP requires that in these instances that both the Python and C code *must* be semantically identical (except in cases where implementation details of a VM prevents it entirely). It is also required that new C-based modules lacking a pure Python equivalent implementation get special permissions to be added to the standard library. Rationale ========= Python has grown beyond the CPython virtual machine (VM). IronPython_, Jython_, and PyPy_ all currently being viable alternatives to the CPython VM. This VM ecosystem that has sprung up around the Python programming language has led to Python being used in many different areas where CPython cannot be used, e.g., Jython allowing Python to be used in Java applications. A problem all of the VMs other than CPython face is handling modules from the standard library that are implemented in C. Since they do not typically support the entire `C API of Python`_ they are unable to use the code used to create the module. Often times this leads these other VMs to either re-implement the modules in pure Python or in the programming language used to implement the VM (e.g., in C# for IronPython). This duplication of effort between CPython, PyPy, Jython, and IronPython is extremely unfortunate as implementing a module *at least* in pure Python would help mitigate this duplicate effort. The purpose of this PEP is to minimize this duplicate effort by mandating that all new modules added to Python's standard library *must* have a pure Python implementation _unless_ special dispensation is given. This makes sure that a module in the stdlib is available to all VMs and not just to CPython. Re-implementing parts (or all) of a module in C (in the case of CPython) is still allowed for performance reasons, but any such accelerated code must semantically match the pure Python equivalent to prevent divergence. To accomplish this, the pure Python and C code must be thoroughly tested with the *same* test suite to verify compliance. This is to prevent users from accidentally relying on semantics that are specific to the C code and are not reflected in the pure Python implementation that other VMs rely upon, e.g., in CPython 3.2.0, ``heapq.heappop()`` raises different exceptions depending on whether the accelerated C code is used or not:: from test.support import import_fresh_module c_heapq = import_fresh_module('heapq', fresh=['_heapq']) py_heapq = import_fresh_module('heapq', blocked=['_heapq']) class Spam: """Tester class which defines no other magic methods but __len__().""" def __len__(self): return 0 try: c_heapq.heappop(Spam()) except TypeError: # "heap argument must be a list" pass try: py_heapq.heappop(Spam()) except AttributeError: # "'Foo' object has no attribute 'pop'" pass This kind of divergence is a problem for users as they unwittingly write code that is CPython-specific. This is also an issue for other VM teams as they have to deal with bug reports from users thinking that they incorrectly implemented the module when in fact it was caused by an untested case. Details ======= Starting in Python 3.3, any modules added to the standard library must have a pure Python implementation. This rule can only be ignored if the Python development team grants a special exemption for the module. Typically the exemption would be granted only when a module wraps a specific C-based library (e.g., sqlite3_). In granting an exemption it will be recognized that the module will most likely be considered exclusive to CPython and not part of Python's standard library that other VMs are expected to support. Usage of ``ctypes`` to provide an API for a C library will continue to be frowned upon as ``ctypes`` lacks compiler guarantees that C code typically relies upon to prevent certain errors from occurring (e.g., API changes). Even though a pure Python implementation is mandated by this PEP, it does not preclude the use of a companion acceleration module. If an acceleration module is provided it is to be named the same as the module it is accelerating with an underscore attached as a prefix, e.g., ``_warnings`` for ``warnings``. The common pattern to access the accelerated code from the pure Python implementation is to import it with an ``import *``, e.g., ``from _warnings import *``. This is typically done at the end of the module to allow it to overwrite specific Python objects with their accelerated equivalents. This kind of import can also be done before the end of the module when needed, e.g., an accelerated base class is provided but is then subclassed by Python code. This PEP does not mandate that pre-existing modules in the stdlib that lack a pure Python equivalent gain such a module. But if people do volunteer to provide and maintain a pure Python equivalent (e.g., the PyPy team volunteering their pure Python implementation of the ``csv`` module and maintaining it) then such code will be accepted. Any accelerated code must be semantically identical to the pure Python implementation. The only time any semantics are allowed to be different are when technical details of the VM providing the accelerated code prevent matching semantics from being possible, e.g., a class being a ``type`` when implemented in C. The semantics equivalence requirement also dictates that no public API be provided in accelerated code that does not exist in the pure Python code. Without this requirement people could accidentally come to rely on a detail in the acclerated code which is not made available to other VMs that use the pure Python implementation. To help verify that the contract of semantic equivalence is being met, a module must be tested both with and without its accelerated code as thoroughly as possible. As an example, to write tests which exercise both the pure Python and C acclerated versions of a module, a basic idiom can be followed:: import collections.abc from test.support import import_fresh_module, run_unittest import unittest c_heapq = import_fresh_module('heapq', fresh=['_heapq']) py_heapq = import_fresh_module('heapq', blocked=['_heapq']) class ExampleTest(unittest.TestCase): def test_heappop_exc_for_non_MutableSequence(self): # Raise TypeError when heap is not a # collections.abc.MutableSequence. class Spam: """Test class lacking many ABC-required methods (e.g., pop()).""" def __len__(self): return 0 heap = Spam() self.assertFalse(isinstance(heap, collections.abc.MutableSequence)) with self.assertRaises(TypeError): self.heapq.heappop(heap) class AcceleratedExampleTest(ExampleTest): """Test using the acclerated code.""" heapq = c_heapq class PyExampleTest(ExampleTest): """Test with just the pure Python code.""" heapq = py_heapq def test_main(): run_unittest(AcceleratedExampleTest, PyExampleTest) if __name__ == '__main__': test_main() Thoroughness of the test can be verified using coverage measurements with branching coverage on the pure Python code to verify that all possible scenarios are tested using (or not using) accelerator code. Copyright ========= This document has been placed in the public domain. .. _IronPython: http://ironpython.net/ .. _Jython: http://www.jython.org/ .. _PyPy: http://pypy.org/ .. _C API of Python: http://docs.python.org/py3k/c-api/index.html .. _sqlite3: http://docs.python.org/py3k/library/sqlite3.html [View Less]

19 80

Python 2.6.7
by Barry Warsaw April 19, 2011

April 19, 2011

With Martin getting ready to release 2.5.6, I think it's time to prepare a 2.6.7 source-only security release. I'll work my way through the NEWS file and recent commits, but if there is anything that you know is missing from the 2.6 branch, please let me know. It would be especially helpful if there were bugs for any such issues. Cheers, -Barry

2 2

Re: [Python-Dev] cpython: #11731: simplify/enhance parser/generator API by introducing policy objects.
by Georg Brandl April 19, 2011

April 19, 2011

On 18.04.2011 20:00, r.david.murray wrote: > diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst > --- a/Doc/library/email.parser.rst > +++ b/Doc/library/email.parser.rst > @@ -112,8 +118,13 @@ > :class:`~email.message.Message` (see :mod:`email.message`). The factory will > be called without arguments. > > - .. versionchanged:: 3.2 > - Removed the *strict* argument that was deprecated in 2.4. > + The *policy* keyword … [View More]specifies a :mod:`~email.policy` object that controls a > + number of aspects of the parser's operation. The default policy maintains > + backward compatibility. > + > + .. versionchanged:: 3.3 > + Removed the *strict* argument that was deprecated in 2.4. Added the > + *policy* keyword. Hmm, so *strict* wasn't actually removed in 3.2? > @@ -187,12 +204,15 @@ > > .. currentmodule:: email > > -.. function:: message_from_string(s, _class=email.message.Message, strict=None) > +.. function:: message_from_string(s, _class=email.message.Message, *, \ > + policy=policy.default) > > Return a message object structure from a string. This is exactly equivalent to > - ``Parser().parsestr(s)``. Optional *_class* and *strict* are interpreted as > + ``Parser().parsestr(s)``. *_class* and *policy* are interpreted as > with the :class:`Parser` class constructor. > > + .. versionchanged:: removed *strict*, added *policy* > + The 3.3 version is missing here. Also, please always end version directive text with a period. > .. function:: message_from_bytes(s, _class=email.message.Message, strict=None) > > Return a message object structure from a byte string. This is exactly > @@ -200,21 +220,27 @@ > *strict* are interpreted as with the :class:`Parser` class constructor. > > .. versionadded:: 3.2 > + .. versionchanged:: 3.3 removed *strict*, added *policy* See above. > -.. function:: message_from_file(fp, _class=email.message.Message, strict=None) > +.. function:: message_from_file(fp, _class=email.message.Message, *, \ > + policy=policy.default) > > Return a message object structure tree from an open :term:`file object`. > - This is exactly equivalent to ``Parser().parse(fp)``. Optional *_class* > - and *strict* are interpreted as with the :class:`Parser` class constructor. > + This is exactly equivalent to ``Parser().parse(fp)``. *_class* > + and *policy* are interpreted as with the :class:`Parser` class constructor. > > -.. function:: message_from_binary_file(fp, _class=email.message.Message, strict=None) > + .. versionchanged:: 3.3 removed *strict*, added *policy* See above. > +.. function:: message_from_binary_file(fp, _class=email.message.Message, *, \ > + policy=policy.default) > > Return a message object structure tree from an open binary :term:`file > object`. This is exactly equivalent to ``BytesParser().parse(fp)``. > - Optional *_class* and *strict* are interpreted as with the :class:`Parser` > + *_class* and *policy* are interpreted as with the :class:`Parser` > class constructor. > > .. versionadded:: 3.2 > + .. versionchanged:: 3.3 removed *strict*, added *policy* See above. > --- /dev/null > +++ b/Doc/library/email.policy.rst > @@ -0,0 +1,179 @@ > +:mod:`email`: Policy Objects > +---------------------------- > + > +.. module:: email.policy > + :synopsis: Controlling the parsing and generating of messages This file should have a ".. versionadded:: 3.3" (without further content) here. > +The :mod:`email` package's prime focus is the handling of email messages as > +described by the various email and MIME RFCs. However, the general format of > +email messages (a block of header fields each consisting of a name followed by > +a colon followed by a value, the whole block followed by a blank line and an > +arbitrary 'body'), is a format that has found utility outside of the realm of > +email. Some of these uses conform fairly closely to the main RFCs, some do > +not. And even when working with email, there are times when it is desirable to > +break strict compliance with the RFCs. > + > +Policy objects are the mechanism used to provide the email package with the > +flexibility to handle all these disparate use cases, Looks like something is missing from this sentence :) [...] > +As an example, the following code could be used to read an email message from a > +file on disk and pass it to the system ``sendmail`` program on a ``unix`` > +system:: Should be Unix, not ``unix``. > + >>> from email import msg_from_binary_file > + >>> from email.generator import BytesGenerator > + >>> import email.policy > + >>> from subprocess import Popen, PIPE > + >>> with open('mymsg.txt', 'b') as f: > + >>> msg = msg_from_binary_file(f, policy=email.policy.mbox) > + >>> p = Popen(['sendmail', msg['To'][0].address], stdin=PIPE) > + >>> g = BytesGenerator(p.stdin, email.policy.policy=SMTP) That keyword arg doesn't look right. > + >>> g.flatten(msg) > + >>> p.stdin.close() > + >>> rc = p.wait() Also, if you put interactive prompts, please use them correctly ("..." prompt and one blank line for the with block). > +Some email package methods accept a *policy* keyword argument, allowing the > +policy to be overridden for that method. For example, the following code use "uses" > +the :meth:`email.message.Message.as_string` method to the *msg* object from the ^^^^^^ Something is missing around here. > +previous example and re-write it to a file using the native line separators for > +the platform on which it is running:: > + > + >>> import os > + >>> mypolicy = email.policy.Policy(linesep=os.linesep) > + >>> with open('converted.txt', 'wb') as f: > + ... f.write(msg.as_string(policy=mypolicy)) > + > +Policy instances are immutable, but they can be cloned, accepting the same > +keyword arguments as the class constructor and returning a new :class:`Policy` > +instance that is a copy of the original but with the specified attributes > +values changed. For example, the following creates an SMTP policy that will > +raise any defects detected as errors:: > + > + >>> strict_SMTP = email.policy.SMTP.clone(raise_on_defect=True) > + > +Policy objects can also be combined using the addition operator, producing a > +policy object whose settings are a combination of the non-default values of the > +summed objects:: > + > + >>> strict_SMTP = email.policy.SMTP + email.policy.strict Interesting API :) > +This operation is not commutative; that is, the order in which the objects are > +added matters. To illustrate:: > + > + >>> Policy = email.policy.Policy > + >>> apolicy = Policy(max_line_length=100) + Policy(max_line_length=80) > + >>> apolicy.max_line_length > + 80 > + >>> apolicy = Policy(max_line_length=80) + Policy(max_line_length=100) > + >>> apolicy.max_line_length > + 100 > + > + > +.. class:: Policy(**kw) > + > + The valid constructor keyword arguments are any of the attributes listed > + below. > + > + .. attribute:: max_line_length > + > + The maximum length of any line in the serialized output, not counting the > + end of line character(s). Default is 78, per :rfc:`5322`. A value of > + ``0`` or :const:`None` indicates that no line wrapping should be > + done at all. > + > + .. attribute:: linesep > + > + The string to be used to terminate lines in serialized output. The > + default is '\\n' because that's the internal end-of-line discipline used > + by Python, though '\\r\\n' is required by the RFCs. See `Policy > + Instances`_ for policies that use an RFC conformant linesep. Setting it > + to :attr:`os.linesep` may also be useful. These string constants are probably better off in code markup, i.e. ``'\n'``. > + .. attribute:: must_be_7bit > + > + If :const:`True`, data output by a bytes generator is limited to ASCII > + characters. If :const:`False` (the default), then bytes with the high > + bit set are preserved and/or allowed in certain contexts (for example, > + where possible a content transfer encoding of ``8bit`` will be used). > + String generators act as if ``must_be_7bit`` is `True` regardless of the > + policy in effect, since a string cannot represent non-ASCII bytes. Please use either :const:`True` or ``True``. > + .. attribute:: raise_on_defect > + > + If :const:`True`, any defects encountered will be raised as errors. If > + :const:`False` (the default), defects will be passed to the > + :meth:`register_defect` method. A short sentence that the following are methods would be nice. > + .. method:: handle_defect(obj, defect) > + > + *obj* is the object on which to register the defect. What kind of object is *obj*? > *defect* should be > + an instance of a subclass of :class:`~email.errors.Defect`. > + If :attr:`raise_on_defect` > + is ``True`` the defect is raised as an exception. Otherwise *obj* and > + *defect* are passed to :meth:`register_defect`. This method is intended > + to be called by parsers when they encounter defects, and will not be > + called by code that uses the email library unless that code is > + implementing an alternate parser. > + > + .. method:: register_defect(obj, defect) > + > + *obj* is the object on which to register the defect. *defect* should be > + a subclass of :class:`~email.errors.Defect`. This method is part of the > + public API so that custom ``Policy`` subclasses can implement alternate > + handling of defects. The default implementation calls the ``append`` > + method of the ``defects`` attribute of *obj*. > + > + .. method:: clone(obj, *kw): > + > + Return a new :class:`Policy` instance whose attributes have the same > + values as the current instance, except where those attributes are > + given new values by the keyword arguments. > + > + > +Policy Instances > +................ We're usually using "^^^^" for underlining this level of headings, but it's not really important. > +The following instances of :class:`Policy` provide defaults suitable for > +specific common application domains. Indentation switches to 4 spaces below here... > +.. data:: default > + > + An instance of :class:`Policy` with all defaults unchanged. > + > +.. data:: SMTP > + > + Output serialized from a message will conform to the email and SMTP > + RFCs. The only changed attribute is :attr:`linesep`, which is set to > + ``\r\n``. > + > +.. data:: HTTP > + > + Suitable for use when serializing headers for use in HTTP traffic. > + :attr:`linesep` is set to ``\r\n``, and :attr:`max_line_length` is set to > + :const:`None` (unlimited). > + > +.. data:: strict > + > + :attr:`raise_on_defect` is set to :const:`True`. Sorry for the long review. Georg [View Less]

2 2

Post from Mark Summerfield
by skip＠pobox.com April 18, 2011

April 18, 2011

Mark Summerfield responded to Martin's python-announce post. Rather than approving it I rejected it and forwarded it here. (I suppose I could have forwarded it directly to Martin, but that would have required that I recall or look up his email address...) Skip

1 0

Forward of moderated message
by python-announce-list-bounces＠python.org April 18, 2011

April 18, 2011

1 0

Status of json (simplejson) in cpython
by Sandro Tosi April 18, 2011

April 18, 2011

Hi all, it all started with issue10019. The version we have in cpython of json is simplejson 2.0.9 highly patched (either because it was converted to py3k, and because of the normal flow of issues/bugfixes) while upstream have already released 2.1.13 . Their 2 roads had diverged a lot, and since this blocks any further update of cpython's json from upstream, I'd like to close this gap. This isn't exactly an easy task, and this email is more about a brainstorming on the ways we have to … [View More]

12 44

[ANN] Python 2.5.6 Release Candidate 1
by "Martin v. Löwis" April 18, 2011

April 18, 2011

On behalf of the Python development team and the Python community, I'm happy to announce the release candidate 1 of Python 2.5.6. This is a source-only release that only includes security fixes. The last full bug-fix release of Python 2.5 was Python 2.5.4. Users are encouraged to upgrade to the latest release of Python 2.7 (which is 2.7.1 at this point). This releases fixes issues with the urllib, urllib2, SimpleHTTPServer, and audiop modules. See the release notes at the website (also … [View More]

2 2