Mailman 3 March 2014 - Python-Dev

Treating tokenize.Untokenizer as private
by Terry Reedy March 31, 2014

March 31, 2014

I am working through the multiple bugs afflicting tokenize.untokenize, which is described in the tokenize doc and has an even longer docstring. While the function could be implemented as one 70-line function, it happens to be implemented as a 4-line wrapper for a completely undocumented (Untokenizer class with 4 methods. (It is unmentioned in the doc and there are currently no docstrings.) I view the class as a private implementation detail and would like to treat it as such, and perhaps … [View More]

3 4

collections.sortedtree
by Marko Rauhamaa March 31, 2014

March 31, 2014

I have made a full implementation of a balanced tree and would like to know what the process is to have it considered for inclusion in Python 3. To summarize, the implementation closely parallels dict() features and resides in _collectionsmodule.c under the name collections.sortedtree. It uses solely the "<" operator to compare keys. I have chosen the AVL tree as an implementation technique. The views support a number of optional arguments: sorteddict.keys(reversed=False, start=… [View More]

24 74

Making proxy types easier to write and maintain
by Antoine Pitrou March 31, 2014

March 31, 2014

Hello, It is known to be cumbersome to write a proxy type that will correctly proxy all special methods (this has to be done manually on the type, since special methods are not looked up on the instance: a __getattr__ method would not work). Recently we've had reports of two stdlib types that forgot to implement some special methods: - weakref.proxy doesn't implement __reversed__: http://bugs.python.org/issue19359 - mock.MagicMock doesn't implement __truediv__: http://bugs.python.org/… [View More]

8 14

freeze build slave
by "Martin v. Löwis" March 31, 2014

March 31, 2014

I have created a buildbot configuration to test freeze. At the moment, it has only one builder: http://buildbot.python.org/all/waterfall?show=AMD64%20Ubuntu%20LTS%20Freeze… which currently fails as freeze doesn't actually work. The test itself works by first building Python in release mode, then installing it, then running freeze on a test program, then building the test programm (and ultimately running it). The question then is: how should that integrate with the rest of the builders? I … [View More]

4 3

PyErr_ Fetch | Restore & friends
by Ethan Furman March 29, 2014

March 29, 2014

I'm working on issue 1615 [1] and came up with this tidbit, which works [2], but not well enough: slot_tp_getattr_hook(PyObject *self, PyObject *name) { ... + PyObject *error_type, *error_value, *error_traceback; ... + /* if an AttributeError is set, save it and call getattr; if getattr + sets an AttributeError, discard it and return the original + AttributeError */ if (res == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) { + … [View More]

2 2

Re: [Python-Dev] cpython: inspect.Signature: Add 'Signature.from_callable' classmethod. Closes #17373
by Antoine Pitrou March 29, 2014

March 29, 2014

On Thu, 27 Mar 2014 17:12:02 +0100 (CET) yury.selivanov <python-checkins(a)python.org> wrote: > > + .. classmethod:: Signature.from_callable(obj) > + > + Return a :class:`Signature` (or its subclass) object for a given callable > + ``obj``. This method simplifies subclassing of :class:`Signature`: > + > + :: > + > + class MySignature(Signature): > + pass > + sig = MySignature.from_callable(min) > + … [View More]

3 2

Fwd: shutdown concurrent.futures.ProcessPoolExecutor
by Giampaolo Rodola' March 29, 2014

March 29, 2014

This is a follow up of: https://groups.google.com/forum/#!topic/python-tulip/91NCCqV4SFs According to the information I collected so far it seems it's not possible (or at least very hard) to cleanly shutdown a process pool and all its workers in case of KeyboardInterrupt / SIGINT. Literally, what I would like to do is this: - intercept SIGINT form the main process and/or from each worker process - send SIGINT to all workers - wait for them to finish with a timeout - finally exit Whereas it … [View More]exists a solution based on multiprocessing: http://noswap.com/blog/python-multiprocessing-keyboardinterrupt ...concurrent.futures.ProcessPoolExecutor does not expose all the necessary bits to emulate it. By opening this thread I hope either to find a solution and document it or if it turns out that it is simply not possible to do this via concurrent.futures then discuss whether it's the case to expose more ProcessPoolExecutor APIs, because right now the interface it offers is pretty limited compared to multiprocessing.Pool: http://docs.python.org/3.5/library/multiprocessing.html#multiprocessing.poo… In particular, it won't let you pass an initializer function to pass to multiprocessing.Pool nor terminate() the process pool (only wait() for it). ---------- Forwarded message ---------- From: Guido van Rossum <guido(a)python.org> Date: Sat, Mar 29, 2014 at 12:05 AM Subject: Re: [python-tulip] How to cleanly shutdown the IO loop when using run_in_executor()? To: Giampaolo Rodola' <g.rodola(a)gmail.com> Cc: python-tulip <python-tulip(a)googlegroups.com> You're going to have to move the discussion to python-dev or the python issue tracker then. On Fri, Mar 28, 2014 at 4:02 PM, Giampaolo Rodola' <g.rodola(a)gmail.com>wrote: > > On Wed, Mar 26, 2014 at 7:35 PM, Guido van Rossum <guido(a)python.org>wrote: > >> Another thing to investigate might be how the executor creates the >> processes, and if anything happens to signals there. > > > After some further digging this seems to be related with the problem at > hand. > According to this: > http://noswap.com/blog/python-multiprocessing-keyboardinterrupt > ...it appears a feasible solution is to prevent workers to ever receive > KeyboardInterrupt and have the main process shutdown the pool via > pool.terminate() and finally pool.join(). > Unfortunately concurrent.futures.ProcessPoolExecutor does not expose all > the necessary bits to do that (multiprocessing.Pool(initializer=...) > argument and terminate() method). > > I also suspect that in order to emulate the suggested solution the > underlying Process instance should be daemonized, similarly to what > multiprocessing.Pool does: > > http://hg.python.org/cpython/file/3567d7ebd382/Lib/multiprocessing/pool.py#… > > I wonder whether concurrent.futures.ProcessPoolExecutor would be better > off exposing more APIs in order to facilitate tasks like these. > > > -- > Giampaolo - http://grodola.blogspot.com > > -- --Guido van Rossum (python.org/~guido) -- Giampaolo - http://grodola.blogspot.com [View Less]

1 0

PEP 461: Adding % formatting to bytes and bytearray -- Request for Pronouncement
by Ethan Furman March 28, 2014

March 28, 2014

Requesting pronouncement on PEP 461. Full text below. =============================================================================== PEP: 461 Title: Adding % formatting to bytes and bytearray Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman <ethan(a)stoneleaf.us> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2014-01-13 Python-Version: 3.5 Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25, 2014-03-27 Resolution: … [View More] Abstract ======== This PEP proposes adding % formatting operations similar to Python 2's ``str`` type to ``bytes`` and ``bytearray`` [1]_ [2]_. Rationale ========= While interpolation is usually thought of as a string operation, there are cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the work needed to make up for this missing functionality detracts from the overall readability of the code. Motivation ========== With Python 3 and the split between ``str`` and ``bytes``, one small but important area of programming became slightly more difficult, and much more painful -- wire format protocols [3]_. This area of programming is characterized by a mixture of binary data and ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in writing new wire format code, and in porting Python 2 wire format code. Common use-cases include ``dbf`` and ``pdf`` file formats, ``email`` formats, and ``FTP`` and ``HTTP`` communications, among many others. Proposed semantics for ``bytes`` and ``bytearray`` formatting ============================================================= %-interpolation --------------- All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``, ``e``, ``E``, ``f``, ``F``, ``g``, ``G``, and any that are subsequently added to Python 3) will be supported, and will work as they do for str, including the padding, justification and other related modifiers (currently ``#``, ``0``, ``-``, `` `` (space), and ``+`` (plus any added to Python 3)). The only non-numeric codes allowed are ``c``, ``b``, ``a``, and ``s`` (which is a synonym for b). For the numeric codes, the only difference between ``str`` and ``bytes`` (or ``bytearray``) interpolation is that the results from these codes will be ASCII-encoded text, not unicode. In other words, for any numeric formatting code `%x`:: b"%x" % val is equivalent to:: ("%x" % val).encode("ascii") Examples:: >>> b'%4x' % 10 b' a' >>> b'%#4x' % 10 ' 0xa' >>> b'%04X' % 10 '000A' ``%c`` will insert a single byte, either from an ``int`` in range(256), or from a ``bytes`` argument of length 1, not from a ``str``. Examples:: >>> b'%c' % 48 b'0' >>> b'%c' % b'a' b'a' ``%b`` will insert a series of bytes. These bytes are collected in one of two ways: - input type supports ``Py_buffer`` [4]_? use it to collect the necessary bytes - input type is something else? use its ``__bytes__`` method [5]_ ; if there isn't one, raise a ``TypeError`` In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is rejected as the string to bytes conversion requires an encoding, and we are refusing to guess; numbers are rejected because: - what makes a number is fuzzy (float? Decimal? Fraction? some user type?) - allowing numbers would lead to ambiguity between numbers and textual representations of numbers (3.14 vs '3.14') - given the nature of wire formats, explicit is definitely better than implicit ``%s`` is included as a synonym for ``%b`` for the sole purpose of making 2/3 code bases easier to maintain. Python 3 only code should use ``%b``. Examples:: >>> b'%b' % b'abc' b'abc' >>> b'%b' % 'some string'.encode('utf8') b'some string' >>> b'%b' % 3.14 Traceback (most recent call last): ... TypeError: b'%b' does not accept 'float' >>> b'%b' % 'hello world!' Traceback (most recent call last): ... TypeError: b'%b' does not accept 'str' ``%a`` will give the equivalent of ``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated value. Use cases include developing a new protocol and writing landmarks into the stream; debugging data going into an existing protocol to see if the problem is the protocol itself or bad data; a fall-back for a serialization format; or any situation where defining ``__bytes__`` would not be appropriate but a readable/informative representation is needed [6]_. Examples:: >>> b'%a' % 3.14 b'3.14' >>> b'%a' % b'abc' b"b'abc'" >>> b'%a' % 'def' b"'def'" Unsupported codes ----------------- ``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported. Compatibility with Python 2 =========================== As noted above, ``%s`` is being included solely to help ease migration from, and/or have a single code base with, Python 2. This is important as there are modules both in the wild and behind closed doors that currently use the Python 2 ``str`` type as a ``bytes`` container, and hence are using ``%s`` as a bytes interpolator. However, ``%b`` should be used in new, Python 3 only code, so ``%s`` will immediately be deprecated, but not removed until the next major Python release. Proposed variations =================== It has been proposed to automatically use ``.encode('ascii','strict')`` for ``str`` arguments to ``%b``. - Rejected as this would lead to intermittent failures. Better to have the operation always fail so the trouble-spot can be correctly fixed. It has been proposed to have ``%b`` return the ascii-encoded repr when the value is a ``str`` (b'%b' % 'abc' --> b"'abc'"). - Rejected as this would lead to hard to debug failures far from the problem site. Better to have the operation always fail so the trouble-spot can be easily fixed. Originally this PEP also proposed adding format-style formatting, but it was decided that format and its related machinery were all strictly text (aka ``str``) based, and it was dropped. Various new special methods were proposed, such as ``__ascii__``, ``__format_bytes__``, etc.; such methods are not needed at this time, but can be visited again later if real-world use shows deficiencies with this solution. A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [7]_, also exists. Objections ========== The objections raised against this PEP were mainly variations on two themes: - the ``bytes`` and ``bytearray`` types are for pure binary data, with no assumptions about encodings - offering %-interpolation that assumes an ASCII encoding will be an attractive nuisance and lead us back to the problems of the Python 2 ``str``/``unicode`` text model As was seen during the discussion, ``bytes`` and ``bytearray`` are also used for mixed binary data and ASCII-compatible segments: file formats such as ``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc. ``bytes`` and ``bytearray`` already have several methods which assume an ASCII compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to name just a few. %-interpolation, with its very restricted mini-language, will not be any more of a nuisance than the already existing methods. Some have objected to allowing the full range of numeric formatting codes with the claim that decimal alone would be sufficient. However, at least two formats (dbf and pdf) make use of non-decimal numbers. Footnotes ========= .. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting .. [2] neither string.Template, format, nor str.format are under consideration .. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html .. [4] http://docs.python.org/3/c-api/buffer.html examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` .. [5] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ .. [6] https://mail.python.org/pipermail/python-dev/2014-February/132750.html .. [7] http://python.org/dev/peps/pep-0460/ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

3 3

ISO 8601 durations and datetime.timedelta
by Skip Montanaro March 28, 2014

March 28, 2014

Andrew wrote: > I meant ISO 8601 syntax for "durations" [1]. That's exactly what I was referring to. Consider this session: >>> now = datetime.datetime.now() >>> now datetime.datetime(2014, 3, 28, 12, 4, 38, 517110) >>> then = now - datetime.timedelta(days=57, hours=12, minutes=12, seconds=12) >>> now datetime.datetime(2014, 3, 28, 12, 4, 38, 517110) >>> then datetime.datetime(2014, 1, 29, 23, 52, 26, 517110) so, the difference is: >>&… [View More]

2 1

Summary of Python tracker Issues
by Python tracker March 28, 2014

March 28, 2014

ACTIVITY SUMMARY (2014-03-21 - 2014-03-28) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4534 (+23) closed 28324 (+51) total 32858 (+74) Open issues with patches: 2090 Issues opened (61) ================== #15968: Incorporate Tcl/Tk/Tix into the Windows build process http://bugs.python.org/issue15968 reopened by haypo #16047: Tools/freeze no … [View More]

1 0