Python-Dev March 2015

python-dev@python.org

90 participants
68 discussions

by Larry Hastings

Argument Clinic "converters" specify how to convert an individual argument to the function you're defining. Although a converter could theoretically represent any sort of conversion, most of the time they directly represent types like "int" or "double" or "str". Because there's such variety in argument parsing, the converters are customizable with parameters. Many of these are common enough that Argument Clinic suggests some standard names. Examples: "zeroes=True" for strings and buffers means "permit internal \0 characters", and "bitwise=True" for unsigned integers means "copy the bits over, even if there's overflow/underflow, and even if the original is negative". A third example is "nullable=True", which means "also accept None for this parameter". This was originally intended for use with strings (compare the "s" and "z" format units for PyArg_ParseTuple), however it looks like we'll have a use for "nullable ints" in the ongoing Argument Clinic conversion work. Several people have said they found the name "nullable" surprising, suggesting I use another name like "allow_none" or "noneable". I, in turn, find their surprise surprising; "nullable" is a term long associated with exactly this concept. It's used in C# and SQL, and the term even has its own Wikipedia page: http://en.wikipedia.org/wiki/Nullable_type Most amusingly, Vala *used* to have an annotation called "(allow-none)", but they've broken it out into two annotations, "(nullable)" and "(optional)". http://blogs.gnome.org/desrt/2014/05/27/allow-none-is-dead-long-live-nullab… Before you say "the term 'nullable' will confuse end users", let me remind you: this is not user-facing. This is a parameter for an Argument Clinic converter, and will only ever be seen by CPython core developers. A group which I hope is not so easily confused. It's my contention that "nullable" is the correct name. But I've been asked to bring up the topic for discussion, to see if a consensus forms around this or around some other name. Let the bike-shedding begin, //arry/

5 years, 5 months

PEP 448 review

by Guido van Rossum

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

5 years, 5 months

OpenBSD buildbot has many failures

by Victor Stinner

Hi, The OpenBSD buildbot always fail with the same failures since many months (ex: test_socket). Is someone interested to fix them? If no, would it be possible to turn it off to get a better view of regressions? Currently, they are too many red buildbots to quickly see regressions introduced by recent commits. http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x Victor

5 years, 6 months

http.client: Determining Content-Length

by Demian Brecht

Hi all, I'm not sure whether this should be python-list or here, but given it's a premature code review for http.client, I figured I'd post here first. I'm in the process of adding support for chunked transfer encoding to http.client (issue #12319). One of the bits of functionality that I'm working on in is ironing out some of the kinks out in determining the content length of the request bodies. Given the number of data types allowed as bodies it does get a little messy, so I wanted to get some feedback here and see if anyone can shoot holes through it prior to updating the current patch. The tests are passing, but there may be use cases not accounted for in the new implementation. https://gist.github.com/demianbrecht/f94be5a51e32bb9c81e1 The above is intended to replace _set_content_length (current state of the patch can be seen here: http://bugs.python.org/review/12319/diff/14331/Lib/http/client.py). There is a comprehensive list of data types currently supported can be found here: http://bugs.python.org/issue23740. Comments and feedback are much appreciated. Thanks, Demian

5 years, 6 months

MemoryError and other bugs on AMD64 OpenIndiana 3.x

by Victor Stinner

Hi, The buildbot AMD64 OpenIndiana 3.x is always red since at least 6 months. Almost always, tests fail because the buildbot has not enough memory. Well, it's fun to see how Python handles MemoryError (it's pretty good, there is no hard crash! my work on failmalloc was maybe useful ;-)), but it doesn't help to detect regressions. Would it be possible to enhance this buildbot? I copy this email to the owner of buildbot. But I also sent it to python-dev because MemoryError is not the only error, please take care of our buildbots! Thanks, Victor

5 years, 6 months

importante!!

by Alan Armour

its french! lol I just wanted to see if you could, as well as making python able to have assembly written, you should totally use blender as your main IDE it would be so epic and do pyjackd, and i think pyaudio will write some stuff, and like pycoreaudio and like directpy for windows direct x oh having blender as the main ide would just be epic

5 years, 6 months

RFC: PEP 490 - Chain exceptions at C level

by Victor Stinner

Hi, Here is the first draft of my PEP to chain automatically exceptions at C level in Python 3.5. Plain text version of the PEP below. HTML version of the PEP: https://www.python.org/dev/peps/pep-0490/ It has already an implementation, the pyerr_chain.patch file attached to http://bugs.python.org/issue23763 The patch is very short. We may modify more functions to hide the previous exception when the new exception contains as much or more information. Keeping the previous exception waste memory, make reference cycles more likely and is useless when displaying the full exception chain. I wrote a huge patch modifying a lot of functions to explicitly *not* chain exceptions: pyerr_match_clear-2.patch of the issue #23763. I wrote the patch to show where exceptions will be chained with pyerr_chain.patch applied. But we might apply it if we want to keep exactly the same behaviour between Python 3.4 and 3.5 for C modules the stdlib. pyerr_match_clear-2.patch adds also checks on the type of the previous exception to not replace an "unexpected" exception. For example, we can replace an expected TypeError with a ValueError (without chaining them), but we should not raise a ValueError if we got a KeyboardInterrupt, a MemoryError or other unexpected exceptions. But such changes are unrelated to the PEP and may be done in a separated issue. The starting point of my PEP is my work on CPython done with the pyfailmalloc module. This module injects MemoryError. I fixed a ton of issues in the code handling errors in CPython (see the issue #18408). I noticed that Python didn't complain when exceptions are lost. I added many assertions to detect these mistakes. Recently, I modified Python 3.5 to raise an exception even in release mode (issue #23571) if a C function returns NULL without an exception set, or if a C function returns a result with an exception set. Victor PEP: 490 Title: Chain exceptions at C level Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 25-March-2015 Python-Version: 3.5 Abstract ======== Chain exceptions at C level, as already done at Python level. Rationale ========= Python 3 introduced a new killer feature: exceptions are chained by default, PEP 3134. Example:: try: raise TypeError("err1") except TypeError: raise ValueError("err2") Output:: Traceback (most recent call last): File "test.py", line 2, in <module> raise TypeError("err1") TypeError: err1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "test.py", line 4, in <module> raise ValueError("err2") ValueError: err2 Exceptions are chained by default in Python code, but not in extensions written in C. A new private ``_PyErr_ChainExceptions()`` function was introduced in Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called explicitly to chain exceptions and its usage is not trivial. Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport`` module to chain the previous ``OSError`` to a new ``ZipImportError`` exception:: PyObject *exc, *val, *tb; PyErr_Fetch(&exc, &val, &tb); PyErr_Format(ZipImportError, "can't open Zip file: %R", archive); _PyErr_ChainExceptions(exc, val, tb); This PEP proposes to also chain exceptions automatically at C level to stay consistent and give more information on failures to help debugging. The previous example becomes simply:: PyErr_Format(ZipImportError, "can't open Zip file: %R", archive); Proposal ======== Modify PyErr_*() functions to chain exceptions ---------------------------------------------- Modify C functions raising exceptions of the Python C API to automatically chain exceptions: modify ``PyErr_SetString()``, ``PyErr_Format()``, ``PyErr_SetNone()``, etc. Modify functions to not chain exceptions ---------------------------------------- Keeping the previous exception is not always interesting when the new exception contains information of the previous exception or even more information, especially when the two exceptions have the same type. Example of an useless exception chain with ``int(str)``:: TypeError: a bytes-like object is required, not 'type' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: int() argument must be a string, a bytes-like object or a number, not 'type' The new ``TypeError`` exception contains more information than the previous exception. The previous exception should be hidden. The ``PyErr_Clear()`` function can be called to clear the current exception before raising a new exception, to not chain the current exception with a new exception. Modify functions to chain exceptions ------------------------------------ Some functions save and then restore the current exception. If a new exception is raised, the exception is currently displayed into sys.stderr or ignored depending on the function. Some of these functions should be modified to chain exceptions instead. Examples of function ignoring the new exception(s): * ``ptrace_enter_call()``: ignore exception * ``subprocess_fork_exec()``: ignore exception raised by enable_gc() * ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised by trying to display the bootstrap function to ``sys.stderr`` * ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore exception raised by looking for a key in the dictionary * ``_PyErr_TrySetFromCause()``: ignore exception * ``PyFrame_LocalsToFast()``: ignore exception raised by ``dict_to_map()`` * ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used to debug, to inspect a running process, it should not modify the Python state. * ``Py_ReprLeave()``: ignore exception "because there is no way to report them" * ``type_dealloc()``: ignore exception raised by ``remove_all_subclasses()`` * ``PyObject_ClearWeakRefs()``: ignore exception? * ``call_exc_trace()``, ``call_trace_protected()``: ignore exception * ``remove_importlib_frames()``: ignore exception * ``do_mktuple()``, helper used by ``Py_BuildValue()`` for example: ignore exception? * ``flush_io()``: ignore exception * ``sys_write()``, ``sys_format()``: ignore exception * ``_PyTraceback_Add()``: ignore exception * ``PyTraceBack_Print()``: ignore exception Examples of function displaying the new exception to ``sys.stderr``: * ``atexit_callfuncs()``: display exceptions with ``PyErr_Display()`` and return the latest exception, the function calls multiple callbacks and only returns the latest exception * ``sock_dealloc()``: log the ``ResourceWarning`` exception with ``PyErr_WriteUnraisable()`` * ``slot_tp_del()``: display exception with ``PyErr_WriteUnraisable()`` * ``_PyGen_Finalize()``: display ``gen_close()`` exception with ``PyErr_WriteUnraisable()`` * ``slot_tp_finalize()``: display exception raised by the ``__del__()`` method with ``PyErr_WriteUnraisable()`` * ``PyErr_GivenExceptionMatches()``: display exception raised by ``PyType_IsSubtype()`` with ``PyErr_WriteUnraisable()`` Backward compatibility ====================== A side effect of chaining exceptions is that exceptions store traceback objects which store frame objects which store local variables. Local variables are kept alive by exceptions. A common issue is a reference cycle between local variables and exceptions: an exception is stored in a local variable and the frame indirectly stored in the exception. The cycle only impacts applications storing exceptions. The reference cycle can now be fixed with the new ``traceback.TracebackException`` object introduced in Python 3.5. It stores informations required to format a full textual traceback without storing local variables. The ``asyncio`` is impacted by the reference cycle issue. This module is also maintained outside Python standard library to release a version for Python 3.3. ``traceback.TracebackException`` will maybe be backported in a private ``asyncio`` module to fix reference cycle issues. Alternatives ============ No change --------- A new private ``_PyErr_ChainExceptions()`` function is enough to chain manually exceptions. Exceptions will only be chained explicitly where it makes sense. New helpers to chain exceptions ------------------------------- Functions like ``PyErr_SetString()`` don't chain automatically exceptions. To make the usage of ``_PyErr_ChainExceptions()`` easier, new private functions are added: * ``_PyErr_SetStringChain(exc_type, message)`` * ``_PyErr_FormatChain(exc_type, format, ...)`` * ``_PyErr_SetNoneChain(exc_type)`` * ``_PyErr_SetObjectChain(exc_type, exc_value)`` Helper functions to raise specific exceptions like ``_PyErr_SetKeyError(key)`` or ``PyErr_SetImportError(message, name, path)`` don't chain exceptions. The generic ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` should be used to chain exceptions with these helper functions. Appendix ======== PEPs ---- * `PEP 3134 -- Exception Chaining and Embedded Tracebacks <https://www.python.org/dev/peps/pep-3134/>`_ (Python 3.0): new ``__context__`` and ``__cause__`` attributes for exceptions * `PEP 415 - Implement context suppression with exception attributes <https://www.python.org/dev/peps/pep-0415/>`_ (Python 3.3): ``raise exc from None`` * `PEP 409 - Suppressing exception context <https://www.python.org/dev/peps/pep-0409/>`_ (superseded by the PEP 415) Python C API ------------ The header file ``Include/pyerror.h`` declares functions related to exceptions. Functions raising exceptions: * ``PyErr_SetNone(exc_type)`` * ``PyErr_SetObject(exc_type, exc_value)`` * ``PyErr_SetString(exc_type, message)`` * ``PyErr_Format(exc, format, ...)`` Helpers to raise specific exceptions: * ``PyErr_BadArgument()`` * ``PyErr_BadInternalCall()`` * ``PyErr_NoMemory()`` * ``PyErr_SetFromErrno(exc)`` * ``PyErr_SetFromWindowsErr(err)`` * ``PyErr_SetImportError(message, name, path)`` * ``_PyErr_SetKeyError(key)`` * ``_PyErr_TrySetFromCause(prefix_format, ...)`` Manage the current exception: * ``PyErr_Clear()``: clear the current exception, like ``except: pass`` * ``PyErr_Fetch(exc_type, exc_value, exc_tb)`` * ``PyErr_Restore(exc_type, exc_value, exc_tb)`` * ``PyErr_GetExcInfo(exc_type, exc_value, exc_tb)`` * ``PyErr_SetExcInfo(exc_type, exc_value, exc_tb)`` Others function to handle exceptions: * ``PyErr_ExceptionMatches(exc)``: check to implement ``except exc: ...`` * ``PyErr_GivenExceptionMatches(exc1, exc2)`` * ``PyErr_NormalizeException(exc_type, exc_value, exc_tb)`` * ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` Python Issues ------------- Chain exceptions: * `Issue #23763: Chain exceptions in C <http://bugs.python.org/issue23763>`_ * `Issue #23696: zipimport: chain ImportError to OSError <http://bugs.python.org/issue23696>`_ * `Issue #21715: Chaining exceptions at C level <http://bugs.python.org/issue21715>`_: added ``_PyErr_ChainExceptions()`` * `Issue #18488: sqlite: finalize() method of user function may be called with an exception set if a call to step() method failed <http://bugs.python.org/issue18488>`_ * `Issue #23781: Add private _PyErr_ReplaceException() in 2.7 <http://bugs.python.org/issue23781>`_ * `Issue #23782: Leak in _PyTraceback_Add <http://bugs.python.org/issue23782>`_ Changes preventing to loose exceptions: * `Issue #23571: Raise SystemError if a function returns a result with an exception set <http://bugs.python.org/issue23571>`_ * `Issue #18408: Fixes crashes found by pyfailmalloc <http://bugs.python.org/issue18408>`_ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

5 years, 6 months

[RELEASED] Python 3.5.0a3 is now available

by Larry Hastings

On behalf of the Python development community and the Python 3.5 release team, I'm thrilled to announce the availability of Python 3.5.0a3. Python 3.5.0a3 is the third alpha release of Python 3.5, which will be the next major release of Python. Python 3.5 is still under heavy development, and is far from complete. This is a preview release, and its use is not recommended for production settings. Two important notes for Windows users about Python 3.5.0a3: * If you have previously installed Python 3.5.0a1, you may need to manually uninstall it before installing Python 3.5.0a3 (issue23612). * If installing Python 3.5.0a3 as a non-privileged user, you may need to escalate to administrator privileges to install an update to your C runtime libraries. You can find Python 3.5.0a3 here: https://www.python.org/downloads/release/python-350a3/ Happy hacking, //arry/

5 years, 6 months

Re: [Python-Dev] cpython: Issue #23752: When built from an existing file descriptor, io.FileIO() now only

by Serhiy Storchaka

On 30.03.15 04:22, victor.stinner wrote: > https://hg.python.org/cpython/rev/bc2a22eaa0af > changeset: 95269:bc2a22eaa0af > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Mon Mar 30 03:21:06 2015 +0200 > summary: > Issue #23752: When built from an existing file descriptor, io.FileIO() now only > calls fstat() once. Before fstat() was called twice, which was not necessary. > > files: > Misc/NEWS | 26 ++++++++++++++++++++++++++ > Modules/_io/fileio.c | 24 ------------------------ > 2 files changed, 26 insertions(+), 24 deletions(-) > > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -2,6 +2,32 @@ > Python News > +++++++++++ > > +What's New in Python 3.5.0 alpha 4? > +=================================== Return time machine back Victor. Current version is 3.5.0a2+.

5 years, 6 months

Sporadic failures of test_subprocess and test_multiprocessing_spawn

by Victor Stinner

Hi, Can you please take a look at the following issue and try to reproduce it? http://bugs.python.org/issue23771 The following tests sometimes hang on "x86 Ubuntu Shared 3.x" and "AMD64 Debian root 3.x" buildbots: - test_notify_all() of test_multiprocessing_spawn - test_double_close_on_error() of test_subprocess - other sporadic failures of test_subprocess I'm quite sure that they are regressions, maybe related to the implementation of the PEP 475. In the middle of all PEP 475 changes, I changed some functions to release the GIL on I/O, it wasn't the case before. I may be related. Are you able to reproduce these issues? I'm unable to reproduce them on Fedora 21. Maybe they are more likely on Debian-like operating systems? Victor

5 years, 6 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev March 2015