Mailman 3 July 2013 - Python-Dev

Fwd: Issue 1229 in nativeclient: Get Python to work under Native Client
by anatoly techtonik 18 Jul '13

18 Jul '13

Does Python build system support cross-compiling? NaCl projects seems to have problem with that. Just thought you might be interested to know about it. ---------- Forwarded message ---------- From: <nativeclient(a)googlecode.com> Date: Tue, Jul 16, 2013 at 9:24 PM Subject: Re: Issue 1229 in nativeclient: Get Python to work under Native Client To: techtonik(a)gmail.com Updates: Status: Fixed Comment #8 on issue 1229 by mseaborn(a)chromium.org: Get Python to work under Native Client http://code.google.com/p/**nativeclient/issues/detail?id=**1229<http://code.google.com/p/nativeclient/issues/detail?id=1229> This is done in naclports. Sam Clegg added initial support for building against nacl-glibc in https://code.google.com/p/** naclports/source/detail?r=802<https://code.google.com/p/naclports/source/detail?r=802> . Sam found a way to work around the Python build's lack of support for cross-compiling. The workaround is to build a native "pgen" first and inject it into the later cross-build. Building Python with nacl-newlib and PNaCl is a work in progress: https://codereview.chromium.**org/18112024/<https://codereview.chromium.org/18112024/>

1 0

Misc re.match() complaint
by Guido van Rossum 18 Jul '13

18 Jul '13

In a discussion about mypy I discovered that the Python 3 version of the re module's Match object behaves subtly different from the Python 2 version when the target string (i.e. the haystack, not the needle) is a buffer object. In Python 2, the type of the return value of group() is always either a Unicode string or an 8-bit string, and the type is determined by looking at the target string -- if the target is unicode, group() returns a unicode string, otherwise, group() returns an 8-bit string. In particular, if the target is a buffer object, group() returns an 8-bit string. I think this is the appropriate behavior: otherwise using regular expression matching to extract a small substring from a large target string would unnecessarily keep the large target string alive as long as the substring is alive. But in Python 3, the behavior of group() has changed so that its return type always matches that of the target string. I think this is bad -- apart from the lifetime concern, it means that if your target happens to be a bytearray, the return value isn't even hashable! Does anyone remember whether this was a conscious decision? Is it too late to fix? -- --Guido van Rossum (python.org/~guido)

10 28

Tweaking PEP 8 guidelines for use of leading underscores
by Nick Coghlan 17 Jul '13

17 Jul '13

Currently, the naming section of PEP 8 doesn't say very much about what a leading underscore *means* in the Python standard library. I would like to add a new "Private interfaces" subsection under "Naming Conventions" to say the following: ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as many Python users rely on introspection to identify available functionality and may be mislead into believing an API without the leading underscore is in fact a public API with the standard backwards compatibility guarantees. All test modules are also considered private interfaces. Even though they typically lack the leading underscore, modules imported by another module are also considered an implementation detail. Other modules *should* not rely on indirect access to such modules unless they are an explicitly documented part of the API (such as ``os.path``). ================= Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

15 53

PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets
by Victor Stinner 17 Jul '13

17 Jul '13

HTML version: http://www.python.org/dev/peps/pep-0446/ PEP: 446 Title: Add new parameters to configure the inherance of files and for non-blocking sockets Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-July-2013 Python-Version: 3.4 Abstract ======== This PEP proposes new portable parameters and functions to configure the inherance of file descriptors and the non-blocking flag of sockets. Rationale ========= Inherance of file descriptors ----------------------------- The inherance of file descriptors in child processes can be configured on each file descriptor using a *close-on-exec* flag. By default, the close-on-exec flag is not set. On Windows, file descriptors are not inherited if the ``bInheritHandles`` parameter of the ``CreateProcess()`` function is ``FALSE``, even if the close-on-exec flag is not set. On UNIX, file descriptors with the close-and-exec flag set are closed at the execution of a new program (ex: when calling ``execv()``). The flag has no effect on ``fork()``, all file descriptors are inherited by the child process. Issues of the inherance of file descriptors ------------------------------------------- Inherance of file descriptors causes issues. For example, closing a file descriptor in the parent process does not release the resource (file, socket, ...), because the file descriptor is still open in the child process. Leaking file descriptors is also a major security vulnerability. An untrusted child process can read sensitive data like passwords and take control of the parent process though leaked file descriptors. It is for example a known vulnerability to escape from a chroot. Non-blocking sockets -------------------- To handle multiple network clients in a single thread, a multiplexing function like ``select()`` can be used. For best performances, sockets must be configured as non-blocking. Operations like ``send()`` and ``recv()`` return an ``EAGAIN`` or ``EWOULDBLOCK`` error if the operation would block. By default, newly created sockets are blocking. Setting the non-blocking mode requires additional system calls. Setting flags at the creation of the file descriptor ---------------------------------------------------- Windows and recent versions of other operating systems like Linux support setting the close-on-exec flag directly at the creation of file descriptors, and close-on-exec and blocking flags at the creation of sockets. Setting these flags at the creation is atomic and avoids additional system calls. Proposal ======== New cloexec And blocking Parameters ----------------------------------- Add a new optional *cloexec* on functions creating file descriptors: * ``io.FileIO`` * ``io.open()`` * ``open()`` * ``os.dup()`` * ``os.dup2()`` * ``os.fdopen()`` * ``os.open()`` * ``os.openpty()`` * ``os.pipe()`` * ``select.devpoll()`` * ``select.epoll()`` * ``select.kqueue()`` Add new optional *cloexec* and *blocking* parameters to functions creating sockets: * ``asyncore.dispatcher.create_socket()`` * ``socket.socket()`` * ``socket.socket.accept()`` * ``socket.socket.dup()`` * ``socket.socket.fromfd`` * ``socket.socketpair()`` The default value of *cloexec* is ``False`` and the default value of *blocking* is ``True``. The atomicity is not guaranteed. If the platform does not support setting close-on-exec and blocking flags at the creation of the file descriptor or socket, the flags are set using additional system calls. New Functions ------------- Add new functions the get and set the close-on-exec flag of a file descriptor: * ``os.get_cloexec(fd:int) -> bool`` * ``os.set_cloexec(fd:int, cloexec: bool)`` Other Changes ------------- The ``subprocess.Popen`` class must clear the close-on-exec flag of file descriptors of the ``pass_fds`` parameter. The close-on-exec flag must also be set on private file descriptors and sockets in the Python standard library. For example, on UNIX, os.urandom() opens ``/dev/urandom`` to read some random bytes and the file descriptor is closed at function exit. The file descriptor is not expected to be inherited by child processes. Rejected Alternatives ===================== PEP 433 ------- The PEP 433 entitled "Easier suppression of file descriptor inheritance" is a previous attempt proposing various other alternatives, but no consensus could be reached. This PEP has a well defined behaviour (the default value of the new *cloexec* parameter is not configurable), is more conservative (no backward compatibility issue), and is much simpler. Add blocking parameter for file descriptors and Windows overlapped I/O ---------------------------------------------------------------------- Windows supports non-blocking operations on files using an extension of the Windows API called "Overlapped I/O". Using this extension requires to modify the Python standard library and applications to pass a ``OVERLAPPED`` structure and an event loop to wait for the completion of operations. This PEP only tries to expose portable flags on file descriptors and sockets. Supporting overlapped I/O requires an abstraction providing a high-level and portable API for asynchronous operations on files and sockets. Overlapped I/O are out of the scope of this PEP. UNIX supports non-blocking files, moreover recent versions of operating systems support setting the non-blocking flag at the creation of a file descriptor. It would be possible to add a new optional *blocking* parameter to Python functions creating file descriptors. On Windows, creating a file descriptor with ``blocking=False`` would raise a ``NotImplementedError``. This behaviour is not acceptable for the ``os`` module which is designed as a thin wrapper on the C functions of the operating system. If a platform does not support a function, the function should not be available on the platform. For example, the ``os.fork()`` function is not available on Windows. For all these reasons, this alternative was rejected. The PEP 3156 proposes an abstraction for asynchronous I/O supporting non-blocking files on Windows. Links ===== Python issues: * `#10115: Support accept4() for atomic setting of flags at socket creation <http://bugs.python.org/issue10115>`_ * `#12105: open() does not able to set flags, such as O_CLOEXEC <http://bugs.python.org/issue12105>`_ * `#12107: TCP listening sockets created without FD_CLOEXEC flag <http://bugs.python.org/issue12107>`_ * `#16850: Add "e" mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_ * `#16860: Use O_CLOEXEC in the tempfile module <http://bugs.python.org/issue16860>`_ * `#16946: subprocess: _close_open_fd_range_safe() does not set close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined <http://bugs.python.org/issue16946>`_ * `#17070: Use the new cloexec to improve security and avoid bugs <http://bugs.python.org/issue17070>`_ Other links: * `Secure File Descriptor Handling <http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper, 2008) Copyright ========= This document has been placed into the public domain.

5 25

Why does PEP 8 advise against explicit relative imports?
by Nick Coghlan 17 Jul '13

17 Jul '13

PEP 8 advises developers to use absolute imports rather than explicit relative imports. Why? Using absolute imports couple the internal implementation of a package to its public name - you can't just change the top level directory name any more, you have to go through and change all the absolute imports as well. You also can't easily vendor a package that uses absolute imports inside another project either, since all the absolute imports will break. What's the *concrete* benefit of using absolute imports that overcomes these significant shortcomings in maintainability and composability? Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

7 10

Is it safe to call PyEval_EvalFrameEx() with an exception set?
by Victor Stinner 17 Jul '13

17 Jul '13

Hi, I'm working on the issue #18408 (fix issues found by my pyfailmalloc tool). To analyze some bugs, I have to add debug traces in various functions to find which function returned NULL without setting an error, or the opposite: returned a valid object, but with an exception set. I would like to add assertions in Python/ceval.c to detect such bugs earlier. The problem is that some functions rely on the ability to call PyEval_EvalFrameEx() with an exception set. Is it expected? Should it be avoided? The current exception can be replaced with a new exception. Example 1: module = PyImport_Import(module_name); if (module == NULL) { PyErr_Format(PicklingError, "Can't pickle %R: import of module %R failed", obj, module_name); goto error; } Formatting the obj argument calls indirectly PyEval_EvalFrameEx(), whereas an ImportError was raised. The original ImportError is replaced with a new PickleError exception. To not call PyEval_EvalFrameEx() with an exception set, PyErr_Format() can call PyErr_Clear() before calling PyUnicode_FromFormatV(), maybe only in debug mode. Example 2: test_sqlite uses sqlite.connection.create_aggregate() to create a Python callback called by the C sqlite3 library. The "callback" is a class with 3 methods: constructor, step() and finalize(). The test AggregateTests.CheckAggrNoStep() uses a class with no step() method: it ensures that an AttributeError is raised. The problem is that the finalize() method is called even if the class has no step method, and so there is a current AttributeError exception. I don't know if finalize() should be called in this case. It may be a bug in the sqlite module. More generally, Victor

2 3

Re: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure
by Serhiy Storchaka 17 Jul '13

17 Jul '13

17.07.13 00:09, victor.stinner написав(ла): > http://hg.python.org/cpython/rev/533eb9ab895a > changeset: 84669:533eb9ab895a > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Tue Jul 16 21:36:02 2013 +0200 > summary: > Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure > > bytes is NULL on _PyBytes_Resize() failure > > files: > Modules/_io/fileio.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c > --- a/Modules/_io/fileio.c > +++ b/Modules/_io/fileio.c > @@ -739,7 +739,7 @@ > > if (n != size) { > if (_PyBytes_Resize(&bytes, n) < 0) { > - Py_DECREF(bytes); > + Py_CLEAR(bytes); > return NULL; > } > } Why not Py_DECREF?

3 4

Python 3.4 and Windows XP: just 45 days until EOL
by Christian Heimes 16 Jul '13

16 Jul '13

Hi, how do you feel about dropping Windows XP support for Python 3.4? It would enable us to use some features that are only available on Windows Vista and newer, for example http://bugs.python.org/issue6926 and http://bugs.python.org/issue1763 . PEP 11 says: A new feature release X.Y.0 will support all Windows releases whose extended support phase is not yet expired. For Python 3.4 is going to be a very close call. According to PEP 429 3.4.0 final is scheduled for February 22, 2014. The extended support phase of Windows XP ends merely 45 days later on April 8, 2014. Do we really have to restrict ourselves to an API that is going to become deprecated 45 days after the estimated release of 3.4.0? Christian

15 17

Re: [Python-Dev] [Python-checkins] cpython: Issue #18393: Remove use of deprecated API on OSX
by Zachary Ware 16 Jul '13

16 Jul '13

On Mon, Jul 15, 2013 at 11:32 AM, ronald.oussoren <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/ccbaf6762b54 > changeset: 84634:ccbaf6762b54 > user: Ronald Oussoren <ronaldoussoren(a)mac.com> > date: Mon Jul 15 18:32:09 2013 +0200 > summary: > Issue #18393: Remove use of deprecated API on OSX > > The "Gestalt" function on OSX is deprecated (starting with OSX 10.8), > remove its usage from the stdlib. The patch removes a number of private I believe this means that Lib/test/leakers/test_gestalt.py can be removed as well. > functions and a private module, but does not change the public API. > > The removed code was effectively dead, the platform module has used > other code to fetch the OSX version for years and could only use > on the Gestalt-based code as a fallback. That fallback can only trigger > on broken OSX installs (that is, someone has removed parts of the system > install) > > files: > Lib/platform.py | 64 +------------------------- > Misc/NEWS | 4 + > Modules/_gestalt.c | 84 ---------------------------------- > setup.py | 4 - > 4 files changed, 5 insertions(+), 151 deletions(-) > > > <snip diffs> > > -- > Repository URL: http://hg.python.org/cpython > -- Zach

2 1

Re: [Python-Dev] [Python-checkins] cpython (3.3): check the return value of new_string() (closes #18470)
by Victor Stinner 16 Jul '13

16 Jul '13

2013/7/16 benjamin.peterson <python-checkins(a)python.org>: > http://hg.python.org/cpython/rev/c3a510b22218 > changeset: 84653:c3a510b22218 > branch: 3.3 > parent: 84651:e22dd5fda5a8 > user: Benjamin Peterson <benjamin(a)python.org> > date: Mon Jul 15 19:15:34 2013 -0700 > summary: > check the return value of new_string() (closes #18470) > > ... > > > diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c > --- a/Parser/tokenizer.c > +++ b/Parser/tokenizer.c > @@ -250,17 +253,21 @@ > t++; > > if (begin < t) { > - char* r = new_string(begin, t - begin); > + char* r = new_string(begin, t - begin, tok); > + if (!r) > + return 0; > char* q = get_normal_name(r); > ... Visual Studio does support instructions between declarations, and so this changeset broke Windows buildbots. Should we add the "-Werror=declaration-after-statement" compiler flag to the 3.3 branch? (in debug mode?) Victor

2 1