Mailman 3 August 2013 - Python-Dev

NULL allowed in PyErr_SetString and friends?
by Antoine Pitrou 12 Aug '13

12 Aug '13

Hello, It seems NULL is allowed as the first argument of PyErr_Format, PyErr_SetString and PyErr_SetObject. Moreover, it means "clear the error indicator". However, this is not mentioned in the docs. I was wondering if we should officialize this behaviour or change it. Regards Antoine.

2 1

Strange artifacts with PEP 3121 and monkey-patching sys.modules (in csv, ElementTree and others)
by Eli Bendersky 12 Aug '13

12 Aug '13

Hello, Recently as part of the effort of untangling the tests of ElementTree and general code improvements (e.g. http://bugs.python.org/issue15651), I ran into something strange about PEP 3121-compliant modules. I'll demonstrate with csv, just as an example. PEP 3121 mandates this function to look up the module-specific state in the current sub-interpreter: PyObject* PyState_FindModule(struct PyModuleDef*); This appears to make the following assumption: a given sub-interpreter only imports any C extension *once*. If it happens more than once, the assumption breaks in troubling ways. In normal code, it should never happen more than once because of the caching in sys.modules; However, many of our tests monkey-patch sys.modules (mainly by calling test.support.import_fresh_module) and hell breaks use. Here's a simple example: ---- import sys csv = __import__('csv') csv.register_dialect('unixpwd', delimiter=':', quoting=csv.QUOTE_NONE) print(csv.list_dialects()) # ==> ['unixpwd', 'excel-tab', 'excel', 'unix'] del sys.modules['csv'] # FUN del sys.modules['_csv'] some_other_csv = __import__('csv') print(csv.list_dialects()) # ==> ['excel-tab', 'excel', 'unix'] ---- Note how doing some sys.modules acrobatics and re-importing suddenly changes the internal state of a previously imported module. This happens because: 1. The first import of 'csv' (which then imports `_csv) creates module-specific state on the heap and associates it with the current sub-interpreter. The list of dialects, amongst other things, is in that state. 2. The 'del's wipe 'csv' and '_csv' from the cache. 3. The second import of 'csv' also creates/initializes a new '_csv' module because it's not in sys.modules. This *replaces* the per-sub-interpreter cached version of the module's state with the clean state of a new module So essentially, while PEP 3121 moves state from C-file globals to per-module state, the state is still global, and this fact can be exposed from pure Python code. The above is a toy example. Here's a more serious case I ran into with ET, but once again is demonstrated with 'csv' for simplicity: ---- import io from test.support import import_fresh_module import csv csv_other = import_fresh_module('csv', fresh=['_csv', 'csv']) f = io.StringIO('foo\x00,bar\nbaz,42') reader = csv.reader(f) try: for row in reader: print(row) except csv.Error as e: print('Caught csv.error', e) except Exception as e: print('Caught Exception', e) ---- In the above, the reader throws 'csv.Error' (because of the NULL byte) but the exception clause does not catch it where expected, because it's a different exception class called `csv.Error`, due to the same problem demonstrated above (if the seemingly innocent import_fresh_module is removed, all is good). Any ideas/suggestion regarding this are welcome. This is quite an esoteric problem, but I believe it's serious. PEP 3121 is not used much (yet), but recently there was talk again about committing some of the patches created for converting Modules/*.c extensions to it during a GSoC project. I believe that we should understand the implications first. There can be a number of solutions; including modifying the PEP 3121 implementation machinery to really create/keep state "per module" and not just "per kind of module in a single sub-interpreter". Eli

4 26

(New) PEP 446: Make newly created file descriptors non-inheritable
by Victor Stinner 12 Aug '13

12 Aug '13

Hi, My second try (old PEP 446) to change how the inheritance of file descriptors was somehow rejected. Here is a third try (new PEP 446) which should include all information of the recent discussions on python-dev, especially how file descriptors and handles are inherited on Windows. I added tables to summarize the status of Python 3.3 and the status of atomic flags. I rewrote the rationale to introduce and explain the inheritance of file descriptors, which is not an simple subject. I removed the O_NONBLOCK flag from the PEP, it is a different issue and a new PEP must be written. Online HTML version: http://www.python.org/dev/peps/pep-0446/ You may also want to read my first attempt: http://www.python.org/dev/peps/pep-0433/ PEP: 446 Title: Make newly created file descriptors non-inheritable Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-August-2013 Python-Version: 3.4 Abstract ======== Leaking file descriptors in child processes causes various annoying issues and is a known major security vulnerability. This PEP proposes to make all file descriptors created by Python non-inheritable by default to reduces the risk of these issues. This PEP fixes also a race condition in multithreaded applications on operating systems supporting atomic flags to create non-inheritable file descriptors. Rationale ========= Inheritance of File Descriptors ------------------------------- Each operating system handles the inheritance of file descriptors differently. Windows creates non-inheritable file descriptors by default, whereas UNIX creates inheritable file descriptors by default. Python prefers the POSIX API over the native Windows API to have a single code base, and so it creates inheritable file descriptors. There is one exception: ``os.pipe()`` creates non-inheritable pipes on Windows, whereas it creates inheritable pipes on UNIX. The reason is an implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on Windows (native API), whereas it calls ``pipe()`` on UNIX (POSIX API). The call to ``CreatePipe()`` was added in Python in 1994, before the introduction of ``pipe()`` in the POSIX API in Windows 98. The `issue #4708 <http://bugs.python.org/issue4708>`_ proposes to change ``os.pipe()`` on Windows to create inheritable pipes. Inheritance of File Descriptors on Windows ------------------------------------------ On Windows, the native type of file objects are handles (C type ``HANDLE``). These handles have a ``HANDLE_FLAG_INHERIT`` flag which defines if a handle can be inherited in a child process or not. For the POSIX API, the C runtime (CRT) provides also file descriptors (C type ``int``). The handle of a file descriptor can be get using the function ``_get_osfhandle(fd)``. A file descriptor can be created from a handle using the function ``_open_osfhandle(handle)``. Using `CreateProcess() <http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%…>`_, handles are only inherited if their inheritable flag (``HANDLE_FLAG_INHERIT``) is set and if the ``bInheritHandles`` parameter of ``CreateProcess()`` is ``TRUE``; all file descriptors except standard streams (0, 1, 2) are closed in the child process, even if ``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all inheritable handles and all inheritable file descriptors are inherited in the child process. This function uses the undocumented fields *cbReserved2* and *lpReserved2* of the `STARTUPINFO <http://msdn.microsoft.com/en-us/library/windows/desktop/ms686331%28v=vs.85%…>`_ structure to pass an array of file descriptors. To replace standard streams (stdin, stdout, stderr) using ``CreateProcess()``, the ``STARTF_USESTDHANDLES`` flag must be set in the *dwFlags* field of the ``STARTUPINFO`` structure and the *bInheritHandles* parameter of ``CreateProcess()`` must be set to ``TRUE``. So when at least one standard stream is replaced, all inheritable handles are inherited by the child process. See also: * `Handle Inheritance <http://msdn.microsoft.com/en-us/library/windows/desktop/ms724466%28v=vs.85%…>`_ * `Q315939: PRB: Child Inherits Unintended Handles During CreateProcess Call <http://support.microsoft.com/kb/315939/en-us>`_ Inheritance of File Descriptors on UNIX --------------------------------------- POSIX provides a *close-on-exec* flag on file descriptors to close automatically a file descriptor when the C function ``execv()`` is called. File descriptors with the *close-on-exec* flag cleared are inherited in the child process, file descriptors with the flag set are closed in the child process. The flag can be set in two syscalls (one to get current flags, a second to set new flags) using ``fcntl()``:: int flags, res; flags = fcntl(fd, F_GETFD); if (flags == -1) { /* handle the error */ } flags |= FD_CLOEXEC; /* or "flags &= ~FD_CLOEXEC;" to clear the flag */ res = fcntl(fd, F_SETFD, flags); if (res == -1) { /* handle the error */ } FreeBSD, Linux, Mac OS X, NetBSD, OpenBSD and QNX support also setting the flag in a single syscall using ioctl():: int res; res = ioctl(fd, FIOCLEX, 0); if (!res) { /* handle the error */ } The *close-on-exec* flag has no effect on ``fork()``: all file descriptors are inherited by the child process. The `Python issue #16500 "Add an atfork module" <http://bugs.python.org/issue16500>`_ proposes to add a new ``atfork`` module to execute code at fork, it may be used to close automatically file descriptors. Issues with Inheritable File Descriptors ---------------------------------------- Most of the time, inheritable file descriptors "leaked" in child processes are not noticed, because they don't cause major bugs. It does not mean that these bugs must not be fixed. Two examples of common issues with inherited file descriptors: * On Windows, a directory cannot be removed before all file handles open in the directory are closed. The same issue can be seen with files, except if the file was created with the ``FILE_SHARE_DELETE`` flag (``O_TEMPORARY`` mode for ``open()``). * If a listening socket is leaked in a child process, the socket address cannot be reused before the parent and child processes terminated. For example, if a web server spawn a new program to handle a process, and the server restarts while the program is not done: the server cannot start because the TCP port is still in use. Leaking file descriptors is also a well known security vulnerability: read `FIO42-C. Ensure files are properly closed when they are no longer needed <https://www.securecoding.cert.org/confluence/display/seccode/FIO42-C.+Ensur…>`_ of the CERT. An untrusted child process can read sensitive data like passwords and take control of the parent process though leaked file descriptors. It is for example a way to escape from a chroot. With a leaked listening socket, a child process can accept new connections to read sensitive data. Atomic Creation of non-inheritable File Descriptors --------------------------------------------------- In a multithreaded application, a inheritable file descriptor can be created just before a new program is spawn, before the file descriptor is made non-inheritable. In this case, the file descriptor is leaked to the child process. This race condition could be avoided if the file descriptor is created directly non-inheritable. FreeBSD, Linux, Mac OS X, Windows and many other operating systems support creating non-inheritable file descriptors with the inheritable flag cleared atomically at the creation of the file descriptor. A new ``WSA_FLAG_NO_HANDLE_INHERIT`` flag for ``WSASocket()`` was added in Windows 7 SP1 and Windows Server 2008 R2 SP1 to create non-inheritable sockets. If this flag is used on an older Windows version (ex: Windows XP SP3), ``WSASocket()`` fails with ``WSAEPROTOTYPE``. On UNIX, new flags were added for files and sockets: * ``O_CLOEXEC``: available on Linux (2.6.23), FreeBSD (8.3), OpenBSD 5.0, Solaris 11, QNX, BeOS, next NetBSD release (6.1?). This flag is part of POSIX.1-2008. * ``SOCK_CLOEXEC`` flag for ``socket()`` and ``socketpair()``, available on Linux 2.6.27, OpenBSD 5.2, NetBSD 6.0. * ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24, OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0, Solaris 11. This flag is part of POSIX.1-2008. * ``fcntl()``: ``F_DUP2FD_CLOEXEC`` flag, available on FreeBSD 9.1 and Solaris 11. * ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23, NetBSD 6.0. On Linux older than 2.6.23, ``O_CLOEXEC`` flag is simply ignored. So ``fcntl()`` must be called to check if the file descriptor is non-inheritable: ``O_CLOEXEC`` is not supported if the ``FD_CLOEXEC`` flag is missing. On Linux older than 2.6.27, ``socket()`` or ``socketpair()`` fail with ``errno`` set to ``EINVAL`` if the ``SOCK_CLOEXEC`` flag is set in the socket type. New functions: * ``dup3()``: available on Linux 2.6.27 (and glibc 2.9) * ``pipe2()``: available on Linux 2.6.27 (and glibc 2.9) * ``accept4()``: available on Linux 2.6.28 (and glibc 2.10) On Linux older than 2.6.28, ``accept4()`` fails with ``errno`` set to ``ENOSYS``. Summary: =========================== =============== ==================================== Operating System Atomic File Atomic Socket =========================== =============== ==================================== FreeBSD 8.3 (2012) X Linux 2.6.23 (2007) 2.6.27 (2008) Mac OS X 10.8 (2012) X NetBSD 6.1 (?) 6.0 (2012) OpenBSD 5.0 (2011) 5.2 (2012) Solaris 11 (2011) X Windows XP (2001) Seven SP1 (2011), 2008 R2 SP1 (2011) =========================== =============== ==================================== Legend: * "Atomic File": first version of the operating system supporting creating atomically a non-inheritable file descriptor using ``open()`` * "Atomic Socket": first version of the operating system supporting creating atomically a non-inheritable socket * "X": not supported yet Status of Python 3.3 -------------------- Python 3.3 creates inheritable file descriptors on all platforms, except ``os.pipe()`` which creates non-inheritable file descriptors on Windows. New constants and functions related to the atomic creation of non-inheritable file descriptors were added to Python 3.3: ``os.O_CLOEXEC``, ``os.pipe2()`` and ``socket.SOCK_CLOEXEC``. On UNIX, the ``subprocess`` module closes all file descriptors in the child process by default, except standard streams (0, 1, 2) and file descriptors of the *pass_fds* parameter. If the *close_fds* parameter is set to ``False``, all inheritable file descriptors are inherited in the child process. On Windows, the ``subprocess`` closes all handles and file descriptors in the child process by default. If at least one standard stream (stdin, stdout or stderr) is replaced (ex: redirected into a pipe), all inheritable handles are inherited in the child process. All inheritable file descriptors are inherited by the child process using the functions of the ``os.execv*()`` and ``os.spawn*()`` families. On UNIX, the ``multiprocessing`` module uses ``os.fork()`` and so all file descriptors are inherited by child processes. On Windows, all inheritable handles are inherited by the child process using the ``multiprocessing`` module, all file descriptors except standard streams are closed. Summary: =========================== ============= ================== ============= Module FD on UNIX Handles on Windows FD on Windows =========================== ============= ================== ============= subprocess, default STD, pass_fds none STD subprocess, close_fds=False all all STD os.execv(), os.spawn() all all all multiprocessing all all STD =========================== ============= ================== ============= Legend: * "all": all *inheritable* file descriptors or handles are inherited in the child process * "none": all handles are closed in the child process * "STD": only file descriptors 0 (stdin), 1 (stdout) and 2 (stderr) are inherited in the child process * "pass_fds": file descriptors of the *pass_fds* parameter of the subprocess are inherited Proposal ======== Non-inheritable File Descriptors -------------------------------- The following functions are modified to make newly created file descriptors non-inheritable by default: * ``asyncore.dispatcher.create_socket()`` * ``io.FileIO`` * ``io.open()`` * ``open()`` * ``os.dup()`` * ``os.dup2()`` * ``os.fdopen()`` * ``os.open()`` * ``os.openpty()`` * ``os.pipe()`` * ``select.devpoll()`` * ``select.epoll()`` * ``select.kqueue()`` * ``socket.socket()`` * ``socket.socket.accept()`` * ``socket.socket.dup()`` * ``socket.socket.fromfd()`` * ``socket.socketpair()`` When available, atomic flags are used to make file descriptors non-inheritable. The atomicity is not guaranteed because a fallback is required when atomic flags are not available. New Functions ------------- * ``os.get_inheritable(fd: int)``: return ``True`` if the file descriptor can be inherited by child processes, ``False`` otherwise. * ``os.set_inheritable(fd: int, inheritable: bool)``: clear or set the inheritable flag of the specified file descriptor. These new functions are available on all platforms. On Windows, these functions accept also file descriptors of sockets: the result of ``sockobj.fileno()``. Other Changes ------------- * On UNIX, subprocess makes file descriptors of the *pass_fds* parameter inheritable. The file descriptor is made inheritable in the child process after the ``fork()`` and before ``execv()``, so the inheritable flag of file descriptors is unchanged in the parent process. * ``os.dup2(fd, fd2)`` makes *fd2* inheritable if *fd2* is ``0`` (stdin), ``1`` (stdout) or ``2`` (stderr) and *fd2* is different than *fd*. Backward Compatibility ====================== This PEP break applications relying on inheritance of file descriptors. Developers are encouraged to reuse the high-level Python module ``subprocess`` which handles the inheritance of file descriptors in a portable way. Applications using the ``subprocess`` module with the *pass_fds* parameter or using ``os.dup2()`` to redirect standard streams should not be affected. Python does no more conform to POSIX, since file descriptors are now made non-inheritable by default. Python was not designed to conform to POSIX, but was designed to develop portable applications. Related Work ============ The programming languages Go, Perl and Ruby make newly created file descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0 (1987) and Ruby 2.0 (2013). The SCons project overrides builtin functions ``file()`` and ``open()`` to make files non-inheritable on Windows: see `win32.py <https://bitbucket.org/scons/scons/src/c8dbbaa4598e7119ae80f72068386be105b5a…>`_. Rejected Alternatives ===================== PEP 433 ------- The PEP 433 entitled "Easier suppression of file descriptor inheritance" is a previous attempt proposing various other alternatives, but no consensus could be reached. No special case for standard streams ------------------------------------ Functions handling file descriptors should not handle standard streams (file descriptors ``0``, ``1``, ``2``) differently. This option does not work on Windows. On Windows, calling ``SetHandleInformation()`` to set or clear ``HANDLE_FLAG_INHERIT`` flag on standard streams (0, 1, 2) fails with the Windows error 87 (invalid argument). If ``os.dup2(fd, fd2)`` would always make *fd2* non-inheritable, the function would raise an exception when used to redirect standard streams. Another option is to add a new *inheritable* parameter to ``os.dup2()``. This PEP has a special-case for ``os.dup2()`` to not break backward compatibility on applications redirecting standard streams before calling the C function ``execv()``. Examples in the Python standard library: ``CGIHTTPRequestHandler.run_cgi()`` and ``pty.fork()`` use ``os.dup2()`` to redict stdin, stdout and stderr. Links ===== Python issues: * `#10115: Support accept4() for atomic setting of flags at socket creation <http://bugs.python.org/issue10115>`_ * `#12105: open() does not able to set flags, such as O_CLOEXEC <http://bugs.python.org/issue12105>`_ * `#12107: TCP listening sockets created without FD_CLOEXEC flag <http://bugs.python.org/issue12107>`_ * `#16850: Add "e" mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_ * `#16860: Use O_CLOEXEC in the tempfile module <http://bugs.python.org/issue16860>`_ * `#16946: subprocess: _close_open_fd_range_safe() does not set close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined <http://bugs.python.org/issue16946>`_ * `#17070: Use the new cloexec to improve security and avoid bugs <http://bugs.python.org/issue17070>`_ * `#18571: Implementation of the PEP 446: non-inheriable file descriptors <http://bugs.python.org/issue18571>`_ Other links: * `Secure File Descriptor Handling <http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper, 2008) * `Ghosts of Unix past, part 2: Conflated designs <http://lwn.net/Articles/412131/>`_ (Neil Brown, 2010) explains the history of ``O_CLOEXEC`` and ``O_NONBLOCK`` flags Copyright ========= This document has been placed into the public domain.

2 7

Green buildbot failure.
by Terry Reedy 11 Aug '13

11 Aug '13

This run recorded here shows a green test (it appears to have timed out) http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/7017 but the corresponding log for this Windows bot http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/7017/st… has the expected os.chown failure. Are such green failures intended? -- Terry Jan Reedy

5 6

Reaping threads and subprocesses
by Serhiy Storchaka 11 Aug '13

11 Aug '13

Some tests uses the following idiom: def test_main(): try: test.support.run_unittest(...) finally: test.support.reap_children() Other tests uses the following idiom: def test_main(): key = test.support.threading_setup() try: test.support.run_unittest(...) finally: test.support.threading_cleanup(*key) or in other words: @test.support.reap_threads def test_main(): test.support.run_unittest(...) These tests are not discoverable. There are some ways to make them discoverable. 1. Create unittest.TestCase subclasses or mixins with overloaded the run() method. class ThreadReaped: def run(self, result): key = test.support.threading_setup() try: return super().run(result) finally: test.support.threading_cleanup(*key) class ChildReaped: def run(self, result): try: return super().run(result) finally: test.support.reap_children() 2. Create unittest.TestCase subclasses or mixins with overloaded setUpClass() and tearDownClass() methods. class ThreadReaped: @classmethod def setUpClass(cls): cls._threads = test.support.threading_setup() @classmethod def tearDownClass(cls): test.support.threading_cleanup(*cls._threads) class ChildReaped: @classmethod def tearDownClass(cls): test.support.reap_children() 3. Create unittest.TestCase subclasses or mixins with overloaded setUp() and tearDown() methods. class ThreadReaped: def setUp(self): self._threads = test.support.threading_setup() def tearDown(self): test.support.threading_cleanup(*self._threads) class ChildReaped: def tearDown(self): test.support.reap_children() 4. Create unittest.TestCase subclasses or mixins with using addCleanup() in constructor. class ThreadReaped: def __init__(self): self.addCleanup(test.support.threading_cleanup, *test.support.threading_setup()) class ChildReaped: def __init__(self): self.addCleanup(test.support.reap_children) Of course instead subclassing we can use decorators which modify test class. What method is better? Do you have other suggestions? The issue where this problem was first occurred: http://bugs.python.org/issue16968.

1 0

Buildbot failure puzzle
by Terry Reedy 11 Aug '13

11 Aug '13

At least the following 3.4 buildbots have failed today with an error I do not understand: AMD64 FreeBSD, PPC64, x86Ubuntu, x86 WinServer 2003. Except for the Windows BB, it was the only failure and hence the only reason to not be green. ERROR: test_xmlcharnamereplace (test.test_codeccallbacks.CodecCallbackTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/test/test_codeccallbacks.py", line 112, in test_xmlcharnamereplace self.assertEqual(sin.encode("ascii", "test.xmlcharnamereplace"), sout) File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/test/test_codeccallbacks.py", line 102, in xmlcharnamereplace l.append("&%s;" % html.entities.codepoint2name[ord(c)]) AttributeError: 'module' object has no attribute 'entities' test_codeccallbacks.py: lines from 2008-05-17 line 002: import html.entities ... line 102: l.append("&%s;" % html.entities.codepoint2name[ord(c)]) I checked with an editor and these are the only two appearances of 'html' (so it was not rebound to anything else) and the spellings in the file are the same. Indeed, the same code has worked on at least some of the same machines. -- Terry Jan Reedy

2 1

Summary of Python tracker Issues
by Python tracker 09 Aug '13

09 Aug '13

ACTIVITY SUMMARY (2013-08-02 - 2013-08-09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4148 (+20) closed 26321 (+47) total 30469 (+67) Open issues with patches: 1874 Issues opened (56) ================== #4322: function with modified __name__ uses original name when there' http://bugs.python.org/issue4322 reopened by benjamin.peterson #17741: event-driven XML parser http://bugs.python.org/issue17741 reopened by pitrou #18630: mingw: exclude unix only modules http://bugs.python.org/issue18630 opened by rpetrov #18631: mingw: setup msvcrt and _winapi modules http://bugs.python.org/issue18631 opened by rpetrov #18632: mingw: build extensions with GCC http://bugs.python.org/issue18632 opened by rpetrov #18633: mingw: use Mingw32CCompiler as default compiler for mingw* bu http://bugs.python.org/issue18633 opened by rpetrov #18634: mingw find import library http://bugs.python.org/issue18634 opened by rpetrov #18636: mingw: setup _ssl module http://bugs.python.org/issue18636 opened by rpetrov #18637: mingw: export _PyNode_SizeOf as PyAPI for parser module http://bugs.python.org/issue18637 opened by rpetrov #18638: mingw: generalization of posix build in sysconfig.py http://bugs.python.org/issue18638 opened by rpetrov #18639: mingw: avoid circular dependency from time module during nativ http://bugs.python.org/issue18639 opened by rpetrov #18640: mingw: generalization of posix build in distutils/sysconfig.py http://bugs.python.org/issue18640 opened by rpetrov #18641: mingw: customize site http://bugs.python.org/issue18641 opened by rpetrov #18643: implement socketpair() on Windows http://bugs.python.org/issue18643 opened by neologix #18644: Got ResourceWarning: unclosed file when using test function fr http://bugs.python.org/issue18644 opened by vajrasky #18645: Add a configure option for performance guided optimization http://bugs.python.org/issue18645 opened by rhettinger #18646: Improve tutorial entry on 'Lambda Forms'. http://bugs.python.org/issue18646 opened by terry.reedy #18647: re.error: nothing to repeat http://bugs.python.org/issue18647 opened by serhiy.storchaka #18648: FP Howto and the PEP 8 lambda guildline http://bugs.python.org/issue18648 opened by terry.reedy #18650: intermittent test_pydoc failure on 3.4.0a1 http://bugs.python.org/issue18650 opened by ned.deily #18651: test failures on KFreeBSD http://bugs.python.org/issue18651 opened by doko #18652: Add itertools.first_true (return first true item in iterable) http://bugs.python.org/issue18652 opened by hynek #18653: mingw-meta: build core modules http://bugs.python.org/issue18653 opened by rpetrov #18654: modernize mingw&cygwin compiler classes http://bugs.python.org/issue18654 opened by rpetrov #18655: GUI apps take long to launch on Windows http://bugs.python.org/issue18655 opened by netrick #18659: test_precision in test_format.py is not executed and has unuse http://bugs.python.org/issue18659 opened by vajrasky #18660: os.read behavior on Linux http://bugs.python.org/issue18660 opened by dugres #18663: In unittest.TestCase.assertAlmostEqual doc specify the delta d http://bugs.python.org/issue18663 opened by py.user #18664: occasional test_threading failure http://bugs.python.org/issue18664 opened by pitrou #18667: missing HAVE_FCHOWNAT http://bugs.python.org/issue18667 opened by salinger #18669: curses.chgat() moves cursor, documentation says it shouldn't http://bugs.python.org/issue18669 opened by productivememberofsociety666 #18670: Using read_mime_types function from mimetypes module gives res http://bugs.python.org/issue18670 opened by vajrasky #18672: Fix format specifiers for debug output in _sre.c http://bugs.python.org/issue18672 opened by serhiy.storchaka #18673: Add and use O_TMPFILE for Linux 3.11 http://bugs.python.org/issue18673 opened by christian.heimes #18674: Store weak references in modules_by_index http://bugs.python.org/issue18674 opened by pitrou #18675: Daemon Threads can seg fault http://bugs.python.org/issue18675 opened by guettli #18676: Queue: document that zero is accepted as timeout value http://bugs.python.org/issue18676 opened by zyluo #18677: Enhanced context managers with ContextManagerExit and None http://bugs.python.org/issue18677 opened by kristjan.jonsson #18678: Wrong struct members name for spwd module http://bugs.python.org/issue18678 opened by vajrasky #18679: include a codec to handle escaping only control characters but http://bugs.python.org/issue18679 opened by underrun #18680: JSONDecoder should document that it raises a ValueError for ma http://bugs.python.org/issue18680 opened by corey #18681: typo in imp.reload http://bugs.python.org/issue18681 opened by felloak #18682: [PATCH] remove bogus codepath from pprint._safe_repr http://bugs.python.org/issue18682 opened by mvyskocil #18683: Core dumps on CentOS http://bugs.python.org/issue18683 opened by schlamar #18684: Pointers point out of array bound in _sre.c http://bugs.python.org/issue18684 opened by serhiy.storchaka #18685: Restore re performance to pre-PEP393 level http://bugs.python.org/issue18685 opened by serhiy.storchaka #18686: Tkinter focus_get on menu results in KeyError http://bugs.python.org/issue18686 opened by jgoeders #18687: Lib/test/leakers/test_ctypes.py still mentions the need to upd http://bugs.python.org/issue18687 opened by iwontbecreative #18688: Document undocumented Unicode object API http://bugs.python.org/issue18688 opened by serhiy.storchaka #18689: add argument for formatter to logging.Handler and subclasses i http://bugs.python.org/issue18689 opened by underrun #18690: memoryview not considered a sequence http://bugs.python.org/issue18690 opened by sfeltman #18691: sqlite3.Cursor.execute expects sequence as second argument. http://bugs.python.org/issue18691 opened by Andrew.Myers #18693: help() not helpful with enum http://bugs.python.org/issue18693 opened by ethan.furman #18694: getxattr on Linux ZFS native filesystem happily returns partia http://bugs.python.org/issue18694 opened by larry #18695: os.statvfs() not working well with unicode paths http://bugs.python.org/issue18695 opened by giampaolo.rodola #18696: In unittest.TestCase.longMessage doc remove a redundant senten http://bugs.python.org/issue18696 opened by py.user Most recent 15 issues with no replies (15) ========================================== #18696: In unittest.TestCase.longMessage doc remove a redundant senten http://bugs.python.org/issue18696 #18695: os.statvfs() not working well with unicode paths http://bugs.python.org/issue18695 #18694: getxattr on Linux ZFS native filesystem happily returns partia http://bugs.python.org/issue18694 #18691: sqlite3.Cursor.execute expects sequence as second argument. http://bugs.python.org/issue18691 #18690: memoryview not considered a sequence http://bugs.python.org/issue18690 #18689: add argument for formatter to logging.Handler and subclasses i http://bugs.python.org/issue18689 #18688: Document undocumented Unicode object API http://bugs.python.org/issue18688 #18687: Lib/test/leakers/test_ctypes.py still mentions the need to upd http://bugs.python.org/issue18687 #18684: Pointers point out of array bound in _sre.c http://bugs.python.org/issue18684 #18681: typo in imp.reload http://bugs.python.org/issue18681 #18675: Daemon Threads can seg fault http://bugs.python.org/issue18675 #18672: Fix format specifiers for debug output in _sre.c http://bugs.python.org/issue18672 #18670: Using read_mime_types function from mimetypes module gives res http://bugs.python.org/issue18670 #18669: curses.chgat() moves cursor, documentation says it shouldn't http://bugs.python.org/issue18669 #18663: In unittest.TestCase.assertAlmostEqual doc specify the delta d http://bugs.python.org/issue18663 Most recent 15 issues waiting for review (15) ============================================= #18696: In unittest.TestCase.longMessage doc remove a redundant senten http://bugs.python.org/issue18696 #18695: os.statvfs() not working well with unicode paths http://bugs.python.org/issue18695 #18694: getxattr on Linux ZFS native filesystem happily returns partia http://bugs.python.org/issue18694 #18687: Lib/test/leakers/test_ctypes.py still mentions the need to upd http://bugs.python.org/issue18687 #18685: Restore re performance to pre-PEP393 level http://bugs.python.org/issue18685 #18684: Pointers point out of array bound in _sre.c http://bugs.python.org/issue18684 #18682: [PATCH] remove bogus codepath from pprint._safe_repr http://bugs.python.org/issue18682 #18678: Wrong struct members name for spwd module http://bugs.python.org/issue18678 #18677: Enhanced context managers with ContextManagerExit and None http://bugs.python.org/issue18677 #18676: Queue: document that zero is accepted as timeout value http://bugs.python.org/issue18676 #18674: Store weak references in modules_by_index http://bugs.python.org/issue18674 #18672: Fix format specifiers for debug output in _sre.c http://bugs.python.org/issue18672 #18670: Using read_mime_types function from mimetypes module gives res http://bugs.python.org/issue18670 #18663: In unittest.TestCase.assertAlmostEqual doc specify the delta d http://bugs.python.org/issue18663 #18659: test_precision in test_format.py is not executed and has unuse http://bugs.python.org/issue18659 Top 10 most discussed issues (10) ================================= #18652: Add itertools.first_true (return first true item in iterable) http://bugs.python.org/issue18652 27 msgs #17741: event-driven XML parser http://bugs.python.org/issue17741 23 msgs #18264: enum.IntEnum is not compatible with JSON serialisation http://bugs.python.org/issue18264 22 msgs #18647: re.error: nothing to repeat http://bugs.python.org/issue18647 18 msgs #18606: Add statistics module to standard library http://bugs.python.org/issue18606 17 msgs #18629: future division breaks timedelta division by integer http://bugs.python.org/issue18629 16 msgs #18659: test_precision in test_format.py is not executed and has unuse http://bugs.python.org/issue18659 15 msgs #15651: PEP 3121, 384 refactoring applied to elementtree module http://bugs.python.org/issue15651 12 msgs #18677: Enhanced context managers with ContextManagerExit and None http://bugs.python.org/issue18677 11 msgs #16853: add a Selector to the select module http://bugs.python.org/issue16853 9 msgs Issues closed (45) ================== #3591: elementtree tests do not include bytes handling http://bugs.python.org/issue3591 closed by eli.bendersky #4885: mmap enhancement request http://bugs.python.org/issue4885 closed by pitrou #8860: Rounding in timedelta constructor is inconsistent with that in http://bugs.python.org/issue8860 closed by belopolsky #8998: add crypto routines to stdlib http://bugs.python.org/issue8998 closed by gregory.p.smith #10427: 24:00 Hour in DateTime http://bugs.python.org/issue10427 closed by belopolsky #10897: UNIX mmap unnecessarily dup() file descriptor http://bugs.python.org/issue10897 closed by neologix #13083: _sre: getstring() releases the buffer before using it http://bugs.python.org/issue13083 closed by serhiy.storchaka #13612: xml.etree.ElementTree says unknown encoding of a regular encod http://bugs.python.org/issue13612 closed by eli.bendersky #14323: Normalize math precision in RGB/YIQ conversion http://bugs.python.org/issue14323 closed by serhiy.storchaka #15301: os.chown: OverflowError: Python int too large to convert to C http://bugs.python.org/issue15301 closed by larry #15866: encode(..., 'xmlcharrefreplace') produces entities for surroga http://bugs.python.org/issue15866 closed by serhiy.storchaka #15966: concurrent.futures: Executor.submit keyword arguments may not http://bugs.python.org/issue15966 closed by mark.dickinson #16067: UAC prompt for installation shows temporary file name http://bugs.python.org/issue16067 closed by loewis #16741: `int()`, `float()`, etc think python strings are null-terminat http://bugs.python.org/issue16741 closed by serhiy.storchaka #17011: ElementPath ignores different namespace mappings for the same http://bugs.python.org/issue17011 closed by eli.bendersky #17046: test_subprocess test_executable_without_cwd fails when run wit http://bugs.python.org/issue17046 closed by ned.deily #17216: sparc linux build fails with "could not import runpy module" http://bugs.python.org/issue17216 closed by ned.deily #17372: provide pretty printer for xml.etree.ElementTree http://bugs.python.org/issue17372 closed by eli.bendersky #17478: Tkinter's split() inconsistent for bytes and unicode strings http://bugs.python.org/issue17478 closed by serhiy.storchaka #17902: Document that _elementtree C API cannot use custom TreeBuilder http://bugs.python.org/issue17902 closed by eli.bendersky #17934: Add a frame method to clear expensive details http://bugs.python.org/issue17934 closed by pitrou #18151: Idlelib: update to "with open ... except OSError" (in 2.7, le http://bugs.python.org/issue18151 closed by terry.reedy #18201: distutils write into symlinks instead of replacing them http://bugs.python.org/issue18201 closed by mgorny #18273: Simplify calling and discovery of json test package http://bugs.python.org/issue18273 closed by ezio.melotti #18357: add tests for dictview set difference operations http://bugs.python.org/issue18357 closed by ezio.melotti #18396: test_signal.test_issue9324() fails on buildbot AMD64 Windows7 http://bugs.python.org/issue18396 closed by python-dev #18443: Misc/Readme still documents TextMate http://bugs.python.org/issue18443 closed by ezio.melotti #18532: hashlib.HASH objects should officially expose the hash name http://bugs.python.org/issue18532 closed by christian.heimes #18563: No unit test for yiq to rgb and rgb to yiq converting functio http://bugs.python.org/issue18563 closed by serhiy.storchaka #18570: OverflowError in division: wrong message http://bugs.python.org/issue18570 closed by mark.dickinson #18581: Duplicate test and missing class test in test_abc.py http://bugs.python.org/issue18581 closed by ezio.melotti #18621: site.py keeps too much stuff alive when it patches builtins http://bugs.python.org/issue18621 closed by pitrou #18635: Enum sets _member_type_ to instantiated values but not the cla http://bugs.python.org/issue18635 closed by python-dev #18642: enhancement for operator 'assert' http://bugs.python.org/issue18642 closed by mark.dickinson #18649: list2cmdline function in subprocess module handles \" sequence http://bugs.python.org/issue18649 closed by sbt #18656: setting function.__name__ doesn't affect repr() http://bugs.python.org/issue18656 closed by benjamin.peterson #18657: Remove duplicate ACKS entries http://bugs.python.org/issue18657 closed by r.david.murray #18658: Mercurial CPython log ticket link is broken http://bugs.python.org/issue18658 closed by ned.deily #18661: Typo in grpmodule.c http://bugs.python.org/issue18661 closed by mark.dickinson #18662: re.escape should not escape the hyphen http://bugs.python.org/issue18662 closed by jjl #18665: Typos in frame object related code http://bugs.python.org/issue18665 closed by pitrou #18666: Unused variable in test_frame.py http://bugs.python.org/issue18666 closed by pitrou #18668: Properly document setting m_size in PyModuleDef http://bugs.python.org/issue18668 closed by python-dev #18671: enhance formatting in logging package http://bugs.python.org/issue18671 closed by vinay.sajip #18692: Connection change in compiled code http://bugs.python.org/issue18692 closed by r.david.murray

1 0

xml.etree.ElementTree.IncrementalParser (was: ElementTree iterparse string)
by Stefan Behnel 09 Aug '13

09 Aug '13

[from python-ideas] Antoine Pitrou, 07.08.2013 08:04: > Take a look at IncrementalParser: > http://docs.python.org/dev/library/xml.etree.elementtree.html#incremental-p… Hmm, that seems to be a somewhat recent addition (April 2013). I would have preferred hearing about it before it got added. I don't like the fact that it adds a second interface to iterparse() that allows injecting arbitrary content into the parser. You can now run iterparse() to read from a file, and at an arbitrary iteration position, send it a byte string to parse from, before it goes reading more data from the file. Or take out some events before iteration continues. I think the implementation should be changed to make iterparse() return something that wraps an IncrementalParser, not something that is an IncrementalParser. Also, IMO it should mimic the interface of the TreeBuilder, which calls the data reception method "data()" and the termination method "close()". There is no reason to add yet another set of methods names just to do what others do already. Stefan

2 4

The Return Of Argument Clinic
by Larry Hastings 08 Aug '13

08 Aug '13

It's time to discuss Argument Clinic again. I think the implementation is ready for public scrutiny. (It was actually ready a week ago, but I lost a couple of days to "make distclean" corrupting my hg data store--yes, I hadn't upped my local clinic branch in a while. Eventually I gave up on repairing it and just brute-forcd it. Anyway...) My Clinic test branch is here: https://bitbucket.org/larry/python-clinic/ And before you ask, no, the above branch should never ever ever be merged back into trunk. We'll start clean once Clinic is ready for merging and do a nice neat job. ___________________________________________________________________ There's no documentation, apart from the PEP. But you can see plenty of test cases of using Clinic, just grep for the string "clinic" in */*.c. But for reference here's the list: Modules/_cursesmodule.c Modules/_datetimemodule.c Modules/_dbmmodule.c Modules/posixmodule.c Modules/unicodedata.c Modules/_weakref.c Modules/zlibmodule.c Objects/dictobject.c Objects/unicodeobject.c I haven't reimplemented every PyArg_ParseTuple "format unit" in the retooled Clinic, so it's not ready to try with every single builtin yet. The syntax is as Guido dictated it during our meeting after the Language Summit at PyCon US 2013. The implementation has been retooled, several times, and is now both nicer and more easily extensible. The internals are just a little messy, but the external interfaces are all ready for critique. ___________________________________________________________________ Here are the external interfaces as I forsee them. If you add your own data types, you'll subclass "Converter" and maybe "ReturnConverter". Take a look at the existing subclasses to get a feel for what that's like. If you implemented your own DSL, you'd make something that quacked like "PythonParser" (implementing __init__ and parse methods), and you'd deal with "Block", "Module", "Class", "Function", and "Parameter" objects a lot. What do you think? ___________________________________________________________________ What follows are six questions I'd like to put to the community, ranked oddly enough in order of how little to how much I care about the answer. BTW, by convention, every time I need a arbitrary sample function I use "os.stat". (Please quote the question line in your responses, otherwise I fear we'll get lost in the sea of text.) ___________________________________________________________________ Question 0: How should we integrate Clinic into the build process? Clinic presents a catch-22: you want it as part of the build process, but it needs Python to be built before it'll run. Currently it requires Python 3.3 or newer; it might work in 3.2, I've never tried it. We can't depend on Python 3 being available when we build. This complicates the build process somewhat. I imagine it's a solvable problem on UNIX... with the right wizardry. I have no idea how one'd approach it on Windows, but obviously we need to solve the problem there too. ___________________________________________________________________ Question 1: Which C function nomenclature? Argument Clinic generates two functions prototypes per Python function: one specifying one of the traditional signatures for builtins, whose code is generated completely by Clinic, and the other with a custom-generated signature for just that call whose code is written by the user. Currently the former doesn't have any specific name, though I have been thinking of it as the "parse" function. The latter is definitely called the "impl" (pronounced IM-pull), short for "implementation". When Clinic generates the C code, it uses the name of the Python function to create the C functions' names, with underscores in place of dots. Currently the "parse" function gets the base name ("os_stat"), and the "impl" function gets an "_impl" added to the end ("os_stat_impl"). Argument Clinic is agnostic about the names of these functions. It's possible it'd be nicer to name these the other way around, say "os_stat_parse" for the parse function and "os_stat" for the impl. Anyone have a strong opinion one way or the other? I don't much care; all I can say is that the "obvious" way to do it when I started was to add "_impl" to the impl, as it is the new creature under the sun. ___________________________________________________________________ Question 2: Emit code for modules and classes? Argument Clinic now understands the structure of the modules and classes it works with. You declare them like so: module os class os.ImaginaryClassHere def os.ImaginaryClassHere.stat(...): ... Currently it does very little with the information; right now it mainly just gets baked into the documentation. In the future I expect it to get used in the introspection metadata, and it'll definitely be relevant to external consumers of the Argument Clinic information (IDEs building per-release databases, other implementations building metadata for library interface conformance testing). Another way we could use this metadata: have Argument Clinic generate more of the boilerplate for a class or module. For example, it could kick out all the PyMethodDef structures for the class or module. If we grew Argument Clinic some, and taught it about the data members of classes and modules, it could also generate the PyModuleDef and PyTypeObject structures, and even generate a function that initialized them at runtime for you. (Though that does seem like mission creep to me.) There are some complications to this, one of which I'll discuss next. But I put it to you, gentle reader: how much boilerplate should Argument Clinic undertake to generate, and how much more class and module metadata should be wired in to it? ___________________________________________________________________ Question 3: #ifdef support for functions? Truth be told, I did experiment with having Argument Clinic generate more of the boilerplate associated with modules. Clinic already generates a macro per function defining that function's PyMethodDef structure, for example: #define OS_STAT_METHODDEF \ {"stat", (PyCFunction)os_stat, \ METH_VARARGS|METH_KEYWORDS, os_stat__doc__} For a while I had it generating the PyMethodDef structures, like so: /*[clinic] generate_method_defs os [clinic]*/ #define OS_METHODDEFS \ OS_STAT_METHODDEF, \ OS_ACCESS_METHODDEF, \ OS_TTYNAME_METHODDEF, \ static PyMethodDef os_methods[] = { OS_METHODDEFS /* existing methoddefs here... */ NULL } But I ran into trouble with os.ttyname(), which is only created and exposed if the platform defines HAVE_TTYNAME. Initially I'd just thrown all the Clinic stuff relevant to os.ttyname in the #ifdef block. But Clinic pays no attention to #ifdef statements--so it would still add OS_TTYNAME_METHODDEF, to OS_METHODDEFS. And kablooey! Right now I've backed out of this--I had enough to do without getting off into extra credit like this. But I'd like to return to it. It just seems natural to have Clinic generate this nasty boilerplate. Four approaches suggest themselves to me, listed below in order of least- to most-preferable in my opinion: 0) Don't have Clinic participate in populating the PyMethodDefs. 1) Teach Clinic to understand simple C preprocessor statements, just enough so it implicitly understands that os.ttyname was defined inside an #ifdef HAVE_TTYPE block. It would then intelligently generate the code to take this into account. 2) Explicitly tell Clinic that os.ttyname must have HAVE_TTYNAME defined in order to be active. Clinic then generates the code intelligently taking this into account, handwave handwave. 3) Change the per-function methoddef macro to have the trailing comma: #define OS_STAT_METHODDEF \ {"stat", (PyCFunction)os_stat, \ METH_VARARGS|METH_KEYWORDS, os_stat__doc__}, and suppress it in the macro Clinic generates: /*[clinic] generate_method_defs os [clinic]*/ #define OS_METHODDEFS \ OS_STAT_METHODDEF \ OS_ACCESS_METHODDEF \ OS_TTYNAME_METHODDEF \ And then the code surrounding os.ttyname can look like this: #ifdef HAVE_TTYNAME // ... real os.ttyname stuff here #else #define OS_STAT_TTYNAME #endif And I think that would work great, actually. But I haven't tried it. Do you agree that Argument Clinic should generate this information, and it should use the approach in 3) ? ___________________________________________________________________ Question 4: Return converters returning success/failure? With the addition of the "return converter", we have the lovely feature of being able to *return* a C type and have it converted back into a Python type. Your C extensions have never been more readable! The problem is that the PyObject * returned by a C builtin function serves two simultaneous purposes: it contains the return value on success, but also it is NULL if the function threw an exception. We can probably still do that for all pointer-y return types (I'm not sure, I haven't played with it yet). But if the impl function now returns "int", or some decidedly other non-pointer-y type, there's no longer a magic return value we can use to indicate "we threw an exception". This isn't the end of the world; I can detect that the impl threw an exception by calling PyErr_Occurred(). But I've been chided before for calling this unnecessarily; it's ever-so slightly expensive, in that it has to dereference TLS, and does so with an atomic operation. Not to mention that it's a function call! The impl should know whether or not it failed. So it's the interface we're defining that forces it to throw away that information. If we provided a way for it to return that information, we could shave off some cycles. The problem is, how do we do that in a way that doesn't suck? Four approaches suggest themselves to me, and sadly I think they all suck to one degree or another. In order of sucking least to most: 0) Return the real type and detect the exception with PyErr_Occurred(). This is by far the loveliest option, but it incurs runtime overhead. 1) Have the impl take an extra parameter, "int *failed". If the function fails, it sets that to a true value and returns whatever. 2) Have the impl return its calculated return value through an extra pointer-y parameter ("int *return_value"), and its actual return value is an int indicating success or failure. 3) Have the impl return a structure containing both the real return value and a success/failure integer. Then its return lines would look like this: return {-1, 0}; or maybe return {-3, PY_HORRIBLE_CLINIC_INTERFACE__SUCCESS}; Can we live with PyErr_Occurred() here? ___________________________________________________________________ Question 5: Keep too-magical class decorator Converter.wrap? Converter is the base class for converter objects, the objects that handle the details of converting a Python object into its C equivalent. The signature for Converter.__init__ has become complicated: def __init__(self, name, function, default=unspecified, *, doc_default=None, required=False) "name" is the name of the function ("stat"), "function" is an object representing the function for which this Converter is handling an argument (duck-type compatible with inspect.Signature), and default is the default (Python) value if any. "doc_default" is a string that overrides repr(default) in the documentation, handy if repr(default) is too ugly or you just want to mislead the user. "required", if True specifies that the parameter should be considered required, even if it has a default value. Complicating the matter further, converter subclasses may take extra (keyword-only and optional) parameters to configure exotic custom behavior. For example, the "Py_buffer" converter takes "zeroes" and "nullable"; the "path_t" converter implemented in posixmodule.c takes "allow_fd" and "nullable". This means that converter subclasses have to define a laborious __init__, including three parameters with defaults, then turn right around and pass most of the parameters back into super().__init__. This interface has changed several times during the development of Clinic, and I got tired of fixing up all my existing prototypes and super calls. So I made a class decorator that did it for me. Shield your eyes from the sulferous dark wizardry of Converter.wrap: @staticmethod def wrap(cls): class WrappedConverter(cls, Converter): def __init__(self, name, function, default=unspecified, *, doc_default=None, required=False, **kwargs): super(cls, self).__init__(name, function, default, doc_default=doc_default, required=required) cls.__init__(self, **kwargs) return functools.update_wrapper(WrappedConverter, cls, updated=()) When you decorate your class with Converter.wrap, you only define in your __init__ your custom arguments. All the arguments Converter.__init__ cares about are taken care of for you (aka hidden from you). As an example, here's the relevant bits of path_t_converter from posixmodule.c: @Converter.wrap class path_t_converter(Converter): def __init__(self, *, allow_fd=False, nullable=False): ... So on the one hand I admit it's smelly. On the other hand it hides a lot of stuff that the user needn't care about, and it makes the code simpler and easier to read. And it means we can change the required arguments for Converter.__init__ without breaking any code (as I have already happily done once or twice). I'd like to keep it in, and anoint it as the preferred way of declaring Converter subclasses. Anybody else have a strong opinion on this either way? (I don't currently have an equivalent mechanism for return converters--their interface is a lot simpler, and I just haven't needed it so far.) ___________________________________________________________________ Well! That's quite enough for now. //arry/

7 10

PEP 442 clarification for type hierarchies
by Stefan Behnel 08 Aug '13

08 Aug '13

Hi, I'm currently catching up on PEP 442, which managed to fly completely below my radar so far. It's a really helpful change that could end up fixing a major usability problem that Cython was suffering from: user provided deallocation code now has a safe execution environment (well, at least in Py3.4+). That makes Cython a prime candidate for testing this, and I've just started to migrate the implementation. One thing that I found to be missing from the PEP is inheritance handling. The current implementation doesn't seem to care about base types at all, so it appears to be the responsibility of the type to call its super type finalisation function. Is that really intended? Couldn't the super type call chain be made a part of the protocol? Another bit is the exception handling. According to the documentation, tp_finalize() is supposed to first save the current exception state, then do the cleanup, then call WriteUnraisable() if necessary, then restore the exception state. http://docs.python.org/3.4/c-api/typeobj.html#PyTypeObject.tp_finalize Is there a reason why this is left to the user implementation, rather than doing it generically right in PyObject_CallFinalizer() ? That would also make it more efficient to call through the super type hierarchy, I guess. I don't see a need to repeat this exception state swapping at each level. So, essentially, I'm wondering whether PyObject_CallFinalizer() couldn't just set up the execution environment and then call all finalisers of the type hierarchy in bottom-up order. Stefan

2 16

Python-Dev August 2013