Mailman 3 August 2013 - Python-Dev

NULL allowed in PyErr_SetString and friends?
by Antoine Pitrou Aug. 12, 2013

Aug. 12, 2013

Hello, It seems NULL is allowed as the first argument of PyErr_Format, PyErr_SetString and PyErr_SetObject. Moreover, it means "clear the error indicator". However, this is not mentioned in the docs. I was wondering if we should officialize this behaviour or change it. Regards Antoine.

2 1

Strange artifacts with PEP 3121 and monkey-patching sys.modules (in csv, ElementTree and others)
by Eli Bendersky Aug. 12, 2013

Aug. 12, 2013

Hello, Recently as part of the effort of untangling the tests of ElementTree and general code improvements (e.g. http://bugs.python.org/issue15651), I ran into something strange about PEP 3121-compliant modules. I'll demonstrate with csv, just as an example. PEP 3121 mandates this function to look up the module-specific state in the current sub-interpreter: PyObject* PyState_FindModule(struct PyModuleDef*); This appears to make the following assumption: a given sub-interpreter only … [View More]imports any C extension *once*. If it happens more than once, the assumption breaks in troubling ways. In normal code, it should never happen more than once because of the caching in sys.modules; However, many of our tests monkey-patch sys.modules (mainly by calling test.support.import_fresh_module) and hell breaks use. Here's a simple example: ---- import sys csv = __import__('csv') csv.register_dialect('unixpwd', delimiter=':', quoting=csv.QUOTE_NONE) print(csv.list_dialects()) # ==> ['unixpwd', 'excel-tab', 'excel', 'unix'] del sys.modules['csv'] # FUN del sys.modules['_csv'] some_other_csv = __import__('csv') print(csv.list_dialects()) # ==> ['excel-tab', 'excel', 'unix'] ---- Note how doing some sys.modules acrobatics and re-importing suddenly changes the internal state of a previously imported module. This happens because: 1. The first import of 'csv' (which then imports `_csv) creates module-specific state on the heap and associates it with the current sub-interpreter. The list of dialects, amongst other things, is in that state. 2. The 'del's wipe 'csv' and '_csv' from the cache. 3. The second import of 'csv' also creates/initializes a new '_csv' module because it's not in sys.modules. This *replaces* the per-sub-interpreter cached version of the module's state with the clean state of a new module So essentially, while PEP 3121 moves state from C-file globals to per-module state, the state is still global, and this fact can be exposed from pure Python code. The above is a toy example. Here's a more serious case I ran into with ET, but once again is demonstrated with 'csv' for simplicity: ---- import io from test.support import import_fresh_module import csv csv_other = import_fresh_module('csv', fresh=['_csv', 'csv']) f = io.StringIO('foo\x00,bar\nbaz,42') reader = csv.reader(f) try: for row in reader: print(row) except csv.Error as e: print('Caught csv.error', e) except Exception as e: print('Caught Exception', e) ---- In the above, the reader throws 'csv.Error' (because of the NULL byte) but the exception clause does not catch it where expected, because it's a different exception class called `csv.Error`, due to the same problem demonstrated above (if the seemingly innocent import_fresh_module is removed, all is good). Any ideas/suggestion regarding this are welcome. This is quite an esoteric problem, but I believe it's serious. PEP 3121 is not used much (yet), but recently there was talk again about committing some of the patches created for converting Modules/*.c extensions to it during a GSoC project. I believe that we should understand the implications first. There can be a number of solutions; including modifying the PEP 3121 implementation machinery to really create/keep state "per module" and not just "per kind of module in a single sub-interpreter". Eli [View Less]

4 26

(New) PEP 446: Make newly created file descriptors non-inheritable
by Victor Stinner Aug. 12, 2013

Aug. 12, 2013

Hi, My second try (old PEP 446) to change how the inheritance of file descriptors was somehow rejected. Here is a third try (new PEP 446) which should include all information of the recent discussions on python-dev, especially how file descriptors and handles are inherited on Windows. I added tables to summarize the status of Python 3.3 and the status of atomic flags. I rewrote the rationale to introduce and explain the inheritance of file descriptors, which is not an simple subject. I … [View More]removed the O_NONBLOCK flag from the PEP, it is a different issue and a new PEP must be written. Online HTML version: http://www.python.org/dev/peps/pep-0446/ You may also want to read my first attempt: http://www.python.org/dev/peps/pep-0433/ PEP: 446 Title: Make newly created file descriptors non-inheritable Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-August-2013 Python-Version: 3.4 Abstract ======== Leaking file descriptors in child processes causes various annoying issues and is a known major security vulnerability. This PEP proposes to make all file descriptors created by Python non-inheritable by default to reduces the risk of these issues. This PEP fixes also a race condition in multithreaded applications on operating systems supporting atomic flags to create non-inheritable file descriptors. Rationale ========= Inheritance of File Descriptors ------------------------------- Each operating system handles the inheritance of file descriptors differently. Windows creates non-inheritable file descriptors by default, whereas UNIX creates inheritable file descriptors by default. Python prefers the POSIX API over the native Windows API to have a single code base, and so it creates inheritable file descriptors. There is one exception: ``os.pipe()`` creates non-inheritable pipes on Windows, whereas it creates inheritable pipes on UNIX. The reason is an implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on Windows (native API), whereas it calls ``pipe()`` on UNIX (POSIX API). The call to ``CreatePipe()`` was added in Python in 1994, before the introduction of ``pipe()`` in the POSIX API in Windows 98. The `issue #4708 <http://bugs.python.org/issue4708>`_ proposes to change ``os.pipe()`` on Windows to create inheritable pipes. Inheritance of File Descriptors on Windows ------------------------------------------ On Windows, the native type of file objects are handles (C type ``HANDLE``). These handles have a ``HANDLE_FLAG_INHERIT`` flag which defines if a handle can be inherited in a child process or not. For the POSIX API, the C runtime (CRT) provides also file descriptors (C type ``int``). The handle of a file descriptor can be get using the function ``_get_osfhandle(fd)``. A file descriptor can be created from a handle using the function ``_open_osfhandle(handle)``. Using `CreateProcess() <http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%…>`_, handles are only inherited if their inheritable flag (``HANDLE_FLAG_INHERIT``) is set and if the ``bInheritHandles`` parameter of ``CreateProcess()`` is ``TRUE``; all file descriptors except standard streams (0, 1, 2) are closed in the child process, even if ``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all inheritable handles and all inheritable file descriptors are inherited in the child process. This function uses the undocumented fields *cbReserved2* and *lpReserved2* of the `STARTUPINFO <http://msdn.microsoft.com/en-us/library/windows/desktop/ms686331%28v=vs.85%…>`_ structure to pass an array of file descriptors. To replace standard streams (stdin, stdout, stderr) using ``CreateProcess()``, the ``STARTF_USESTDHANDLES`` flag must be set in the *dwFlags* field of the ``STARTUPINFO`` structure and the *bInheritHandles* parameter of ``CreateProcess()`` must be set to ``TRUE``. So when at least one standard stream is replaced, all inheritable handles are inherited by the child process. See also: * `Handle Inheritance <http://msdn.microsoft.com/en-us/library/windows/desktop/ms724466%28v=vs.85%…>`_ * `Q315939: PRB: Child Inherits Unintended Handles During CreateProcess Call <http://support.microsoft.com/kb/315939/en-us>`_ Inheritance of File Descriptors on UNIX --------------------------------------- POSIX provides a *close-on-exec* flag on file descriptors to close automatically a file descriptor when the C function ``execv()`` is called. File descriptors with the *close-on-exec* flag cleared are inherited in the child process, file descriptors with the flag set are closed in the child process. The flag can be set in two syscalls (one to get current flags, a second to set new flags) using ``fcntl()``:: int flags, res; flags = fcntl(fd, F_GETFD); if (flags == -1) { /* handle the error */ } flags |= FD_CLOEXEC; /* or "flags &= ~FD_CLOEXEC;" to clear the flag */ res = fcntl(fd, F_SETFD, flags); if (res == -1) { /* handle the error */ } FreeBSD, Linux, Mac OS X, NetBSD, OpenBSD and QNX support also setting the flag in a single syscall using ioctl():: int res; res = ioctl(fd, FIOCLEX, 0); if (!res) { /* handle the error */ } The *close-on-exec* flag has no effect on ``fork()``: all file descriptors are inherited by the child process. The `Python issue #16500 "Add an atfork module" <http://bugs.python.org/issue16500>`_ proposes to add a new ``atfork`` module to execute code at fork, it may be used to close automatically file descriptors. Issues with Inheritable File Descriptors ---------------------------------------- Most of the time, inheritable file descriptors "leaked" in child processes are not noticed, because they don't cause major bugs. It does not mean that these bugs must not be fixed. Two examples of common issues with inherited file descriptors: * On Windows, a directory cannot be removed before all file handles open in the directory are closed. The same issue can be seen with files, except if the file was created with the ``FILE_SHARE_DELETE`` flag (``O_TEMPORARY`` mode for ``open()``). * If a listening socket is leaked in a child process, the socket address cannot be reused before the parent and child processes terminated. For example, if a web server spawn a new program to handle a process, and the server restarts while the program is not done: the server cannot start because the TCP port is still in use. Leaking file descriptors is also a well known security vulnerability: read `FIO42-C. Ensure files are properly closed when they are no longer needed <https://www.securecoding.cert.org/confluence/display/seccode/FIO42-C.+Ensur…>`_ of the CERT. An untrusted child process can read sensitive data like passwords and take control of the parent process though leaked file descriptors. It is for example a way to escape from a chroot. With a leaked listening socket, a child process can accept new connections to read sensitive data. Atomic Creation of non-inheritable File Descriptors --------------------------------------------------- In a multithreaded application, a inheritable file descriptor can be created just before a new program is spawn, before the file descriptor is made non-inheritable. In this case, the file descriptor is leaked to the child process. This race condition could be avoided if the file descriptor is created directly non-inheritable. FreeBSD, Linux, Mac OS X, Windows and many other operating systems support creating non-inheritable file descriptors with the inheritable flag cleared atomically at the creation of the file descriptor. A new ``WSA_FLAG_NO_HANDLE_INHERIT`` flag for ``WSASocket()`` was added in Windows 7 SP1 and Windows Server 2008 R2 SP1 to create non-inheritable sockets. If this flag is used on an older Windows version (ex: Windows XP SP3), ``WSASocket()`` fails with ``WSAEPROTOTYPE``. On UNIX, new flags were added for files and sockets: * ``O_CLOEXEC``: available on Linux (2.6.23), FreeBSD (8.3), OpenBSD 5.0, Solaris 11, QNX, BeOS, next NetBSD release (6.1?). This flag is part of POSIX.1-2008. * ``SOCK_CLOEXEC`` flag for ``socket()`` and ``socketpair()``, available on Linux 2.6.27, OpenBSD 5.2, NetBSD 6.0. * ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24, OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0, Solaris 11. This flag is part of POSIX.1-2008. * ``fcntl()``: ``F_DUP2FD_CLOEXEC`` flag, available on FreeBSD 9.1 and Solaris 11. * ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23, NetBSD 6.0. On Linux older than 2.6.23, ``O_CLOEXEC`` flag is simply ignored. So ``fcntl()`` must be called to check if the file descriptor is non-inheritable: ``O_CLOEXEC`` is not supported if the ``FD_CLOEXEC`` flag is missing. On Linux older than 2.6.27, ``socket()`` or ``socketpair()`` fail with ``errno`` set to ``EINVAL`` if the ``SOCK_CLOEXEC`` flag is set in the socket type. New functions: * ``dup3()``: available on Linux 2.6.27 (and glibc 2.9) * ``pipe2()``: available on Linux 2.6.27 (and glibc 2.9) * ``accept4()``: available on Linux 2.6.28 (and glibc 2.10) On Linux older than 2.6.28, ``accept4()`` fails with ``errno`` set to ``ENOSYS``. Summary: =========================== =============== ==================================== Operating System Atomic File Atomic Socket =========================== =============== ==================================== FreeBSD 8.3 (2012) X Linux 2.6.23 (2007) 2.6.27 (2008) Mac OS X 10.8 (2012) X NetBSD 6.1 (?) 6.0 (2012) OpenBSD 5.0 (2011) 5.2 (2012) Solaris 11 (2011) X Windows XP (2001) Seven SP1 (2011), 2008 R2 SP1 (2011) =========================== =============== ==================================== Legend: * "Atomic File": first version of the operating system supporting creating atomically a non-inheritable file descriptor using ``open()`` * "Atomic Socket": first version of the operating system supporting creating atomically a non-inheritable socket * "X": not supported yet Status of Python 3.3 -------------------- Python 3.3 creates inheritable file descriptors on all platforms, except ``os.pipe()`` which creates non-inheritable file descriptors on Windows. New constants and functions related to the atomic creation of non-inheritable file descriptors were added to Python 3.3: ``os.O_CLOEXEC``, ``os.pipe2()`` and ``socket.SOCK_CLOEXEC``. On UNIX, the ``subprocess`` module closes all file descriptors in the child process by default, except standard streams (0, 1, 2) and file descriptors of the *pass_fds* parameter. If the *close_fds* parameter is set to ``False``, all inheritable file descriptors are inherited in the child process. On Windows, the ``subprocess`` closes all handles and file descriptors in the child process by default. If at least one standard stream (stdin, stdout or stderr) is replaced (ex: redirected into a pipe), all inheritable handles are inherited in the child process. All inheritable file descriptors are inherited by the child process using the functions of the ``os.execv*()`` and ``os.spawn*()`` families. On UNIX, the ``multiprocessing`` module uses ``os.fork()`` and so all file descriptors are inherited by child processes. On Windows, all inheritable handles are inherited by the child process using the ``multiprocessing`` module, all file descriptors except standard streams are closed. Summary: =========================== ============= ================== ============= Module FD on UNIX Handles on Windows FD on Windows =========================== ============= ================== ============= subprocess, default STD, pass_fds none STD subprocess, close_fds=False all all STD os.execv(), os.spawn() all all all multiprocessing all all STD =========================== ============= ================== ============= Legend: * "all": all *inheritable* file descriptors or handles are inherited in the child process * "none": all handles are closed in the child process * "STD": only file descriptors 0 (stdin), 1 (stdout) and 2 (stderr) are inherited in the child process * "pass_fds": file descriptors of the *pass_fds* parameter of the subprocess are inherited Proposal ======== Non-inheritable File Descriptors -------------------------------- The following functions are modified to make newly created file descriptors non-inheritable by default: * ``asyncore.dispatcher.create_socket()`` * ``io.FileIO`` * ``io.open()`` * ``open()`` * ``os.dup()`` * ``os.dup2()`` * ``os.fdopen()`` * ``os.open()`` * ``os.openpty()`` * ``os.pipe()`` * ``select.devpoll()`` * ``select.epoll()`` * ``select.kqueue()`` * ``socket.socket()`` * ``socket.socket.accept()`` * ``socket.socket.dup()`` * ``socket.socket.fromfd()`` * ``socket.socketpair()`` When available, atomic flags are used to make file descriptors non-inheritable. The atomicity is not guaranteed because a fallback is required when atomic flags are not available. New Functions ------------- * ``os.get_inheritable(fd: int)``: return ``True`` if the file descriptor can be inherited by child processes, ``False`` otherwise. * ``os.set_inheritable(fd: int, inheritable: bool)``: clear or set the inheritable flag of the specified file descriptor. These new functions are available on all platforms. On Windows, these functions accept also file descriptors of sockets: the result of ``sockobj.fileno()``. Other Changes ------------- * On UNIX, subprocess makes file descriptors of the *pass_fds* parameter inheritable. The file descriptor is made inheritable in the child process after the ``fork()`` and before ``execv()``, so the inheritable flag of file descriptors is unchanged in the parent process. * ``os.dup2(fd, fd2)`` makes *fd2* inheritable if *fd2* is ``0`` (stdin), ``1`` (stdout) or ``2`` (stderr) and *fd2* is different than *fd*. Backward Compatibility ====================== This PEP break applications relying on inheritance of file descriptors. Developers are encouraged to reuse the high-level Python module ``subprocess`` which handles the inheritance of file descriptors in a portable way. Applications using the ``subprocess`` module with the *pass_fds* parameter or using ``os.dup2()`` to redirect standard streams should not be affected. Python does no more conform to POSIX, since file descriptors are now made non-inheritable by default. Python was not designed to conform to POSIX, but was designed to develop portable applications. Related Work ============ The programming languages Go, Perl and Ruby make newly created file descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0 (1987) and Ruby 2.0 (2013). The SCons project overrides builtin functions ``file()`` and ``open()`` to make files non-inheritable on Windows: see `win32.py <https://bitbucket.org/scons/scons/src/c8dbbaa4598e7119ae80f72068386be105b5a…>`_. Rejected Alternatives ===================== PEP 433 ------- The PEP 433 entitled "Easier suppression of file descriptor inheritance" is a previous attempt proposing various other alternatives, but no consensus could be reached. No special case for standard streams ------------------------------------ Functions handling file descriptors should not handle standard streams (file descriptors ``0``, ``1``, ``2``) differently. This option does not work on Windows. On Windows, calling ``SetHandleInformation()`` to set or clear ``HANDLE_FLAG_INHERIT`` flag on standard streams (0, 1, 2) fails with the Windows error 87 (invalid argument). If ``os.dup2(fd, fd2)`` would always make *fd2* non-inheritable, the function would raise an exception when used to redirect standard streams. Another option is to add a new *inheritable* parameter to ``os.dup2()``. This PEP has a special-case for ``os.dup2()`` to not break backward compatibility on applications redirecting standard streams before calling the C function ``execv()``. Examples in the Python standard library: ``CGIHTTPRequestHandler.run_cgi()`` and ``pty.fork()`` use ``os.dup2()`` to redict stdin, stdout and stderr. Links ===== Python issues: * `#10115: Support accept4() for atomic setting of flags at socket creation <http://bugs.python.org/issue10115>`_ * `#12105: open() does not able to set flags, such as O_CLOEXEC <http://bugs.python.org/issue12105>`_ * `#12107: TCP listening sockets created without FD_CLOEXEC flag <http://bugs.python.org/issue12107>`_ * `#16850: Add "e" mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_ * `#16860: Use O_CLOEXEC in the tempfile module <http://bugs.python.org/issue16860>`_ * `#16946: subprocess: _close_open_fd_range_safe() does not set close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined <http://bugs.python.org/issue16946>`_ * `#17070: Use the new cloexec to improve security and avoid bugs <http://bugs.python.org/issue17070>`_ * `#18571: Implementation of the PEP 446: non-inheriable file descriptors <http://bugs.python.org/issue18571>`_ Other links: * `Secure File Descriptor Handling <http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper, 2008) * `Ghosts of Unix past, part 2: Conflated designs <http://lwn.net/Articles/412131/>`_ (Neil Brown, 2010) explains the history of ``O_CLOEXEC`` and ``O_NONBLOCK`` flags Copyright ========= This document has been placed into the public domain. [View Less]

2 7

Green buildbot failure.
by Terry Reedy Aug. 11, 2013

Aug. 11, 2013

This run recorded here shows a green test (it appears to have timed out) http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/7017 but the corresponding log for this Windows bot http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/7017/st… has the expected os.chown failure. Are such green failures intended? -- Terry Jan Reedy

5 6

Reaping threads and subprocesses
by Serhiy Storchaka Aug. 11, 2013

Aug. 11, 2013

Some tests uses the following idiom: def test_main(): try: test.support.run_unittest(...) finally: test.support.reap_children() Other tests uses the following idiom: def test_main(): key = test.support.threading_setup() try: test.support.run_unittest(...) finally: test.support.threading_cleanup(*key) or in other words: @test.support.reap_threads def test_main(): test.support.run_unittest(...) These tests are not … [View More]

1 0

Buildbot failure puzzle
by Terry Reedy Aug. 11, 2013

Aug. 11, 2013

At least the following 3.4 buildbots have failed today with an error I do not understand: AMD64 FreeBSD, PPC64, x86Ubuntu, x86 WinServer 2003. Except for the Windows BB, it was the only failure and hence the only reason to not be green. ERROR: test_xmlcharnamereplace (test.test_codeccallbacks.CodecCallbackTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/… [View More]

2 1

Summary of Python tracker Issues
by Python tracker Aug. 9, 2013

Aug. 9, 2013

ACTIVITY SUMMARY (2013-08-02 - 2013-08-09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4148 (+20) closed 26321 (+47) total 30469 (+67) Open issues with patches: 1874 Issues opened (56) ================== #4322: function with modified __name__ uses original name when there' http://bugs.python.org/issue4322 reopened by benjamin.peterson #17741: … [View More]

1 0

xml.etree.ElementTree.IncrementalParser (was: ElementTree iterparse string)
by Stefan Behnel Aug. 9, 2013

Aug. 9, 2013

[from python-ideas] Antoine Pitrou, 07.08.2013 08:04: > Take a look at IncrementalParser: > http://docs.python.org/dev/library/xml.etree.elementtree.html#incremental-p… Hmm, that seems to be a somewhat recent addition (April 2013). I would have preferred hearing about it before it got added. I don't like the fact that it adds a second interface to iterparse() that allows injecting arbitrary content into the parser. You can now run iterparse() to read from a file, and at an arbitrary … [View More]

2 4

The Return Of Argument Clinic
by Larry Hastings Aug. 8, 2013

Aug. 8, 2013

It's time to discuss Argument Clinic again. I think the implementation is ready for public scrutiny. (It was actually ready a week ago, but I lost a couple of days to "make distclean" corrupting my hg data store--yes, I hadn't upped my local clinic branch in a while. Eventually I gave up on repairing it and just brute-forcd it. Anyway...) My Clinic test branch is here: https://bitbucket.org/larry/python-clinic/ And before you ask, no, the above branch should never ever ever be merged … [View More]back into trunk. We'll start clean once Clinic is ready for merging and do a nice neat job. ___________________________________________________________________ There's no documentation, apart from the PEP. But you can see plenty of test cases of using Clinic, just grep for the string "clinic" in */*.c. But for reference here's the list: Modules/_cursesmodule.c Modules/_datetimemodule.c Modules/_dbmmodule.c Modules/posixmodule.c Modules/unicodedata.c Modules/_weakref.c Modules/zlibmodule.c Objects/dictobject.c Objects/unicodeobject.c I haven't reimplemented every PyArg_ParseTuple "format unit" in the retooled Clinic, so it's not ready to try with every single builtin yet. The syntax is as Guido dictated it during our meeting after the Language Summit at PyCon US 2013. The implementation has been retooled, several times, and is now both nicer and more easily extensible. The internals are just a little messy, but the external interfaces are all ready for critique. ___________________________________________________________________ Here are the external interfaces as I forsee them. If you add your own data types, you'll subclass "Converter" and maybe "ReturnConverter". Take a look at the existing subclasses to get a feel for what that's like. If you implemented your own DSL, you'd make something that quacked like "PythonParser" (implementing __init__ and parse methods), and you'd deal with "Block", "Module", "Class", "Function", and "Parameter" objects a lot. What do you think? ___________________________________________________________________ What follows are six questions I'd like to put to the community, ranked oddly enough in order of how little to how much I care about the answer. BTW, by convention, every time I need a arbitrary sample function I use "os.stat". (Please quote the question line in your responses, otherwise I fear we'll get lost in the sea of text.) ___________________________________________________________________ Question 0: How should we integrate Clinic into the build process? Clinic presents a catch-22: you want it as part of the build process, but it needs Python to be built before it'll run. Currently it requires Python 3.3 or newer; it might work in 3.2, I've never tried it. We can't depend on Python 3 being available when we build. This complicates the build process somewhat. I imagine it's a solvable problem on UNIX... with the right wizardry. I have no idea how one'd approach it on Windows, but obviously we need to solve the problem there too. ___________________________________________________________________ Question 1: Which C function nomenclature? Argument Clinic generates two functions prototypes per Python function: one specifying one of the traditional signatures for builtins, whose code is generated completely by Clinic, and the other with a custom-generated signature for just that call whose code is written by the user. Currently the former doesn't have any specific name, though I have been thinking of it as the "parse" function. The latter is definitely called the "impl" (pronounced IM-pull), short for "implementation". When Clinic generates the C code, it uses the name of the Python function to create the C functions' names, with underscores in place of dots. Currently the "parse" function gets the base name ("os_stat"), and the "impl" function gets an "_impl" added to the end ("os_stat_impl"). Argument Clinic is agnostic about the names of these functions. It's possible it'd be nicer to name these the other way around, say "os_stat_parse" for the parse function and "os_stat" for the impl. Anyone have a strong opinion one way or the other? I don't much care; all I can say is that the "obvious" way to do it when I started was to add "_impl" to the impl, as it is the new creature under the sun. ___________________________________________________________________ Question 2: Emit code for modules and classes? Argument Clinic now understands the structure of the modules and classes it works with. You declare them like so: module os class os.ImaginaryClassHere def os.ImaginaryClassHere.stat(...): ... Currently it does very little with the information; right now it mainly just gets baked into the documentation. In the future I expect it to get used in the introspection metadata, and it'll definitely be relevant to external consumers of the Argument Clinic information (IDEs building per-release databases, other implementations building metadata for library interface conformance testing). Another way we could use this metadata: have Argument Clinic generate more of the boilerplate for a class or module. For example, it could kick out all the PyMethodDef structures for the class or module. If we grew Argument Clinic some, and taught it about the data members of classes and modules, it could also generate the PyModuleDef and PyTypeObject structures, and even generate a function that initialized them at runtime for you. (Though that does seem like mission creep to me.) There are some complications to this, one of which I'll discuss next. But I put it to you, gentle reader: how much boilerplate should Argument Clinic undertake to generate, and how much more class and module metadata should be wired in to it? ___________________________________________________________________ Question 3: #ifdef support for functions? Truth be told, I did experiment with having Argument Clinic generate more of the boilerplate associated with modules. Clinic already generates a macro per function defining that function's PyMethodDef structure, for example: #define OS_STAT_METHODDEF \ {"stat", (PyCFunction)os_stat, \ METH_VARARGS|METH_KEYWORDS, os_stat__doc__} For a while I had it generating the PyMethodDef structures, like so: /*[clinic] generate_method_defs os [clinic]*/ #define OS_METHODDEFS \ OS_STAT_METHODDEF, \ OS_ACCESS_METHODDEF, \ OS_TTYNAME_METHODDEF, \ static PyMethodDef os_methods[] = { OS_METHODDEFS /* existing methoddefs here... */ NULL } But I ran into trouble with os.ttyname(), which is only created and exposed if the platform defines HAVE_TTYNAME. Initially I'd just thrown all the Clinic stuff relevant to os.ttyname in the #ifdef block. But Clinic pays no attention to #ifdef statements--so it would still add OS_TTYNAME_METHODDEF, to OS_METHODDEFS. And kablooey! Right now I've backed out of this--I had enough to do without getting off into extra credit like this. But I'd like to return to it. It just seems natural to have Clinic generate this nasty boilerplate. Four approaches suggest themselves to me, listed below in order of least- to most-preferable in my opinion: 0) Don't have Clinic participate in populating the PyMethodDefs. 1) Teach Clinic to understand simple C preprocessor statements, just enough so it implicitly understands that os.ttyname was defined inside an #ifdef HAVE_TTYPE block. It would then intelligently generate the code to take this into account. 2) Explicitly tell Clinic that os.ttyname must have HAVE_TTYNAME defined in order to be active. Clinic then generates the code intelligently taking this into account, handwave handwave. 3) Change the per-function methoddef macro to have the trailing comma: #define OS_STAT_METHODDEF \ {"stat", (PyCFunction)os_stat, \ METH_VARARGS|METH_KEYWORDS, os_stat__doc__}, and suppress it in the macro Clinic generates: /*[clinic] generate_method_defs os [clinic]*/ #define OS_METHODDEFS \ OS_STAT_METHODDEF \ OS_ACCESS_METHODDEF \ OS_TTYNAME_METHODDEF \ And then the code surrounding os.ttyname can look like this: #ifdef HAVE_TTYNAME // ... real os.ttyname stuff here #else #define OS_STAT_TTYNAME #endif And I think that would work great, actually. But I haven't tried it. Do you agree that Argument Clinic should generate this information, and it should use the approach in 3) ? ___________________________________________________________________ Question 4: Return converters returning success/failure? With the addition of the "return converter", we have the lovely feature of being able to *return* a C type and have it converted back into a Python type. Your C extensions have never been more readable! The problem is that the PyObject * returned by a C builtin function serves two simultaneous purposes: it contains the return value on success, but also it is NULL if the function threw an exception. We can probably still do that for all pointer-y return types (I'm not sure, I haven't played with it yet). But if the impl function now returns "int", or some decidedly other non-pointer-y type, there's no longer a magic return value we can use to indicate "we threw an exception". This isn't the end of the world; I can detect that the impl threw an exception by calling PyErr_Occurred(). But I've been chided before for calling this unnecessarily; it's ever-so slightly expensive, in that it has to dereference TLS, and does so with an atomic operation. Not to mention that it's a function call! The impl should know whether or not it failed. So it's the interface we're defining that forces it to throw away that information. If we provided a way for it to return that information, we could shave off some cycles. The problem is, how do we do that in a way that doesn't suck? Four approaches suggest themselves to me, and sadly I think they all suck to one degree or another. In order of sucking least to most: 0) Return the real type and detect the exception with PyErr_Occurred(). This is by far the loveliest option, but it incurs runtime overhead. 1) Have the impl take an extra parameter, "int *failed". If the function fails, it sets that to a true value and returns whatever. 2) Have the impl return its calculated return value through an extra pointer-y parameter ("int *return_value"), and its actual return value is an int indicating success or failure. 3) Have the impl return a structure containing both the real return value and a success/failure integer. Then its return lines would look like this: return {-1, 0}; or maybe return {-3, PY_HORRIBLE_CLINIC_INTERFACE__SUCCESS}; Can we live with PyErr_Occurred() here? ___________________________________________________________________ Question 5: Keep too-magical class decorator Converter.wrap? Converter is the base class for converter objects, the objects that handle the details of converting a Python object into its C equivalent. The signature for Converter.__init__ has become complicated: def __init__(self, name, function, default=unspecified, *, doc_default=None, required=False) "name" is the name of the function ("stat"), "function" is an object representing the function for which this Converter is handling an argument (duck-type compatible with inspect.Signature), and default is the default (Python) value if any. "doc_default" is a string that overrides repr(default) in the documentation, handy if repr(default) is too ugly or you just want to mislead the user. "required", if True specifies that the parameter should be considered required, even if it has a default value. Complicating the matter further, converter subclasses may take extra (keyword-only and optional) parameters to configure exotic custom behavior. For example, the "Py_buffer" converter takes "zeroes" and "nullable"; the "path_t" converter implemented in posixmodule.c takes "allow_fd" and "nullable". This means that converter subclasses have to define a laborious __init__, including three parameters with defaults, then turn right around and pass most of the parameters back into super().__init__. This interface has changed several times during the development of Clinic, and I got tired of fixing up all my existing prototypes and super calls. So I made a class decorator that did it for me. Shield your eyes from the sulferous dark wizardry of Converter.wrap: @staticmethod def wrap(cls): class WrappedConverter(cls, Converter): def __init__(self, name, function, default=unspecified, *, doc_default=None, required=False, **kwargs): super(cls, self).__init__(name, function, default, doc_default=doc_default, required=required) cls.__init__(self, **kwargs) return functools.update_wrapper(WrappedConverter, cls, updated=()) When you decorate your class with Converter.wrap, you only define in your __init__ your custom arguments. All the arguments Converter.__init__ cares about are taken care of for you (aka hidden from you). As an example, here's the relevant bits of path_t_converter from posixmodule.c: @Converter.wrap class path_t_converter(Converter): def __init__(self, *, allow_fd=False, nullable=False): ... So on the one hand I admit it's smelly. On the other hand it hides a lot of stuff that the user needn't care about, and it makes the code simpler and easier to read. And it means we can change the required arguments for Converter.__init__ without breaking any code (as I have already happily done once or twice). I'd like to keep it in, and anoint it as the preferred way of declaring Converter subclasses. Anybody else have a strong opinion on this either way? (I don't currently have an equivalent mechanism for return converters--their interface is a lot simpler, and I just haven't needed it so far.) ___________________________________________________________________ Well! That's quite enough for now. //arry/ [View Less]

7 10

PEP 442 clarification for type hierarchies
by Stefan Behnel Aug. 8, 2013

Aug. 8, 2013

Hi, I'm currently catching up on PEP 442, which managed to fly completely below my radar so far. It's a really helpful change that could end up fixing a major usability problem that Cython was suffering from: user provided deallocation code now has a safe execution environment (well, at least in Py3.4+). That makes Cython a prime candidate for testing this, and I've just started to migrate the implementation. One thing that I found to be missing from the PEP is inheritance handling. The … [View More]

2 16