Mailman 3 January 2013 - Python-Dev

Proposing "Argument Clinic", a new way of specifying arguments to builtins for CPython
by Larry Hastings Feb. 16, 2013

Feb. 16, 2013

Say there, the Python core development community! Have I got a question for you! *ahem* Which of the following four options do you dislike least? ;-) 1) CPython continues to provide no "function signature" objects (PEP 362) or inspect.getfullargspec() information for any function implemented in C. 2) We add new hand-coded data structures representing the metadata necessary for function signatures for builtins. Which means that, when defining arguments to functions in C, … [View More] we'd need to repeat ourselves *even more* than we already do. 3) Builtin function arguments are defined using some seriously uncomfortable and impenetrable C preprocessor macros, which produce all the various types of output we need (argument processing code, function signature metadata, possibly the docstrings too). 4) Builtin function arguments are defined in a small DSL; these are expanded to code and data using a custom compile-time preprocessor step. All the core devs I've asked said "given all that, I'd prefer the hairy preprocessor macros". But by the end of the conversation they'd changed their minds to prefer the custom DSL. Maybe I'll make a believer out of you too--read on! I've named this DSL preprocessor "Argument Clinic", or Clinic for short**. Clinic works similarly to Ned Batchelder's brilliant "Cog" tool: http://nedbatchelder.com/code/cog/ You embed the input to Clinic in a comment in your C file, and the output is written out immediately after that comment. The output's overwritten every time the preprocessor is run. In short it looks something like this: /*[clinic] input to the DSL [clinic]*/ ... output from the DSL, overwritten every time ... /*[clinic end:<checksum>]*/ The input to the DSL includes all the metadata about the function that we need for the function signature: * the name of the function, * the return annotation (if any), * each parameter to the function, including * its name, * its type (in C), * its default value, * and a per-parameter docstring; * and the docstring for the function as a whole. The resulting output contains: * the docstring for the function, * declarations for all your parameters, * C code handling all argument processing for you, * and a #define'd methoddef structure for adding the function to the module. I discussed this with Mark "HotPy" Shannon, and he suggested we break our existing C functions into two. We put the argument processing into its own function, generated entirely by Clinic, and have the implementation in a second function called from the first. I like this approach simply because it makes the code cleaner. (Note that this approach should not cause any overhead with a modern compiler, as both functions will be "static".) But it also provides an optimization opportunity for HotPy: it could read the metadata, and when generating the JIT'd code it could skip building the PyObjects and argument tuple (and possibly keyword argument dict), and the subsequent unpacking/decoding, and just call the implementation function directly, giving it a likely-measurable speed boost. And we can go further! If we add a new extension type API allowing you to register both functions, and external modules start using it, sophisticated Python implementations like PyPy might be able to skip building the tuple for extension type function calls--speeding those up too! Another plausible benefit: alternate implementations of Python could read the metadata--or parse the input to Clinic themselves--to ensure their reimplementations of the Python standard library conform to the same API! Clinic can also run general-purpose Python code ("/*[python]"). All output from "print" is redirected into the output section after the Python code. As you've no doubt already guessed, I've made a prototype of Argument Clinic. You can see it--and some sample conversions of builtins using it for argument processing--at this BitBucket repo: https://bitbucket.org/larry/python-clinic I don't claim that it's fabulous, production-ready code. But it's a definite start! To save you a little time, here's a preview of using Clinic for dbm.open(). The stuff at the same indent as a declaration are options; see the "clinic.txt" in the repo above for full documentation. /*[clinic] dbm.open -> mapping basename=dbmopen const char *filename; The filename to open. const char *flags="r"; How to open the file. "r" for reading, "w" for writing, etc. int mode=0666; default=0o666 If creating a new file, the mode bits for the new file (e.g. os.O_RDWR). Returns a database object. [clinic]*/ PyDoc_STRVAR(dbmopen__doc__, "dbm.open(filename[, flags=\'r\'[, mode=0o666]]) -> mapping\n" "\n" " filename\n" " The filename to open.\n" "\n" " flags\n" " How to open the file. \"r\" for reading, \"w\" for writing, etc.\n" "\n" " mode\n" " If creating a new file, the mode bits for the new file\n" " (e.g. os.O_RDWR).\n" "\n" "Returns a database object.\n" "\n"); #define DBMOPEN_METHODDEF \ {"open", (PyCFunction)dbmopen, METH_VARARGS | METH_KEYWORDS, dbmopen__doc__} static PyObject * dbmopen_impl(PyObject *self, const char *filename, const char *flags, int mode); static PyObject * dbmopen(PyObject *self, PyObject *args, PyObject *kwargs) { const char *filename; const char *flags = "r"; int mode = 0666; static char *_keywords[] = {"filename", "flags", "mode", NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s|si", _keywords, &filename, &flags, &mode)) return NULL; return dbmopen_impl(self, filename, flags, mode); } static PyObject * dbmopen_impl(PyObject *self, const char *filename, const char *flags, int mode) /*[clinic end:eddc886e542945d959b44b483258bf038acf8872]*/ As of this writing, I also have sample conversions in the following files available for your perusal: Modules/_cursesmodule.c Modules/_dbmmodule.c Modules/posixmodule.c Modules/zlibmodule.c Just search in C files for '[clinic]' and you'll find everything soon enough. As you can see, Clinic has already survived some contact with the enemy. I've already converted some tricky functions--for example, os.stat() and curses.window.addch(). The latter required adding a new positional-only processing mode for functions using a legacy argument processing approach. (See "clinic.txt" for more.) If you can suggest additional tricky functions to support, please do! Big unresolved questions: * How would we convert all the builtins to use Clinic? I fear any solution will involve some work by hand. Even if we can automate big chunks of it, fully automating it would require parsing arbitrary C. This seems like overkill for a one-shot conversion. (Mark Shannon says he has some ideas.) * How do we create the Signature objects? My current favorite idea: Clinic also generates a new, TBD C structure defining all the information necessary for the signature, which is also passed in to the new registration API (you remember, the one that takes both the argument-processing function and the implementation function). This is secreted away in some new part of the C function object. At runtime this is converted on-demand into a Signature object. Default values for arguments are represented in C as strings; the conversion process attempts eval() on the string, and if that works it uses the result, otherwise it simply passes through the string. * Right now Clinic paves over the PyArg_ParseTuple API for you. If we convert CPython to use Clinic everywhere, theoretically we could replace the parsing API with something cleaner and/or faster. Does anyone have good ideas (and time, and energy) here? * There's actually a fifth option, proposed by Brett Cannon. We constrain the format of docstrings for builtin functions to make them machine-readable, then generate the function signature objects from that. But consider: generating *everything* in the signature object may get a bit tricky (e.g. Parameter.POSITIONAL_ONLY), and this might gunk up the docstring. But the biggest unresolved question... is this all actually a terrible idea? //arry/ ** "Is this the right room for an argument?" "I've told you once...!" [View Less]

17 33

Re: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build
by Antoine Pitrou Feb. 7, 2013

Feb. 7, 2013

On Thu, 31 Jan 2013 23:52:27 +0100 (CET) matthias.klose <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/8ee6d96a1019 > changeset: 81859:8ee6d96a1019 > branch: 2.7 > parent: 81855:df9f8feb7444 > user: doko(a)python.org > date: Thu Jan 31 23:52:03 2013 +0100 > summary: > - Issue #17086: Backport the patches from the 3.3 branch to cross-build > the package. You aren't supposed to add new features to bugfix … [View More]

7 13

2.7.4
by Benjamin Peterson Feb. 3, 2013

Feb. 3, 2013

It's been almost a year since 2.7.3, so it's time for another 2.7 bugfix release. 2013-02-02 - 2.7.4 release branch created; rc released 2013-02-16 - 2.7.4 released Does this work for you, Martin and Ned? -- Regards, Benjamin

8 8

importlib.find_loader
by Thomas Heller Feb. 1, 2013

Feb. 1, 2013

In Python3.3, I thought that every loaded module has a __loader__ attribute. Apparently this is not the case for a few builtin modules: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import importlib >>> >>> def bug(): ... for name in list(sys.modules.keys()): ... try: ... importlib.find_loader(name) ... except … [View More]

3 3

FreeBSD-9.0 bot running --without-doc-strings
by Stefan Krah Jan. 31, 2013

Jan. 31, 2013

Hi, I've subverted the build master authority on the FreeBSD-9.0 bot by exporting with_doc_strings=no. This is to test #16143 and #10156. Some tests assume that docstrings are present, so there will be a couple of failures. If someone wants to work on easy issues, this is your chance. If the failures get annoying just open a tracker issue, and I'll revert the change. Stefan Krah

8 12

Fwd: I was just thinking that os.path could use some love...
by Guido van Rossum Jan. 31, 2013

Jan. 31, 2013

Thoughts on os.path? What happened to the idea of a new path object? --Guido ---------- Forwarded message ---------- From: Talin <viridia(a)gmail.com> Date: Wed, Jan 30, 2013 at 12:34 PM Subject: Re: I was just thinking that os.path could use some love... To: Guido van Rossum <guido(a)python.org> On Wed, Jan 30, 2013 at 11:33 AM, Guido van Rossum <guido(a)python.org> wrote: > > Hmm... Mind if I just forward to python-dev? IIRC there's been some > discussion about … [View More]a newfangled path object, but I lost track of where > the discussion went. (Or you can post there yourself. :-) Sure - forward away. Is this the same as the path object idea that was floated 5 years ago? I've yet to see any set of convenience methods for paths that are so compelling as to be worth all of the time and energy needed to update all of the various APIs which now expect paths to be passed in as strings. > > > On Wed, Jan 30, 2013 at 11:13 AM, Talin <viridia(a)gmail.com> wrote: > > I just realized that os.path hasn't changed in a long time. Here's a couple > > of ideas for additions: > > > > os.path.splitall(path) - splits all path components into a tuple - so for > > example, 'this/is/a/path' turns into ('this', 'is', 'a', 'path'). If there's > > a trailing slash, the last item in the tuple will be a zero-length string. > > The main reason for having this in os.path is so that we can remain > > separator-character-agnostic. > > Would it also return a leading empty string if the path starts with /? > What about Windows C: or //host/ prefixes??? > I would say that it should only split on directory separators, not any other kind of delimiter, so that each component is a valid filesystem identifier. Further, the operation should be reversible by calling os.path.join(*dirnames). So it's a little more complex than just string.split('/'). Part of the reason for wanting the splitall function is to implement the common prefix function - you take all the paths, bust them into tuples, look for the longest tuple prefix, and then join the result back into a path. This means that os.path.join(*os.path.splitall(path)) must reproduce the original path exactly. > > > An alternative would be to add an optional 'maxsplits' parameter to > > os.path.split. Default would be 1; 0 = unlimited. Main disadvantage would be > > that no one would ever use values 2, 3, etc... but it would be more > > compatible with other split apis. > > > > A version of os.path.commonprefix that always produces valid paths (and > > works by path component, not character-by-character). > > > > I can probably think of others if I look at some other path manipulation > > apis. llvm::sys::path has a bunch if I recall... > > > > -- > > -- Talin > > > > -- > --Guido van Rossum (python.org/~guido) -- -- Talin -- --Guido van Rossum (python.org/~guido) [View Less]

7 6

PEP 433: second try
by Victor Stinner Jan. 30, 2013

Jan. 30, 2013

Hi, Here is a new version of my PEP 433. The default value of the cloexec parameter is now configurable (cmdline option, env var, sys.setdefaultcloexec), it can be disabled using sys.setdefaultcloexec(False). HTML version: http://python.org/dev/peps/pep-0433/ Implementation: http://bugs.python.org/issue17036 My last question is on the implementation: should subprocess.Popen clears the close-on-exec flag on file descriptors passed to pass_fds? It makes sense if cloexec is True by default. --… [View More]

3 4

PEP 433: Choose the default value of the new cloexec parameter
by Victor Stinner Jan. 29, 2013

Jan. 29, 2013

Hi, The PEP 433 proposes different options to add a new cloexec parameter: a) cloexec=False by default b) cloexec=True by default c) configurable default value I tried to list in the PEP 433 advantages and drawbacks of each option. If I recorded correctly opinions, the different options have the following supporters: a) cloexec=False by default b) cloexec=True by default: Charles-François Natali c) configurable default value: Antoine Pitrou, Nick Coghlan, Guido van Rossum I don't … [View More]

10 36

PEP 431 Updates
by Lennart Regebro Jan. 28, 2013

Jan. 28, 2013

I sent this out the 15th, but it seems to have gotten lost since I have reports that it didn't show up and I got no feedback. So here goes again, a new version of PEP 431. http://www.python.org/dev/peps/pep-0431/ //Lennart PEP: 431 Title: Time zone support improvements Version: $Revision$ Last-Modified: $Date$ Author: Lennart Regebro <regebro(a)gmail.com> BDFL-Delegate: Barry Warsaw <barry(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Dec-… [View More]2012 Post-History: 11-Dec-2012, 28-Dec-2012 Abstract ======== This PEP proposes the implementation of concrete time zone support in the Python standard library, and also improvements to the time zone API to deal with ambiguous time specifications during DST changes. Proposal ======== Concrete time zone support -------------------------- The time zone support in Python has no concrete implementation in the standard library outside of a tzinfo baseclass that supports fixed offsets. To properly support time zones you need to include a database over all time zones, both current and historical, including daylight saving changes. But such information changes frequently, so even if we include the last information in a Python release, that information would be outdated just a few months later. Time zone support has therefore only been available through two third-party modules, ``pytz`` and ``dateutil``, both who include and wrap the "zoneinfo" database. This database, also called "tz" or "The Olsen database", is the de-facto standard time zone database over time zones, and it is included in most Unix and Unix-like operating systems, including OS X. This gives us the opportunity to include the code that supports the zoneinfo data in the standard library, but by default use the operating system's copy of the data, which typically will be kept updated by the updating mechanism of the operating system or distribution. For those who have an operating system that does not include the zoneinfo database, for example Windows, the Python source distribution will include a copy of the zoneinfo database, and a distribution containing the latest zoneinfo database will also be available at the Python Package Index, so it can be easily installed with the Python packaging tools such as ``easy_install`` or ``pip``. This could also be done on Unices that are no longer receiving updates and therefore have an outdated database. With such a mechanism Python would have full time zone support in the standard library on any platform, and a simple package installation would provide an updated time zone database on those platforms where the zoneinfo database isn't included, such as Windows, or on platforms where OS updates are no longer provided. The time zone support will be implemented by making the ``datetime`` module into a package, and adding time zone support to ``datetime`` based on Stuart Bishop's ``pytz`` module. Getting the local time zone --------------------------- On Unix there is no standard way of finding the name of the time zone that is being used. All the information that is available is the time zone abbreviations, such as ``EST`` and ``PDT``, but many of those abbreviations are ambiguous and therefore you can't rely on them to figure out which time zone you are located in. There is however a standard for finding the compiled time zone information since it's located in ``/etc/localtime``. Therefore it is possible to create a local time zone object with the correct time zone information even though you don't know the name of the time zone. A function in ``datetime`` should be provided to return the local time zone. The support for this will be made by integrating Lennart Regebro's ``tzlocal`` module into the new ``datetime`` module. For Windows it will look up the local Windows time zone name, and use a mapping between Windows time zone names and zoneinfo time zone names provided by the Unicode consortium to convert that to a zoneinfo time zone. The mapping should be updated before each major or bugfix release, scripts for doing so will be provided in the ``Tools/`` directory. Ambiguous times --------------- When changing over from daylight savings time (DST) the clock is turned back one hour. This means that the times during that hour happens twice, once without DST and then once with DST. Similarly, when changing to daylight savings time, one hour goes missing. The current time zone API can not differentiate between the two ambiguous times during a change from DST. For example, in Stockholm the time of 2012-11-28 02:00:00 happens twice, both at UTC 2012-11-28 00:00:00 and also at 2012-11-28 01:00:00. The current time zone API can not disambiguate this and therefore it's unclear which time should be returned:: # This could be either 00:00 or 01:00 UTC: >>> dt = datetime(2012, 10, 28, 2, 0, tzinfo=zoneinfo('Europe/Stockholm')) # But we can not specify which: >>> dt.astimezone(zoneinfo('UTC')) datetime.datetime(2012, 10, 28, 1, 0, tzinfo=<UTC>) ``pytz`` solved this problem by adding ``is_dst`` parameters to several methods of the tzinfo objects to make it possible to disambiguate times when this is desired. This PEP proposes to add these ``is_dst`` parameters to the relevant methods of the ``datetime`` API, and therefore add this functionality directly to ``datetime``. This is likely the hardest part of this PEP as this involves updating the C version of the ``datetime`` library with this functionality, as this involved writing new code, and not just reorganizing existing external libraries. Implementation API ================== The zoneinfo database --------------------- The latest version of the zoneinfo database should exist in the ``Lib/tzdata`` directory of the Python source control system. This copy of the database should be updated before every Python feature and bug-fix release, but not for releases of Python versions that are in security-fix-only-mode. Scripts to update the database will be provided in ``Tools/``, and the release instructions will be updated to include this update. New configure options ``--enable-internal-timezone-database`` and ``--disable-internal-timezone-database`` will be implemented to enable and disable the installation of this database when installing from source. A source install will default to installing them. Binary installers for systems that have a system-provided zoneinfo database may skip installing the included database since it would never be used for these platforms. For other platforms, for example Windows, binary installers must install the included database. Changes in the ``datetime``-module ---------------------------------- The public API of the new time zone support contains one new class, one new function, one new exception and four new collections. In addition to this, several methods on the datetime object gets a new ``is_dst`` parameter. New class ``DstTzInfo`` ^^^^^^^^^^^^^^^^^^^^^^^^ This class provides a concrete implementation of the ``zoneinfo`` base class that implements DST support. New function ``zoneinfo(name=None, db_path=None)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This function takes a name string that must be a string specifying a valid zoneinfo time zone, i.e. "US/Eastern", "Europe/Warsaw" or "Etc/GMT". If not given, the local time zone will be looked up. If an invalid zone name is given, or the local time zone can not be retrieved, the function raises `UnknownTimeZoneError`. The function also takes an optional path to the location of the zoneinfo database which should be used. If not specified, the function will look for databases in the following order: 1. Use the database in ``/usr/share/zoneinfo``, if it exists. 2. Check if the `tzdata-update` module is installed, and then use that database. 3. Use the Python-provided database in ``Lib/tzdata``. If no database is found an ``UnknownTimeZoneError`` or subclass thereof will be raised with a message explaining that no zoneinfo database can be found, but that you can install one with the ``tzdata-update`` package. New parameter ``is_dst`` ^^^^^^^^^^^^^^^^^^^^^^^^ A new ``is_dst`` parameter is added to several methods to handle time ambiguity during DST changeovers. * ``tzinfo.utcoffset(dt, is_dst=False)`` * ``tzinfo.dst(dt, is_dst=False)`` * ``tzinfo.tzname(dt, is_dst=False)`` * ``datetime.astimezone(tz, is_dst=False)`` The ``is_dst`` parameter can be ``False`` (default), ``True``, or ``None``. ``False`` will specify that the given datetime should be interpreted as not happening during daylight savings time, i.e. that the time specified is after the change from DST. ``True`` will specify that the given datetime should be interpreted as happening during daylight savings time, i.e. that the time specified is before the change from DST. ``None`` will raise an ``AmbiguousTimeError`` exception if the time specified was during a DST change over. It will also raise a ``NonExistentTimeError`` if a time is specified during the "missing time" in a change to DST. New exceptions ^^^^^^^^^^^^^^ * ``UnknownTimeZoneError`` This exception is a subclass of KeyError and raised when giving a time zone specification that can't be found:: >>> datetime.Timezone('Europe/New_York') Traceback (most recent call last): ... UnknownTimeZoneError: There is no time zone called 'Europe/New_York' * ``InvalidTimeError`` This exception serves as a base for ``AmbiguousTimeError`` and ``NonExistentTimeError``, to enable you to trap these two separately. It will subclass from ValueError, so that you can catch these errors together with inputs like the 29th of February 2011. * ``AmbiguousTimeError`` This exception is raised when giving a datetime specification that is ambiguous while setting ``is_dst`` to None:: >>> datetime(2012, 11, 28, 2, 0, tzinfo=zoneinfo('Europe/Stockholm'), is_dst=None) >>> Traceback (most recent call last): ... AmbiguousTimeError: 2012-10-28 02:00:00 is ambiguous in time zone Europe/Stockholm * ``NonExistentTimeError`` This exception is raised when giving a datetime specification that is ambiguous while setting ``is_dst`` to None:: >>> datetime(2012, 3, 25, 2, 0, tzinfo=zoneinfo('Europe/Stockholm'), is_dst=None) >>> Traceback (most recent call last): ... NonExistentTimeError: 2012-03-25 02:00:00 does not exist in time zone Europe/Stockholm New collections ^^^^^^^^^^^^^^^ * ``all_timezones`` is the exhaustive list of the time zone names that can be used, listed alphabethically. * ``all_timezones_set`` is a set of the time zones in ``all_timezones``. * ``common_timezones`` is a list of useful, current time zones, listed alphabethically. * ``common_timezones_set`` is a set of the time zones in ``common_timezones``. The ``tzdata-update``-package ----------------------------- The zoneinfo database will be packaged for easy installation with ``easy_install``/``pip``/``buildout``. This package will not install any Python code, and will not contain any Python code except that which is needed for installation. It will be kept updated with the same tools as the internal database, but released whenever the ``zoneinfo``-database is updated, and use the same version schema. Differences from the ``pytz`` API ================================= * ``pytz`` has the functions ``localize()`` and ``normalize()`` to work around that ``tzinfo`` doesn't have is_dst. When ``is_dst`` is implemented directly in ``datetime.tzinfo`` they are no longer needed. * The ``timezone()`` function is called ``zoneinfo()`` to avoid clashing with the ``timezone`` class introduced in Python 3.2. * ``zoneinfo()`` will return the local time zone if called without arguments. * The class ``pytz.StaticTzInfo`` is there to provide the ``is_dst`` support for static time zones. When ``is_dst`` support is included in ``datetime.tzinfo`` it is no longer needed. * ``InvalidTimeError`` subclasses from ``ValueError``. Resources ========= * http://pytz.sourceforge.net/ * http://pypi.python.org/pypi/tzlocal * http://pypi.python.org/pypi/python-dateutil * http://unicode.org/cldr/data/common/supplemental/windowsZones.xml Copyright ========= This document has been placed in the public domain. [View Less]

4 8

Boost Software License
by Serhiy Storchaka Jan. 28, 2013

Jan. 28, 2013

Is Boost Software License [1] compatible with Python license? Can I steal some code from Boost library [2]? [1] http://www.boost.org/LICENSE_1_0.txt [2] http://www.boost.org/

5 4