Mailman 3 February 2012 - Python-Dev

need help with frozen module/marshal/gc issue involving sub-interpreters for importlib bootstrapping
by Brett Cannon Feb. 7, 2012

Feb. 7, 2012

So my grand quest for bootstrapping importlib into CPython is damn close to coming to fruition; I have one nasty bug blocking my way and I can't figure out what could be causing it. I'm hoping someone here will either know the solution off the top of their head or will have the time to have a quick look to see if they can figure it out as my brain is mush at this point. First, the bug tracking all of this is http://bugs.python.org/issue2377 and the repo where I have been doing my work is ssh://… [View More]

4 6

[Python-ideas] matrix operations on dict :)
by Mark Janssen Feb. 7, 2012

Feb. 7, 2012

On Mon, Feb 6, 2012 at 6:12 PM, Steven D'Aprano <steve(a)pearwood.info> wrote: > On Mon, Feb 06, 2012 at 09:01:29PM +0100, julien tayon wrote: > > Hello, > > > > Proposing vector operations on dict, and acknowledging there was an > > homeomorphism from rooted n-ary trees to dict, was inducing the > > possibility of making matrix of dict / trees. > > This seems interesting to me, but I don't see that they are important > enough to be built-in to … [View More]

1 0

distutils 'depends' management
by Matteo Bertini Feb. 6, 2012

Feb. 6, 2012

Hello, I've noted that distutils manages depends in a way I cannot understand. Suppose I have a minimal setup.py: from distutils.core import setup, Extension setup( name='foo', version='1.0', ext_modules=[ Extension('foo', sources=['foo.c'], depends=['fop.conf'] # <---- note the typo foo->fop ), ] ) Now setup.py will rebuild all every time, this is because the policy of newer_group in build_extension is to consider 'newer' any missing file. http://bit.ly/… [View More]

2 2

Volunteer
by Blockheads Oi Oi Feb. 6, 2012

Feb. 6, 2012

You may remember me from a couple of years ago when I was trying to help out with Python. Unfortunately I trod on a few toes. I now know why. I have been diagnosed with Asperger Syndrome at 55 years old. I would like to give it another go. -- Cheers. Mark Lawrence.

4 3

PEP 408 -- Standard library __preview__ package
by Eli Bendersky Feb. 5, 2012

Feb. 5, 2012

Hello, Following an earlier discussion on python-ideas [1], we would like to propose the following PEP for review. Discussion is welcome. The PEP can also be viewed in HTML form at http://www.python.org/dev/peps/pep-0408/ [1] http://mail.python.org/pipermail/python-ideas/2012-January/013246.html Eli --------------------------- PEP: 408 Title: Standard library __preview__ package Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Eli … [View More]Bendersky <eliben(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2012-01-07 Python-Version: 3.3 Post-History: 2012-01-27 Abstract ======== The process of including a new module into the Python standard library is hindered by the API lock-in and promise of backward compatibility implied by a module being formally part of Python. This PEP proposes a transitional state for modules - inclusion in a special ``__preview__`` package for the duration of a minor release (roughly 18 months) prior to full acceptance into the standard library. On one hand, this state provides the module with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the module's eventual full inclusion into the standard library, or to the stability of its API, which may change for the next release. Proposal - the __preview__ package ================================== Whenever the Python core development team decides that a new module should be included into the standard library, but isn't entirely sure about whether the module's API is optimal, the module can be placed in a special package named ``__preview__`` for a single minor release. In the next minor release, the module may either be "graduated" into the standard library (and occupy its natural place within its namespace, leaving the ``__preview__`` package), or be rejected and removed entirely from the Python source tree. If the module ends up graduating into the standard library after spending a minor release in ``__preview__``, its API may be changed according to accumulated feedback. The core development team explicitly makes no guarantees about API stability and backward compatibility of modules in ``__preview__``. Entry into the ``__preview__`` package marks the start of a transition of the module into the standard library. It means that the core development team assumes responsibility of the module, similarly to any other module in the standard library. Which modules should go through ``__preview__`` ----------------------------------------------- We expect most modules proposed for addition into the Python standard library to go through a minor release in ``__preview__``. There may, however, be some exceptions, such as modules that use a pre-defined API (for example ``lzma``, which generally follows the API of the existing ``bz2`` module), or modules with an API that has wide acceptance in the Python development community. In any case, modules that are proposed to be added to the standard library, whether via ``__preview__`` or directly, must fulfill the acceptance conditions set by PEP 2. It is important to stress that the aim of of this proposal is not to make the process of adding new modules to the standard library more difficult. On the contrary, it tries to provide a means to add *more* useful libraries. Modules which are obvious candidates for entry can be added as before. Modules which due to uncertainties about the API could be stalled for a long time now have a means to still be distributed with Python, via an incubation period in the ``__preview__`` package. Criteria for "graduation" ------------------------- In principle, most modules in the ``__preview__`` package should eventually graduate to the stable standard library. Some reasons for not graduating are: * The module may prove to be unstable or fragile, without sufficient developer support to maintain it. * A much better alternative module may be found during the preview release Essentially, the decision will be made by the core developers on a per-case basis. The point to emphasize here is that a module's appearance in the ``__preview__`` package in some release does not guarantee it will continue being part of Python in the next release. Example ------- Suppose the ``example`` module is a candidate for inclusion in the standard library, but some Python developers aren't convinced that it presents the best API for the problem it intends to solve. The module can then be added to the ``__preview__`` package in release ``3.X``, importable via:: from __preview__ import example Assuming the module is then promoted to the the standard library proper in release ``3.X+1``, it will be moved to a permanent location in the library:: import example And importing it from ``__preview__`` will no longer work. Rationale ========= Benefits for the core development team -------------------------------------- Currently, the core developers are really reluctant to add new interfaces to the standard library. This is because as soon as they're published in a release, API design mistakes get locked in due to backward compatibility concerns. By gating all major API additions through some kind of a preview mechanism for a full release, we get one full release cycle of community feedback before we lock in the APIs with our standard backward compatibility guarantee. We can also start integrating preview modules with the rest of the standard library early, so long as we make it clear to packagers that the preview modules should not be considered optional. The only difference between preview APIs and the rest of the standard library is that preview APIs are explicitly exempted from the usual backward compatibility guarantees. Essentially, the ``__preview__`` package is intended to lower the risk of locking in minor API design mistakes for extended periods of time. Currently, this concern can block new additions, even when the core development team consensus is that a particular addition is a good idea in principle. Benefits for end users ---------------------- For future end users, the broadest benefit lies in a better "out-of-the-box" experience - rather than being told "oh, the standard library tools for task X are horrible, download this 3rd party library instead", those superior tools are more likely to be just be an import away. For environments where developers are required to conduct due diligence on their upstream dependencies (severely harming the cost-effectiveness of, or even ruling out entirely, much of the material on PyPI), the key benefit lies in ensuring that anything in the ``__preview__`` package is clearly under python-dev's aegis from at least the following perspectives: * Licensing: Redistributed by the PSF under a Contributor Licensing Agreement. * Documentation: The documentation of the module is published and organized via the standard Python documentation tools (i.e. ReST source, output generated with Sphinx and published on http://docs.python.org). * Testing: The module test suites are run on the python.org buildbot fleet and results published via http://www.python.org/dev/buildbot. * Issue management: Bugs and feature requests are handled on http://bugs.python.org * Source control: The master repository for the software is published on http://hg.python.org. Candidates for inclusion into __preview__ ========================================= For Python 3.3, there are a number of clear current candidates: * ``regex`` (http://pypi.python.org/pypi/regex) * ``daemon`` (PEP 3143) * ``ipaddr`` (PEP 3144) Other possible future use cases include: * Improved HTTP modules (e.g. ``requests``) * HTML 5 parsing support (e.g. ``html5lib``) * Improved URL/URI/IRI parsing * A standard image API (PEP 368) * Encapsulation of the import state (PEP 368) * Standard event loop API (PEP 3153) * A binary version of WSGI for Python 3 (e.g. PEP 444) * Generic function support (e.g. ``simplegeneric``) Relationship with PEP 407 ========================= PEP 407 proposes a change to the core Python release cycle to permit interim releases every 6 months (perhaps limited to standard library updates). If such a change to the release cycle is made, the following policy for the ``__preview__`` namespace is suggested: * For long term support releases, the ``__preview__`` namespace would always be empty. * New modules would be accepted into the ``__preview__`` namespace only in interim releases that immediately follow a long term support release. * All modules added will either be migrated to their final location in the standard library or dropped entirely prior to the next long term support release. Rejected alternatives and variations ==================================== Using ``__future__`` -------------------- Python already has a "forward-looking" namespace in the form of the ``__future__`` module, so it's reasonable to ask why that can't be re-used for this new purpose. There are two reasons why doing so not appropriate: 1. The ``__future__`` module is actually linked to a separate compiler directives feature that can actually change the way the Python interpreter compiles a module. We don't want that for the preview package - we just want an ordinary Python package. 2. The ``__future__`` module comes with an express promise that names will be maintained in perpetuity, long after the associated features have become the compiler's default behaviour. Again, this is precisely the opposite of what is intended for the preview package - it is almost certain that all names added to the preview will be removed at some point, most likely due to their being moved to a permanent home in the standard library, but also potentially due to their being reverted to third party package status (if community feedback suggests the proposed addition is irredeemably broken). Versioning the package ---------------------- One proposed alternative [1]_ was to add explicit versioning to the ``__preview__`` package, i.e. ``__preview34__``. We think that it's better to simply define that a module being in ``__preview__`` in Python 3.X will either graduate to the normal standard library namespace in Python 3.X+1 or will disappear from the Python source tree altogether. Versioning the ``_preview__`` package complicates the process and does not align well with the main intent of this proposal. Using a package name without leading and trailing underscores ------------------------------------------------------------- It was proposed [1]_ to use a package name like ``preview`` or ``exp``, instead of ``__preview__``. This was rejected in the discussion due to the special meaning a "dunder" package name (that is, a name *with* leading and trailing double-underscores) conveys in Python. Besides, a non-dunder name would suggest normal standard library API stability guarantees, which is not the intention of the ``__preview__`` package. Preserving pickle compatibility ------------------------------- A pickled class instance based on a module in ``__preview__`` in release 3.X won't be unpickle-able in release 3.X+1, where the module won't be in ``__preview__``. Special code may be added to make this work, but this goes against the intent of this proposal, since it implies backward compatibility. Therefore, this PEP does not propose to preserve pickle compatibility. Credits ======= Dj Gilcrease initially proposed the idea of having a ``__preview__`` package in Python [2]_. Although his original proposal uses the name ``__experimental__``, we feel that ``__preview__`` conveys the meaning of this package in a better way. References ========== .. [#] Discussed in this thread: http://mail.python.org/pipermail/python-ideas/2012-January/013246.html .. [#] http://mail.python.org/pipermail/python-ideas/2011-August/011278.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

25 95

OS X build break
by Meador Inge Feb. 4, 2012

Feb. 4, 2012

On Sat, Dec 31, 2011 at 5:56 PM, Guido van Rossum <guido(a)python.org> wrote: > PS. I would propose a specific fix but I can't seem to build a working > CPython from the trunk on my laptop (OS X 10.6, Xcode 4.1). I get this error > late in the build: > > ./python.exe -SE -m sysconfig --generate-posix-vars > Fatal Python error: Py_Initialize: can't initialize sys standard streams > Traceback (most recent call last): > File "/Users/guido/cpython/Lib/io.py", … [View More]

2 2

PEP 409 - final?
by Ethan Furman Feb. 3, 2012

Feb. 3, 2012

I haven't seen any further discussion here or in the bug tracker. Below is the latest version of this PEP, now with a section on Language Details. Who makes the final call on this? Any idea how long that will take? (Not that I'm antsy, or anything... ;) PEP: 409 Title: Suppressing exception context Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman <ethan(a)stoneleaf.us> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 26-Jan-2012 Post-History: 30-… [View More]Aug-2002, 01-Feb-2012 Abstract ======== One of the open issues from PEP 3134 is suppressing context: currently there is no way to do it. This PEP proposes one. Rationale ========= There are two basic ways to generate exceptions: 1) Python does it (buggy code, missing resources, ending loops, etc.) 2) manually (with a raise statement) When writing libraries, or even just custom classes, it can become necessary to raise exceptions; moreover it can be useful, even necessary, to change from one exception to another. To take an example from my dbf module:: try: value = int(value) except Exception: raise DbfError(...) Whatever the original exception was (``ValueError``, ``TypeError``, or something else) is irrelevant. The exception from this point on is a ``DbfError``, and the original exception is of no value. However, if this exception is printed, we would currently see both. Alternatives ============ Several possibilities have been put forth: * ``raise as NewException()`` Reuses the ``as`` keyword; can be confusing since we are not really reraising the originating exception * ``raise NewException() from None`` Follows existing syntax of explicitly declaring the originating exception * ``exc = NewException(); exc.__context__ = None; raise exc`` Very verbose way of the previous method * ``raise NewException.no_context(...)`` Make context suppression a class method. All of the above options will require changes to the core. Proposal ======== I proprose going with the second option:: raise NewException from None It has the advantage of using the existing pattern of explicitly setting the cause:: raise KeyError() from NameError() but because the 'cause' is ``None`` the previous context, while retained, is not displayed by the default exception printing routines. Language Details ================ Currently, ``__context__`` and ``__cause__`` start out as None, and then get set as exceptions occur. To support ``from None``, ``__context__`` will stay as it is, but ``__cause__`` will start out as ``False``, and will change to ``None`` when the ``raise ... from None`` method is used. The default exception printing routine will then: * If ``__cause__`` is ``False`` the ``__context__`` (if any) will be printed. * If ``__cause__`` is ``None`` the ``__context__`` will not be printed. * if ``__cause__`` is anything else, ``__cause__`` will be printed. This has the benefit of leaving the ``__context__`` intact for future logging, querying, etc., while suppressing its display if it is not caught. This is important for those times when trying to debug poorly written libraries with `bad error messages`_. Patches ======= There is a patch for CPython implementing this attached to `Issue 6210`_. References ========== Discussion and refinements in this `thread on python-dev`_. .. _bad error messages: http://bugs.python.org/msg152294 .. _Issue 6210: http://bugs.python.org/issue6210 .. _thread on python-dev: http://mail.python.org/pipermail/python-dev/2012-January/115838.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

14 58

PEP: New timestamp formats
by Victor Stinner Feb. 3, 2012

Feb. 3, 2012

Even if I am not really conviced that a PEP helps to design an API, here is a draft of a PEP to add new timestamp formats to Python 3.3. Don't see the draft as a final proposition, it is just a document supposed to help the discussion :-) --- PEP: xxx Title: New timestamp formats Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)haypocalc.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 01-Feburary-2012 Python-Version: 3.3 … [View More]Abstract ======== Python 3.3 introduced functions supporting nanosecond resolutions. Python 3.3 only supports int or float to store timestamps, but these types cannot be use to store a timestamp with a nanosecond resolution. Motivation ========== Python 2.3 introduced float timestamps to support subsecond resolutions, os.stat() uses float timestamps by default since Python 2.5. Python 3.3 introduced functions supporting nanosecond resolutions: * os.stat() * os.utimensat() * os.futimens() * time.clock_gettime() * time.clock_getres() * time.wallclock() (reuse time.clock_gettime(time.CLOCK_MONOTONIC)) The problem is that floats of 64 bits are unable to store nanoseconds (10^-9) for timestamps bigger than 2^24 seconds (194 days 4 hours: 1970-07-14 for an Epoch timestamp) without loosing precision. .. note:: 64 bits float starts to loose precision with microsecond (10^-6) resolution for timestamp bigger than 2^33 seconds (272 years: 2242-03-16 for an Epoch timestamp). Timestamp formats ================= Choose a new format for nanosecond resolution --------------------------------------------- To support nanosecond resolution, four formats were considered: * 128 bits float * decimal.Decimal * datetime.datetime * tuple of integers Criteria -------- It should be possible to do arithmetic, for example:: t1 = time.time() # ... t2 = time.time() dt = t2 - t1 Two timestamps should be comparable (t2 > t1). The format should have a resolution of a least 1 nanosecond (without loosing precision). It is better if the format can have an arbitrary resolution. 128 bits float -------------- Add a new IEEE 754-2008 quad-precision float type. The IEEE 754-2008 quad precision float has 1 sign bit, 15 bits of exponent and 112 bits of mantissa. 128 bits float is supported by GCC (4.3), Clang and ICC. The problem is that Visual C++ 2008 doesn't support it. Python must be portable and so cannot rely on a type only available on some platforms. Another example: GCC 4.3 does not support __float128 in 32-bit mode on x86 (but gcc 4.4 does). Intel CPUs have FPU supporting 80-bit floats, but not using SSE intructions. Other CPU vendors don't support this float size. There is also a license issue: GCC uses the MPFR library which is distributed under the GNU LGPL license. This license is incompatible with the Python Software License. datetime.datetime ----------------- datetime.datetime only supports microsecond resolution, but can be enhanced to support nanosecond. datetime.datetime has issues: - there is no easy way to convert it into "seconds since the epoch" - any broken-down time has issues of time stamp ordering in the duplicate hour of switching from DST to normal time - time zone support is flaky-to-nonexistent in the datetime module decimal.Decimal --------------- The decimal module is implemented in Python and is not really fast. Using Decimal by default would cause bootstrap issue because the module is implemented in Python. Decimal can store a timestamp with any resolution, not only nanosecond, the resolution is configurable at runtime. Decimal objects support all arithmetics operations and are compatible with int and float. The decimal module is slow, but there is a C reimplementation of the decimal module which is almost ready for inclusion. tuple ----- Various kind of tuples have been proposed. All propositions only use integers: * a) (sec, nsec): C timespec structure, useful for os.futimens() for example * b) (sec, floatpart, exponent): value = sec + floatpart * 10**exponent * c) (sec, floatpart, divisor): value = sec + floatpart / divisor The format (a) only supports nanosecond resolution. The format (a) and (b) may loose precision if the clock divisor is not a power of 10. For format (c) should be enough for most cases. Creating a tuple of integers is fast. Arithmetic operations cannot be done directly on tuple: t2-t1 doesn't work for example. Final formats ------------- The PEP proposes to provide 5 different timestamp formats: * numbers: * int * float * decimal.Decimal * datetime.timedelta * broken-down time: * datetime.datetime API design ========== Change the default result type ------------------------------ Python 2.3 introduced os.stat_float_times(). The problem is that this flag is global, and so may break libraries if the application changes the type. Changing the default result type would break backward compatibility. Callback and creating a new module to convert timestamps -------------------------------------------------------- Use a callback taking integers to create a timestamp. Example with float: def timestamp_to_float(seconds, floatpart, divisor): return seconds + floatpart / divisor The time module can provide some builtin converters, and other module, like datetime, can provide their own converters. Users can define their own types. An alternative is to add new module for all functions converting timestamps. The problem is that we have to design the API of the callback and we cannot change it later. We may need more information for future needs later. os.stat: add new fields ----------------------- It was proposed to add 3 fields to os.stat() structure to get nanoseconds of timestamps. Add an argument to change the result type ----------------------------------------- Add a argument to all functions creating timestamps, like time.time(), to change their result type. It was first proposed to use a string argument, e.g. time.time(format="decimal"). The problem is that the function has to import internally a module. Then it was decided to pass directly the type, e.g. time.time(format=decimal.Decimal). Using a type, the user has first to import the module. There is no direct link between a type and the function used to create the timestamp. By default, the float type is used to keep backward compatibility. For stat functions like os.stat(), the default type depends on os.stat_float_times(). Add new functions ----------------- Add new functions for each type, examples: * time.time_decimal() * os.stat_decimal() * os.stat_datetime() * etc. Changes ======= * Add *format* optional argument to time.clock(), time.clock_gettime(), time.clock_getres(), time.time() and time.wallclock(). * Add *timestamp* optional argument to os.fstat(), os.fstatat(), os.lstat() and os.stat(). Functions accepting timestamp as input should support decimal.Decimal objects without an internal conversion to float which may loose precision: * datetime.datetime.fromtimestamp() * time.localtime() * time.gmtime() TODO: * Change os.utimensat() and os.futimens() to accept Decimal * Change os.utimensat() and os.futimens() to not accept tuple anymore * Drop os.utimensat() and os.futimens() and patch os.utimeat() instead? * datetime should maybe support nanosecond? Backwards Compatibility ======================= Changes only add an new optional argument. The default type is unchanged and there is no impact on performances. Links ===== * `Issue #11457: os.stat(): add new fields to get timestamps as Decimal objects with nanosecond resolution <http://bugs.python.org/issue11457>`_ * `Issue #13882: Add format argument for time.time(), time.clock(), ... to get a timestamp as a Decimal object <http://bugs.python.org/issue13882>`_ * `[Python-Dev] Store timestamps as decimal.Decimal objects <http://mail.python.org/pipermail/python-dev/2012-January/116025.html>`_ Copyright ========= This document has been placed in the public domain. [View Less]

10 33

Re: [Python-Dev] PEP: New timestamp formats
by Victor Stinner Feb. 3, 2012

Feb. 3, 2012

> consider changing the default on any of these that return a time > value. these for example: > * time.clock_gettime() > * time.wallclock() (reuse time.clock_gettime(time.CLOCK_MONOTONIC)) Ah. Nanosecond resolution is overkill is common cases, float is enough and is faster. I prefer to use the same type (float) by default for all functions creating timestamps. Victor

1 0

Dev In a Box: Pumped for a better UX (with video)
by anatoly techtonik Feb. 3, 2012

Feb. 3, 2012

Hi, If you don't know, Dev In a Box is "everything you need to contribute to Python in under 700 MB". I've patched it up to the latest standards of colorless console user interfaces and uploaded a video of the process for you to enjoy. http://www.youtube.com/watch?v=jbJcI9MnO_c This tool can be greatly improved to provide entrypoint for other healthy activities. Like improving docs by editing, comparing, building and sending patches for review. Specialized menus can greatly help with … [View More]

1 0