On Mon, Feb 6, 2012 at 6:12 PM, Steven D'Aprano <steve(a)pearwood.info> wrote:
> On Mon, Feb 06, 2012 at 09:01:29PM +0100, julien tayon wrote:
> > Hello,
> > Proposing vector operations on dict, and acknowledging there was an
> > homeomorphism from rooted n-ary trees to dict, was inducing the
> > possibility of making matrix of dict / trees.
> This seems interesting to me, but I don't see that they are important
> enough to be built-in to dicts. [...]
> Otherwise, this looks rather like a library of functions looking for a
> use. It might help if you demonstrate what concrete problems this helps
> you solve.
I have the problem looking for this solution!
The application for this functionality is in coding a fractal graph (or
"multigraph" in the literature). This is the most powerful structure that
Computer Science has ever conceived. If you look at the evolution of data
structures in compsci, the fractal graph is the ultimate. From lists to
trees to graphs to multigraphs. The latter elements can always encompass
the former with only O(1) extra cost.
It has the potential to encode *any* relationship from the very small to
the very large (as well as across or *laterally*) in one unified structure.
Optimize this one data structure and the whole standard library could be
refactored and simplified by an order of magnitude. Not only that, it will
pave the way for the "re-factored" internet that's being worked on which
creates a content-centric Internet beyond the graph-level, hypertext
Believe, it will be awesome.
I've noted that distutils manages depends in a way I cannot understand.
Suppose I have a minimal setup.py:
from distutils.core import setup, Extension
depends=['fop.conf'] # <---- note the typo foo->fop
Now setup.py will rebuild all every time, this is because the policy of
newer_group in build_extension is to consider 'newer' any missing file.
def build_extension(self, ext):
depends = sources + ext.depends
if not (self.force or newer_group(depends, ext_path, 'newer')):
logger.debug("skipping '%s' extension (up-to-date)", ext.name)
logger.info("building '%s' extension", ext.name)
Can someone suggest me the reason of this choice instead of
missing='error' (at least for ext.depends)?
You may remember me from a couple of years ago when I was trying to help
out with Python. Unfortunately I trod on a few toes. I now know why.
I have been diagnosed with Asperger Syndrome at 55 years old.
I would like to give it another go.
Following an earlier discussion on python-ideas , we would like to
propose the following PEP for review. Discussion is welcome. The PEP
can also be viewed in HTML form at
Title: Standard library __preview__ package
Author: Nick Coghlan <ncoghlan(a)gmail.com>,
Eli Bendersky <eliben(a)gmail.com>
Type: Standards Track
The process of including a new module into the Python standard library is
hindered by the API lock-in and promise of backward compatibility implied by
a module being formally part of Python. This PEP proposes a transitional
state for modules - inclusion in a special ``__preview__`` package for the
duration of a minor release (roughly 18 months) prior to full acceptance into
the standard library. On one hand, this state provides the module with the
benefits of being formally part of the Python distribution. On the other hand,
the core development team explicitly states that no promises are made with
regards to the module's eventual full inclusion into the standard library,
or to the stability of its API, which may change for the next release.
Proposal - the __preview__ package
Whenever the Python core development team decides that a new module should be
included into the standard library, but isn't entirely sure about whether the
module's API is optimal, the module can be placed in a special package named
``__preview__`` for a single minor release.
In the next minor release, the module may either be "graduated" into the
standard library (and occupy its natural place within its namespace, leaving the
``__preview__`` package), or be rejected and removed entirely from the Python
source tree. If the module ends up graduating into the standard library after
spending a minor release in ``__preview__``, its API may be changed according
to accumulated feedback. The core development team explicitly makes no
guarantees about API stability and backward compatibility of modules in
Entry into the ``__preview__`` package marks the start of a transition of the
module into the standard library. It means that the core development team
assumes responsibility of the module, similarly to any other module in the
Which modules should go through ``__preview__``
We expect most modules proposed for addition into the Python standard library
to go through a minor release in ``__preview__``. There may, however, be some
exceptions, such as modules that use a pre-defined API (for example ``lzma``,
which generally follows the API of the existing ``bz2`` module), or modules
with an API that has wide acceptance in the Python development community.
In any case, modules that are proposed to be added to the standard library,
whether via ``__preview__`` or directly, must fulfill the acceptance conditions
set by PEP 2.
It is important to stress that the aim of of this proposal is not to make the
process of adding new modules to the standard library more difficult. On the
contrary, it tries to provide a means to add *more* useful libraries. Modules
which are obvious candidates for entry can be added as before. Modules which
due to uncertainties about the API could be stalled for a long time now have
a means to still be distributed with Python, via an incubation period in the
Criteria for "graduation"
In principle, most modules in the ``__preview__`` package should eventually
graduate to the stable standard library. Some reasons for not graduating are:
* The module may prove to be unstable or fragile, without sufficient developer
support to maintain it.
* A much better alternative module may be found during the preview release
Essentially, the decision will be made by the core developers on a per-case
basis. The point to emphasize here is that a module's appearance in the
``__preview__`` package in some release does not guarantee it will continue
being part of Python in the next release.
Suppose the ``example`` module is a candidate for inclusion in the standard
library, but some Python developers aren't convinced that it presents the best
API for the problem it intends to solve. The module can then be added to the
``__preview__`` package in release ``3.X``, importable via::
from __preview__ import example
Assuming the module is then promoted to the the standard library proper in
release ``3.X+1``, it will be moved to a permanent location in the library::
And importing it from ``__preview__`` will no longer work.
Benefits for the core development team
Currently, the core developers are really reluctant to add new interfaces to
the standard library. This is because as soon as they're published in a
release, API design mistakes get locked in due to backward compatibility
By gating all major API additions through some kind of a preview mechanism
for a full release, we get one full release cycle of community feedback
before we lock in the APIs with our standard backward compatibility guarantee.
We can also start integrating preview modules with the rest of the standard
library early, so long as we make it clear to packagers that the preview
modules should not be considered optional. The only difference between preview
APIs and the rest of the standard library is that preview APIs are explicitly
exempted from the usual backward compatibility guarantees.
Essentially, the ``__preview__`` package is intended to lower the risk of
locking in minor API design mistakes for extended periods of time. Currently,
this concern can block new additions, even when the core development team
consensus is that a particular addition is a good idea in principle.
Benefits for end users
For future end users, the broadest benefit lies in a better "out-of-the-box"
experience - rather than being told "oh, the standard library tools for task X
are horrible, download this 3rd party library instead", those superior tools
are more likely to be just be an import away.
For environments where developers are required to conduct due diligence on
their upstream dependencies (severely harming the cost-effectiveness of, or
even ruling out entirely, much of the material on PyPI), the key benefit lies
in ensuring that anything in the ``__preview__`` package is clearly under
python-dev's aegis from at least the following perspectives:
* Licensing: Redistributed by the PSF under a Contributor Licensing Agreement.
* Documentation: The documentation of the module is published and organized via
the standard Python documentation tools (i.e. ReST source, output generated
with Sphinx and published on http://docs.python.org).
* Testing: The module test suites are run on the python.org buildbot fleet
and results published via http://www.python.org/dev/buildbot.
* Issue management: Bugs and feature requests are handled on
* Source control: The master repository for the software is published
Candidates for inclusion into __preview__
For Python 3.3, there are a number of clear current candidates:
* ``regex`` (http://pypi.python.org/pypi/regex)
* ``daemon`` (PEP 3143)
* ``ipaddr`` (PEP 3144)
Other possible future use cases include:
* Improved HTTP modules (e.g. ``requests``)
* HTML 5 parsing support (e.g. ``html5lib``)
* Improved URL/URI/IRI parsing
* A standard image API (PEP 368)
* Encapsulation of the import state (PEP 368)
* Standard event loop API (PEP 3153)
* A binary version of WSGI for Python 3 (e.g. PEP 444)
* Generic function support (e.g. ``simplegeneric``)
Relationship with PEP 407
PEP 407 proposes a change to the core Python release cycle to permit interim
releases every 6 months (perhaps limited to standard library updates). If
such a change to the release cycle is made, the following policy for the
``__preview__`` namespace is suggested:
* For long term support releases, the ``__preview__`` namespace would always
* New modules would be accepted into the ``__preview__`` namespace only in
interim releases that immediately follow a long term support release.
* All modules added will either be migrated to their final location in the
standard library or dropped entirely prior to the next long term support
Rejected alternatives and variations
Python already has a "forward-looking" namespace in the form of the
``__future__`` module, so it's reasonable to ask why that can't be re-used for
this new purpose.
There are two reasons why doing so not appropriate:
1. The ``__future__`` module is actually linked to a separate compiler
directives feature that can actually change the way the Python interpreter
compiles a module. We don't want that for the preview package - we just want
an ordinary Python package.
2. The ``__future__`` module comes with an express promise that names will be
maintained in perpetuity, long after the associated features have become the
compiler's default behaviour. Again, this is precisely the opposite of what is
intended for the preview package - it is almost certain that all names added to
the preview will be removed at some point, most likely due to their being moved
to a permanent home in the standard library, but also potentially due to their
being reverted to third party package status (if community feedback suggests the
proposed addition is irredeemably broken).
Versioning the package
One proposed alternative _ was to add explicit versioning to the
``__preview__`` package, i.e. ``__preview34__``. We think that it's better to
simply define that a module being in ``__preview__`` in Python 3.X will either
graduate to the normal standard library namespace in Python 3.X+1 or will
disappear from the Python source tree altogether. Versioning the ``_preview__``
package complicates the process and does not align well with the main intent of
Using a package name without leading and trailing underscores
It was proposed _ to use a package name like ``preview`` or ``exp``, instead
of ``__preview__``. This was rejected in the discussion due to the special
meaning a "dunder" package name (that is, a name *with* leading and
trailing double-underscores) conveys in Python. Besides, a non-dunder name
would suggest normal standard library API stability guarantees, which is not
the intention of the ``__preview__`` package.
Preserving pickle compatibility
A pickled class instance based on a module in ``__preview__`` in release 3.X
won't be unpickle-able in release 3.X+1, where the module won't be in
``__preview__``. Special code may be added to make this work, but this goes
against the intent of this proposal, since it implies backward compatibility.
Therefore, this PEP does not propose to preserve pickle compatibility.
Dj Gilcrease initially proposed the idea of having a ``__preview__`` package
in Python _. Although his original proposal uses the name
``__experimental__``, we feel that ``__preview__`` conveys the meaning of this
package in a better way.
.. [#] Discussed in this thread:
.. [#] http://mail.python.org/pipermail/python-ideas/2011-August/011278.html
This document has been placed in the public domain.
On Sat, Dec 31, 2011 at 5:56 PM, Guido van Rossum <guido(a)python.org> wrote:
> PS. I would propose a specific fix but I can't seem to build a working
> CPython from the trunk on my laptop (OS X 10.6, Xcode 4.1). I get this error
> late in the build:
> ./python.exe -SE -m sysconfig --generate-posix-vars
> Fatal Python error: Py_Initialize: can't initialize sys standard streams
> Traceback (most recent call last):
> File "/Users/guido/cpython/Lib/io.py", line 60, in <module>
> make: *** [Lib/_sysconfigdata.py] Abort trap
I am having this problem now too. I am running OS X 10.7.2.
3.2 still builds for me, but I can't build default.
Did you ever get past it? Anyone else seeing this?
I haven't seen any further discussion here or in the bug tracker. Below
is the latest version of this PEP, now with a section on Language Details.
Who makes the final call on this? Any idea how long that will take?
(Not that I'm antsy, or anything... ;)
Title: Suppressing exception context
Author: Ethan Furman <ethan(a)stoneleaf.us>
Type: Standards Track
Post-History: 30-Aug-2002, 01-Feb-2012
One of the open issues from PEP 3134 is suppressing context: currently
there is no way to do it. This PEP proposes one.
There are two basic ways to generate exceptions:
1) Python does it (buggy code, missing resources, ending loops, etc.)
2) manually (with a raise statement)
When writing libraries, or even just custom classes, it can become
necessary to raise exceptions; moreover it can be useful, even
necessary, to change from one exception to another. To take an example
from my dbf module::
value = int(value)
Whatever the original exception was (``ValueError``, ``TypeError``, or
something else) is irrelevant. The exception from this point on is a
``DbfError``, and the original exception is of no value. However, if
this exception is printed, we would currently see both.
Several possibilities have been put forth:
* ``raise as NewException()``
Reuses the ``as`` keyword; can be confusing since we are not really
reraising the originating exception
* ``raise NewException() from None``
Follows existing syntax of explicitly declaring the originating
* ``exc = NewException(); exc.__context__ = None; raise exc``
Very verbose way of the previous method
* ``raise NewException.no_context(...)``
Make context suppression a class method.
All of the above options will require changes to the core.
I proprose going with the second option::
raise NewException from None
It has the advantage of using the existing pattern of explicitly setting
raise KeyError() from NameError()
but because the 'cause' is ``None`` the previous context, while retained,
is not displayed by the default exception printing routines.
Currently, ``__context__`` and ``__cause__`` start out as None, and then get
set as exceptions occur.
To support ``from None``, ``__context__`` will stay as it is, but
``__cause__`` will start out as ``False``, and will change to ``None``
when the ``raise ... from None`` method is used.
The default exception printing routine will then:
* If ``__cause__`` is ``False`` the ``__context__`` (if any) will be
* If ``__cause__`` is ``None`` the ``__context__`` will not be printed.
* if ``__cause__`` is anything else, ``__cause__`` will be printed.
This has the benefit of leaving the ``__context__`` intact for future
logging, querying, etc., while suppressing its display if it is not caught.
This is important for those times when trying to debug poorly written
libraries with `bad error messages`_.
There is a patch for CPython implementing this attached to `Issue 6210`_.
Discussion and refinements in this `thread on python-dev`_.
.. _bad error messages: http://bugs.python.org/msg152294
.. _Issue 6210: http://bugs.python.org/issue6210
.. _thread on python-dev:
This document has been placed in the public domain.
Even if I am not really conviced that a PEP helps to design an API,
here is a draft of a PEP to add new timestamp formats to Python 3.3.
Don't see the draft as a final proposition, it is just a document
supposed to help the discussion :-)
Title: New timestamp formats
Author: Victor Stinner <victor.stinner(a)haypocalc.com>
Type: Standards Track
Python 3.3 introduced functions supporting nanosecond resolutions. Python 3.3
only supports int or float to store timestamps, but these types cannot be use
to store a timestamp with a nanosecond resolution.
Python 2.3 introduced float timestamps to support subsecond resolutions,
os.stat() uses float timestamps by default since Python 2.5. Python 3.3
introduced functions supporting nanosecond resolutions:
* time.wallclock() (reuse time.clock_gettime(time.CLOCK_MONOTONIC))
The problem is that floats of 64 bits are unable to store nanoseconds (10^-9)
for timestamps bigger than 2^24 seconds (194 days 4 hours: 1970-07-14 for an
Epoch timestamp) without loosing precision.
64 bits float starts to loose precision with microsecond (10^-6) resolution
for timestamp bigger than 2^33 seconds (272 years: 2242-03-16 for an Epoch
Choose a new format for nanosecond resolution
To support nanosecond resolution, four formats were considered:
* 128 bits float
* tuple of integers
It should be possible to do arithmetic, for example::
t1 = time.time()
t2 = time.time()
dt = t2 - t1
Two timestamps should be comparable (t2 > t1).
The format should have a resolution of a least 1 nanosecond (without loosing
precision). It is better if the format can have an arbitrary resolution.
128 bits float
Add a new IEEE 754-2008 quad-precision float type. The IEEE 754-2008 quad
precision float has 1 sign bit, 15 bits of exponent and 112 bits of mantissa.
128 bits float is supported by GCC (4.3), Clang and ICC. The problem is that
Visual C++ 2008 doesn't support it. Python must be portable and so cannot rely
on a type only available on some platforms. Another example: GCC 4.3 does not
support __float128 in 32-bit mode on x86 (but gcc 4.4 does).
Intel CPUs have FPU supporting 80-bit floats, but not using SSE intructions.
Other CPU vendors don't support this float size.
There is also a license issue: GCC uses the MPFR library which is distributed
under the GNU LGPL license. This license is incompatible with the Python
datetime.datetime only supports microsecond resolution, but can be enhanced
to support nanosecond.
datetime.datetime has issues:
- there is no easy way to convert it into "seconds since the epoch"
- any broken-down time has issues of time stamp ordering in the
duplicate hour of switching from DST to normal time
- time zone support is flaky-to-nonexistent in the datetime module
The decimal module is implemented in Python and is not really fast.
Using Decimal by default would cause bootstrap issue because the module is
implemented in Python.
Decimal can store a timestamp with any resolution, not only nanosecond, the
resolution is configurable at runtime.
Decimal objects support all arithmetics operations and are compatible with int
The decimal module is slow, but there is a C reimplementation of the decimal
module which is almost ready for inclusion.
Various kind of tuples have been proposed. All propositions only use integers:
* a) (sec, nsec): C timespec structure, useful for os.futimens() for example
* b) (sec, floatpart, exponent): value = sec + floatpart * 10**exponent
* c) (sec, floatpart, divisor): value = sec + floatpart / divisor
The format (a) only supports nanosecond resolution.
The format (a) and (b) may loose precision if the clock divisor is not a
power of 10.
For format (c) should be enough for most cases.
Creating a tuple of integers is fast.
Arithmetic operations cannot be done directly on tuple: t2-t1 doesn't work for
The PEP proposes to provide 5 different timestamp formats:
* broken-down time:
Change the default result type
Python 2.3 introduced os.stat_float_times(). The problem is that this flag
is global, and so may break libraries if the application changes the type.
Changing the default result type would break backward compatibility.
Callback and creating a new module to convert timestamps
Use a callback taking integers to create a timestamp. Example with float:
def timestamp_to_float(seconds, floatpart, divisor):
return seconds + floatpart / divisor
The time module can provide some builtin converters, and other module, like
datetime, can provide their own converters. Users can define their own types.
An alternative is to add new module for all functions converting timestamps.
The problem is that we have to design the API of the callback and we cannot
change it later. We may need more information for future needs later.
os.stat: add new fields
It was proposed to add 3 fields to os.stat() structure to get nanoseconds of
Add an argument to change the result type
Add a argument to all functions creating timestamps, like time.time(), to
change their result type. It was first proposed to use a string argument,
e.g. time.time(format="decimal"). The problem is that the function has
to import internally a module. Then it was decided to pass directly the
type, e.g. time.time(format=decimal.Decimal). Using a type, the user has
first to import the module. There is no direct link between a type and the
function used to create the timestamp.
By default, the float type is used to keep backward compatibility. For stat
functions like os.stat(), the default type depends on os.stat_float_times().
Add new functions
Add new functions for each type, examples:
* Add *format* optional argument to time.clock(), time.clock_gettime(),
time.clock_getres(), time.time() and time.wallclock().
* Add *timestamp* optional argument to os.fstat(), os.fstatat(), os.lstat()
Functions accepting timestamp as input should support decimal.Decimal objects
without an internal conversion to float which may loose precision:
* Change os.utimensat() and os.futimens() to accept Decimal
* Change os.utimensat() and os.futimens() to not accept tuple anymore
* Drop os.utimensat() and os.futimens() and patch os.utimeat() instead?
* datetime should maybe support nanosecond?
Changes only add an new optional argument. The default type is unchanged and
there is no impact on performances.
* `Issue #11457: os.stat(): add new fields to get timestamps as
Decimal objects with nanosecond resolution
* `Issue #13882: Add format argument for time.time(), time.clock(),
... to get a timestamp as a Decimal object
* `[Python-Dev] Store timestamps as decimal.Decimal objects
This document has been placed in the public domain.
> consider changing the default on any of these that return a time
> value. these for example:
> * time.clock_gettime()
> * time.wallclock() (reuse time.clock_gettime(time.CLOCK_MONOTONIC))
Ah. Nanosecond resolution is overkill is common cases, float is enough
and is faster. I prefer to use the same type (float) by default for
all functions creating timestamps.
If you don't know, Dev In a Box is "everything you need to contribute to
Python in under 700 MB". I've patched it up to the latest standards
of colorless console user interfaces and uploaded a video of the process
for you to enjoy.
This tool can be greatly improved to provide entrypoint for other healthy
activities. Like improving docs by editing, comparing, building and sending
patches for review. Specialized menus can greatly help with automating
common tasks, which are not limited by sources fetching.