Mailman 3 January 2012 - Python-Dev

distutils 'depends' management
by Matteo Bertini Feb. 6, 2012

Feb. 6, 2012

Hello, I've noted that distutils manages depends in a way I cannot understand. Suppose I have a minimal setup.py: from distutils.core import setup, Extension setup( name='foo', version='1.0', ext_modules=[ Extension('foo', sources=['foo.c'], depends=['fop.conf'] # <---- note the typo foo->fop ), ] ) Now setup.py will rebuild all every time, this is because the policy of newer_group in build_extension is to consider 'newer' any missing file. http://bit.ly/… [View More]

2 2

PEP 408 -- Standard library __preview__ package
by Eli Bendersky Feb. 5, 2012

Feb. 5, 2012

Hello, Following an earlier discussion on python-ideas [1], we would like to propose the following PEP for review. Discussion is welcome. The PEP can also be viewed in HTML form at http://www.python.org/dev/peps/pep-0408/ [1] http://mail.python.org/pipermail/python-ideas/2012-January/013246.html Eli --------------------------- PEP: 408 Title: Standard library __preview__ package Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Eli … [View More]Bendersky <eliben(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2012-01-07 Python-Version: 3.3 Post-History: 2012-01-27 Abstract ======== The process of including a new module into the Python standard library is hindered by the API lock-in and promise of backward compatibility implied by a module being formally part of Python. This PEP proposes a transitional state for modules - inclusion in a special ``__preview__`` package for the duration of a minor release (roughly 18 months) prior to full acceptance into the standard library. On one hand, this state provides the module with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the module's eventual full inclusion into the standard library, or to the stability of its API, which may change for the next release. Proposal - the __preview__ package ================================== Whenever the Python core development team decides that a new module should be included into the standard library, but isn't entirely sure about whether the module's API is optimal, the module can be placed in a special package named ``__preview__`` for a single minor release. In the next minor release, the module may either be "graduated" into the standard library (and occupy its natural place within its namespace, leaving the ``__preview__`` package), or be rejected and removed entirely from the Python source tree. If the module ends up graduating into the standard library after spending a minor release in ``__preview__``, its API may be changed according to accumulated feedback. The core development team explicitly makes no guarantees about API stability and backward compatibility of modules in ``__preview__``. Entry into the ``__preview__`` package marks the start of a transition of the module into the standard library. It means that the core development team assumes responsibility of the module, similarly to any other module in the standard library. Which modules should go through ``__preview__`` ----------------------------------------------- We expect most modules proposed for addition into the Python standard library to go through a minor release in ``__preview__``. There may, however, be some exceptions, such as modules that use a pre-defined API (for example ``lzma``, which generally follows the API of the existing ``bz2`` module), or modules with an API that has wide acceptance in the Python development community. In any case, modules that are proposed to be added to the standard library, whether via ``__preview__`` or directly, must fulfill the acceptance conditions set by PEP 2. It is important to stress that the aim of of this proposal is not to make the process of adding new modules to the standard library more difficult. On the contrary, it tries to provide a means to add *more* useful libraries. Modules which are obvious candidates for entry can be added as before. Modules which due to uncertainties about the API could be stalled for a long time now have a means to still be distributed with Python, via an incubation period in the ``__preview__`` package. Criteria for "graduation" ------------------------- In principle, most modules in the ``__preview__`` package should eventually graduate to the stable standard library. Some reasons for not graduating are: * The module may prove to be unstable or fragile, without sufficient developer support to maintain it. * A much better alternative module may be found during the preview release Essentially, the decision will be made by the core developers on a per-case basis. The point to emphasize here is that a module's appearance in the ``__preview__`` package in some release does not guarantee it will continue being part of Python in the next release. Example ------- Suppose the ``example`` module is a candidate for inclusion in the standard library, but some Python developers aren't convinced that it presents the best API for the problem it intends to solve. The module can then be added to the ``__preview__`` package in release ``3.X``, importable via:: from __preview__ import example Assuming the module is then promoted to the the standard library proper in release ``3.X+1``, it will be moved to a permanent location in the library:: import example And importing it from ``__preview__`` will no longer work. Rationale ========= Benefits for the core development team -------------------------------------- Currently, the core developers are really reluctant to add new interfaces to the standard library. This is because as soon as they're published in a release, API design mistakes get locked in due to backward compatibility concerns. By gating all major API additions through some kind of a preview mechanism for a full release, we get one full release cycle of community feedback before we lock in the APIs with our standard backward compatibility guarantee. We can also start integrating preview modules with the rest of the standard library early, so long as we make it clear to packagers that the preview modules should not be considered optional. The only difference between preview APIs and the rest of the standard library is that preview APIs are explicitly exempted from the usual backward compatibility guarantees. Essentially, the ``__preview__`` package is intended to lower the risk of locking in minor API design mistakes for extended periods of time. Currently, this concern can block new additions, even when the core development team consensus is that a particular addition is a good idea in principle. Benefits for end users ---------------------- For future end users, the broadest benefit lies in a better "out-of-the-box" experience - rather than being told "oh, the standard library tools for task X are horrible, download this 3rd party library instead", those superior tools are more likely to be just be an import away. For environments where developers are required to conduct due diligence on their upstream dependencies (severely harming the cost-effectiveness of, or even ruling out entirely, much of the material on PyPI), the key benefit lies in ensuring that anything in the ``__preview__`` package is clearly under python-dev's aegis from at least the following perspectives: * Licensing: Redistributed by the PSF under a Contributor Licensing Agreement. * Documentation: The documentation of the module is published and organized via the standard Python documentation tools (i.e. ReST source, output generated with Sphinx and published on http://docs.python.org). * Testing: The module test suites are run on the python.org buildbot fleet and results published via http://www.python.org/dev/buildbot. * Issue management: Bugs and feature requests are handled on http://bugs.python.org * Source control: The master repository for the software is published on http://hg.python.org. Candidates for inclusion into __preview__ ========================================= For Python 3.3, there are a number of clear current candidates: * ``regex`` (http://pypi.python.org/pypi/regex) * ``daemon`` (PEP 3143) * ``ipaddr`` (PEP 3144) Other possible future use cases include: * Improved HTTP modules (e.g. ``requests``) * HTML 5 parsing support (e.g. ``html5lib``) * Improved URL/URI/IRI parsing * A standard image API (PEP 368) * Encapsulation of the import state (PEP 368) * Standard event loop API (PEP 3153) * A binary version of WSGI for Python 3 (e.g. PEP 444) * Generic function support (e.g. ``simplegeneric``) Relationship with PEP 407 ========================= PEP 407 proposes a change to the core Python release cycle to permit interim releases every 6 months (perhaps limited to standard library updates). If such a change to the release cycle is made, the following policy for the ``__preview__`` namespace is suggested: * For long term support releases, the ``__preview__`` namespace would always be empty. * New modules would be accepted into the ``__preview__`` namespace only in interim releases that immediately follow a long term support release. * All modules added will either be migrated to their final location in the standard library or dropped entirely prior to the next long term support release. Rejected alternatives and variations ==================================== Using ``__future__`` -------------------- Python already has a "forward-looking" namespace in the form of the ``__future__`` module, so it's reasonable to ask why that can't be re-used for this new purpose. There are two reasons why doing so not appropriate: 1. The ``__future__`` module is actually linked to a separate compiler directives feature that can actually change the way the Python interpreter compiles a module. We don't want that for the preview package - we just want an ordinary Python package. 2. The ``__future__`` module comes with an express promise that names will be maintained in perpetuity, long after the associated features have become the compiler's default behaviour. Again, this is precisely the opposite of what is intended for the preview package - it is almost certain that all names added to the preview will be removed at some point, most likely due to their being moved to a permanent home in the standard library, but also potentially due to their being reverted to third party package status (if community feedback suggests the proposed addition is irredeemably broken). Versioning the package ---------------------- One proposed alternative [1]_ was to add explicit versioning to the ``__preview__`` package, i.e. ``__preview34__``. We think that it's better to simply define that a module being in ``__preview__`` in Python 3.X will either graduate to the normal standard library namespace in Python 3.X+1 or will disappear from the Python source tree altogether. Versioning the ``_preview__`` package complicates the process and does not align well with the main intent of this proposal. Using a package name without leading and trailing underscores ------------------------------------------------------------- It was proposed [1]_ to use a package name like ``preview`` or ``exp``, instead of ``__preview__``. This was rejected in the discussion due to the special meaning a "dunder" package name (that is, a name *with* leading and trailing double-underscores) conveys in Python. Besides, a non-dunder name would suggest normal standard library API stability guarantees, which is not the intention of the ``__preview__`` package. Preserving pickle compatibility ------------------------------- A pickled class instance based on a module in ``__preview__`` in release 3.X won't be unpickle-able in release 3.X+1, where the module won't be in ``__preview__``. Special code may be added to make this work, but this goes against the intent of this proposal, since it implies backward compatibility. Therefore, this PEP does not propose to preserve pickle compatibility. Credits ======= Dj Gilcrease initially proposed the idea of having a ``__preview__`` package in Python [2]_. Although his original proposal uses the name ``__experimental__``, we feel that ``__preview__`` conveys the meaning of this package in a better way. References ========== .. [#] Discussed in this thread: http://mail.python.org/pipermail/python-ideas/2012-January/013246.html .. [#] http://mail.python.org/pipermail/python-ideas/2011-August/011278.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

25 95

Python 3 optimizations, continued, continued again...
by stefan brunthaler Feb. 3, 2012

Feb. 3, 2012

Hi guys, while there is at least some interest in incorporating my optimizations, response has still been low. I figure that the changes are probably too much for a single big incorporation step. On a recent flight, I thought about cutting it down to make it more easily digestible. The basic idea is to remove the optimized interpreter dispatch loop and advanced instruction format and use the existing ones. Currently (rev. ca8a0dfb2176), opcode.h uses 109 of potentially available 255 … [View More]

11 23

Switching to Visual Studio 2010
by "Martin v. Löwis" Feb. 1, 2012

Feb. 1, 2012

It seems a number of people are interested that the Python trunk switches to Visual Studio 2010 *now*. I've been hesitant to agree to such a change, as I still hope that Python can skip over VS 2010 (a.k.a. VS 10), and go straight to VS 11. However, I just learned that VS 11 supposed ready VS 10 project files just fine, with no need of conversion. So I'd be willing to agree to converting the Python trunk now. It will surely cause all kinds of issues, as any switching of Visual Studio … [View More]

5 12

threading.Semaphore()'s counter can become negative for non-ints
by T.B. Jan. 31, 2012

Jan. 31, 2012

Hello python-dev, This is probably worth of a bug report: While looking at threading.py I noticed that Semaphore's counter can go below zero. This is opposed to the docs: "The counter can never go below zero; ...". Just try: import threading s = threading.Semaphore(0.5) # You can now acquire s as many times as you want! # even when s._value < 0. The fix is tiny: diff -r 265d35e8fe82 Lib/threading.py --- a/Lib/threading.py Fri Jan 27 21:17:04 2012 +0000 +++ b/Lib/threading.py Sat Jan … [View More]

9 14

cdecimal (Was: Store timestamps as decimal.Decimal objects)
by Paul Moore Jan. 31, 2012

Jan. 31, 2012

On 31 January 2012 11:11, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > although decimal.Decimal is also a contender if backed up by > Stefan's C implementation. As you mention this, and given the ongoing thread about __preview__ and "nearly ready for stdlib" modules, what is the current position on cdecimal? I seem to recall it being announced some time ago, but I don't recall any particular discussions/conclusions about including it in the stdlib. Is it being considered for … [View More]

2 1

PEP 3144 ipaddr module (was Re: PEP 408 -- Standard library __preview__ package)
by Nick Coghlan Jan. 30, 2012

Jan. 30, 2012

On Tue, Jan 31, 2012 at 4:19 AM, Scott Dial <scott+python-dev(a)scottdial.com> wrote: > PEP 3144 wasn't pronounced upon because there were significant > disagreements about the design of the API proposed in the PEP. As it > stands, I believe the authorship of ipaddr either decided that they were > not going to compromise their module or lost interest. > > See Nick Coghlan's summary: > > http://mail.python.org/pipermail//python-ideas/2011-August/011305.html Peter … [View More]Moody actually addressed all my comments from last year (alas, I forgot that python-ideas got dropped from the latter part of the email chain, so it became a private discussion between Peter, Guido and myself). I apparently got distracted by other issues and never followed up on Peter's final review request. The branch with the relevant changes is here (these weren't added back into ipaddr mainline since they aren't all backwards compatible with the existing ipaddr API): http://code.google.com/p/ipaddr-py/source/browse/#svn%2Fbranches%2F3144 Peter was very responsive and accommodating during that discussion :) (The notes below are an edited version of Peter's off-list reply to me from last year, reflecting the final state of the ipaddr 3144 branch) On Mon, Aug 29, 2011 at 7:09 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: I believe the PEP would be significantly more palatable with the following changes/additions: 1. Draft ReStructuredText documentation for inclusion in the stdlib docs (still needed) 2. Removal of the "ip" attribute of IP network objects (since it makes the nominal "networks" behave like IP interface definitions) the Class hierarchy now looks like: _IPAddrBase(object) # mother of everything _BaseAddress(_IPAddrBase) # base for addresses _ BaseNetwork(_IPAddrBase) # base for networks and interfaces, could use be renamed. _BaseV4(object) # ipv4 base _BaseV6(object) # ipv6 base IPv4Address(_BaseV4, _BaseAddress) IPv4Interface(_BaseV4, _BaseNetwork) IPv4Network(IPv4Interface) IPv6Address(_BaseV6, _BaseAddress) IPv6Interface(_BaseV6, _BaseNetwork) IPv6Network(IPv6Interface) (essentially, the current ipaddr "Network" objects become "Interface" objects in PEP 3144, with a new strict "Network" object that has no ip attribute) 3. "network" property renamed to "netaddr" (since it returns an address object rather than a network object) renamed to network_address. did the same for the broadcast_address. 4. "strict" parameter removed from class signatures, replaced with class method for non-strict behaviour 'strict' is gone, just create IPv*Interface objects or use the ip_interface API instead. Network objects are always strict. 5. Factory functions renamed so they don't look like class names (ip_network, ip_address, ip) Now ip_address, ip_network, ip_interface 6. "strict" parameter on factory functions modified to default to True rather than False 'strict' is gone. Interfaces allow a host IP, Networks don't. 7. Addition of an explicit "IPInterface" class to cover the association of an address with a specific network that is currently handled by storing arbitrary addresses on IP network objects done. So with a cleanup of the docstrings (and creation of some ReST docs based on them) a definite +1 from me for inclusion of ipaddr (based on the 3144 branch in SVN) in 3.3. (with the tweaks to the API, we may want to use a different name like "ipaddress" or "iptools", though - otherwise people could be legitimately confused by the differences relative to the PyPI "ipaddr" module) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

3 4

Re: [Python-Dev] [Python-checkins] cpython: Issue #8828: Add new function os.replace(), for cross-platform renaming with
by Brett Cannon Jan. 30, 2012

Jan. 30, 2012

Should this end up being used in importlib through _os? On Mon, Jan 30, 2012 at 16:11, antoine.pitrou <python-checkins(a)python.org>wrote: > http://hg.python.org/cpython/rev/80ddbd822227 > changeset: 74689:80ddbd822227 > user: Antoine Pitrou <solipsis(a)pitrou.net> > date: Mon Jan 30 22:08:52 2012 +0100 > summary: > Issue #8828: Add new function os.replace(), for cross-platform renaming > with overwriting. > > files: > Doc/library/os.… [View More]rst | 18 +++++++++- > Lib/test/test_os.py | 12 ++++++ > Misc/NEWS | 3 + > Modules/posixmodule.c | 55 +++++++++++++++++++++--------- > 4 files changed, 69 insertions(+), 19 deletions(-) > > > diff --git a/Doc/library/os.rst b/Doc/library/os.rst > --- a/Doc/library/os.rst > +++ b/Doc/library/os.rst > @@ -1889,8 +1889,9 @@ > Unix flavors if *src* and *dst* are on different filesystems. If > successful, > the renaming will be an atomic operation (this is a POSIX requirement). > On > Windows, if *dst* already exists, :exc:`OSError` will be raised even if > it is a > - file; there may be no way to implement an atomic rename when *dst* > names an > - existing file. > + file. > + > + If you want cross-platform overwriting of the destination, use > :func:`replace`. > > Availability: Unix, Windows. > > @@ -1908,6 +1909,19 @@ > permissions needed to remove the leaf directory or file. > > > +.. function:: replace(src, dst) > + > + Rename the file or directory *src* to *dst*. If *dst* is a directory, > + :exc:`OSError` will be raised. If *dst* exists and is a file, it will > + be replaced silently if the user has permission. The operation may > fail > + if *src* and *dst* are on different filesystems. If successful, > + the renaming will be an atomic operation (this is a POSIX requirement). > + > + Availability: Unix, Windows > + > + .. versionadded:: 3.3 > + > + > .. function:: rmdir(path) > > Remove (delete) the directory *path*. Only works when the directory is > diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py > --- a/Lib/test/test_os.py > +++ b/Lib/test/test_os.py > @@ -129,6 +129,18 @@ > self.fdopen_helper('r') > self.fdopen_helper('r', 100) > > + def test_replace(self): > + TESTFN2 = support.TESTFN + ".2" > + with open(support.TESTFN, 'w') as f: > + f.write("1") > + with open(TESTFN2, 'w') as f: > + f.write("2") > + self.addCleanup(os.unlink, TESTFN2) > + os.replace(support.TESTFN, TESTFN2) > + self.assertRaises(FileNotFoundError, os.stat, support.TESTFN) > + with open(TESTFN2, 'r') as f: > + self.assertEqual(f.read(), "1") > + > > # Test attributes on return values from os.*stat* family. > class StatAttributeTests(unittest.TestCase): > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -463,6 +463,9 @@ > Library > ------- > > +- Issue #8828: Add new function os.replace(), for cross-platform renaming > + with overwriting. > + > - Issue #13848: open() and the FileIO constructor now check for NUL > characters in the file name. Patch by Hynek Schlawack. > > diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c > --- a/Modules/posixmodule.c > +++ b/Modules/posixmodule.c > @@ -3280,17 +3280,16 @@ > #endif /* HAVE_SETPRIORITY */ > > > -PyDoc_STRVAR(posix_rename__doc__, > -"rename(old, new)\n\n\ > -Rename a file or directory."); > - > -static PyObject * > -posix_rename(PyObject *self, PyObject *args) > +static PyObject * > +internal_rename(PyObject *self, PyObject *args, int is_replace) > { > #ifdef MS_WINDOWS > PyObject *src, *dst; > BOOL result; > - if (PyArg_ParseTuple(args, "UU:rename", &src, &dst)) > + int flags = is_replace ? MOVEFILE_REPLACE_EXISTING : 0; > + if (PyArg_ParseTuple(args, > + is_replace ? "UU:replace" : "UU:rename", > + &src, &dst)) > { > wchar_t *wsrc, *wdst; > > @@ -3301,16 +3300,17 @@ > if (wdst == NULL) > return NULL; > Py_BEGIN_ALLOW_THREADS > - result = MoveFileW(wsrc, wdst); > + result = MoveFileExW(wsrc, wdst, flags); > Py_END_ALLOW_THREADS > if (!result) > - return win32_error("rename", NULL); > + return win32_error(is_replace ? "replace" : "rename", NULL); > Py_INCREF(Py_None); > return Py_None; > } > else { > PyErr_Clear(); > - if (!PyArg_ParseTuple(args, "O&O&:rename", > + if (!PyArg_ParseTuple(args, > + is_replace ? "O&O&:replace" : "O&O&:rename", > PyUnicode_FSConverter, &src, > PyUnicode_FSConverter, &dst)) > return NULL; > @@ -3319,15 +3319,15 @@ > goto error; > > Py_BEGIN_ALLOW_THREADS > - result = MoveFileA(PyBytes_AS_STRING(src), > - PyBytes_AS_STRING(dst)); > + result = MoveFileExA(PyBytes_AS_STRING(src), > + PyBytes_AS_STRING(dst), flags); > Py_END_ALLOW_THREADS > > Py_XDECREF(src); > Py_XDECREF(dst); > > if (!result) > - return win32_error("rename", NULL); > + return win32_error(is_replace ? "replace" : "rename", NULL); > Py_INCREF(Py_None); > return Py_None; > > @@ -3337,10 +3337,30 @@ > return NULL; > } > #else > - return posix_2str(args, "O&O&:rename", rename); > -#endif > -} > - > + return posix_2str(args, > + is_replace ? "O&O&:replace" : "O&O&:rename", > rename); > +#endif > +} > + > +PyDoc_STRVAR(posix_rename__doc__, > +"rename(old, new)\n\n\ > +Rename a file or directory."); > + > +static PyObject * > +posix_rename(PyObject *self, PyObject *args) > +{ > + return internal_rename(self, args, 0); > +} > + > +PyDoc_STRVAR(posix_replace__doc__, > +"replace(old, new)\n\n\ > +Rename a file or directory, overwriting the destination."); > + > +static PyObject * > +posix_replace(PyObject *self, PyObject *args) > +{ > + return internal_rename(self, args, 1); > +} > > PyDoc_STRVAR(posix_rmdir__doc__, > "rmdir(path)\n\n\ > @@ -10555,6 +10575,7 @@ > {"readlink", win_readlink, METH_VARARGS, win_readlink__doc__}, > #endif /* !defined(HAVE_READLINK) && defined(MS_WINDOWS) */ > {"rename", posix_rename, METH_VARARGS, posix_rename__doc__}, > + {"replace", posix_replace, METH_VARARGS, > posix_replace__doc__}, > {"rmdir", posix_rmdir, METH_VARARGS, posix_rmdir__doc__}, > {"stat", posix_stat, METH_VARARGS, posix_stat__doc__}, > {"stat_float_times", stat_float_times, METH_VARARGS, > stat_float_times__doc__}, > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > > [View Less]

2 1

Release cycle question
by Ivano Jan. 30, 2012

Jan. 30, 2012

Hello everyone. I'm writing to ask if Python uses a "fixed" release time or if it depends strongly on something else. In example, Blender does and since I'm diving into Python because I would like to extend it, I would like to know if my work will have a default lifetime or not. By the way, Python 3 changed the game AFAIK, will another major change come short? Thanks in advance for any help. Bye, Ivano.

2 1

PEP for allowing 'raise NewException from None'
by Ethan Furman Jan. 30, 2012

Jan. 30, 2012

PEP: XXX Title: Interpreter support for concurrent programming Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman <ethan(a)stoneleaf.us> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 26-Jan-2012 Python-Version: 3.3 Post-History: Abstract ======== One of the open issues from PEP 3134 is suppressing context: currently there is no way to do it. This PEP proposes one. Motivation ========== There are two basic ways to generate exceptions: 1) Python … [View More]

9 20