Mailman 3 July 2011 - Python-Dev

Re: [Python-Dev] [Python-checkins] cpython: Fix closes Issue11281 - smtplib.STMP gets source_address parameter, which adds
by Ezio Melotti 01 Aug '11

01 Aug '11

Hi, On 30/07/2011 5.58, senthil.kumaran wrote: > http://hg.python.org/cpython/rev/26839edf3cc1 > changeset: 71617:26839edf3cc1 > parent: 71613:018e14a46454 > user: Senthil Kumaran<senthil(a)uthcode.com> > date: Sat Jul 30 10:56:50 2011 +0800 > summary: > Fix closes Issue11281 - smtplib.STMP gets source_address parameter, which adds the ability to bind to specific source address on a machine with multiple interfaces. Patch by Paulo Scardine. > > files: > Doc/library/smtplib.rst | 33 +++++++++++++++++----- > Lib/smtplib.py | 40 ++++++++++++++++++--------- > Lib/test/mock_socket.py | 3 +- > Lib/test/test_smtplib.py | 17 +++++++++++ > Misc/NEWS | 4 ++ > 5 files changed, 75 insertions(+), 22 deletions(-) > > > diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst > --- a/Doc/library/smtplib.rst > +++ b/Doc/library/smtplib.rst > @@ -20,7 +20,7 @@ > Protocol) and :rfc:`1869` (SMTP Service Extensions). > > > -.. class:: SMTP(host='', port=0, local_hostname=None[, timeout]) > +.. class:: SMTP(host='', port=0, local_hostname=None[, timeout], source_address=None) The "[, timeout]" now looks weird there, and it would be better to convert it to ", timeout=..." to match the other args. However I don't know what the value should be, since the real value is socket._GLOBAL_DEFAULT_TIMEOUT (i.e. object()) and I don't think it's a good idea to expose that. Maybe "None" can be used instead? > > A :class:`SMTP` instance encapsulates an SMTP connection. It has methods > that support a full repertoire of SMTP and ESMTP operations. If the optional > @@ -29,7 +29,12 @@ > raised if the specified host doesn't respond correctly. The optional > *timeout* parameter specifies a timeout in seconds for blocking operations > like the connection attempt (if not specified, the global default timeout > - setting will be used). > + setting will be used). The optional source_address parameter allows to bind to some > + specific source address in a machine with multiple network interfaces, > + and/or to some specific source tcp port. It takes a 2-tuple (host, port), I think TCP should be uppercase. > + for the socket to bind to as its source address before connecting. If > + ommited (or if host or port are '' and/or 0 respectively) the OS default s/ommited/omitted/ and s/''/``''``/ > + behavior will be used. > > For normal use, you should only require the initialization/connect, > :meth:`sendmail`, and :meth:`quit` methods. An example is included below. > @@ -48,8 +53,10 @@ > .. versionchanged:: 3.3 > Support for the :keyword:`with` statement was added. > > + .. versionadded:: 3.3 > + source_address parameter. I think the convention is to use "versionadded" when the function/method/class/etc has been added, and "versionchanged" for all the changes, including new arguments. > -.. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, certfile=None[, timeout], context=None) > +.. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, certfile=None[, timeout], context=None, source_address=None) Ditto for "[, timeout]" and the typos/markup below. > A :class:`SMTP_SSL` instance behaves exactly the same as instances of > :class:`SMTP`. :class:`SMTP_SSL` should be used for situations where SSL is > @@ -62,18 +69,28 @@ > keyfile and certfile must be None. The optional *timeout* > parameter specifies a timeout in seconds for blocking operations like the > connection attempt (if not specified, the global default timeout setting > - will be used). > + will be used). The optional source_address parameter allows to bind to some > + specific source address in a machine with multiple network interfaces, > + and/or to some specific source tcp port. It takes a 2-tuple (host, port), > + for the socket to bind to as its source address before connecting. If > + ommited (or if host or port are '' and/or 0 respectively) the OS default > + behavior will be used. > > .. versionchanged:: 3.3 > *context* was added. > > + .. versionadded:: 3.3 > + source_address parameter. > > -.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None) > + > +.. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None, source_address=None) > > The LMTP protocol, which is very similar to ESMTP, is heavily based on the > - standard SMTP client. It's common to use Unix sockets for LMTP, so our :meth:`connect` > - method must support that as well as a regular host:port server. To specify a > - Unix socket, you must use an absolute path for *host*, starting with a '/'. > + standard SMTP client. It's common to use Unix sockets for LMTP, so our > + :meth:`connect` method must support that as well as a regular host:port > + server. The optional parameters local_hostname and source_address has the s/has/have/? Also I prefer 'arguments' rather than 'parameters', the smtplib doc uses both, but 'arguments' seems to be used more. > + same meaning as that of SMTP client.To specify a Unix socket, you must use Missing space after the '.' (there should be two spaces, but here a single space is used consistently so it's fine). > + an absolute path for *host*, starting with a '/'. > > Authentication is supported, using the regular SMTP mechanism. When using a Unix > socket, LMTP generally don't support or require any authentication, but your > diff --git a/Lib/smtplib.py b/Lib/smtplib.py > --- a/Lib/smtplib.py > +++ b/Lib/smtplib.py > @@ -215,7 +215,8 @@ > [...] Best Regards, Ezio Melotti

4 3

Re: [Python-Dev] [Python-checkins] cpython: Issue #11651: Improve Makefile test targets.
by Nick Coghlan 31 Jul '11

31 Jul '11

On Sun, Jul 31, 2011 at 9:09 AM, nadeem.vawda <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/b76684d62a8d > changeset: 71645:b76684d62a8d > user: Nadeem Vawda <nadeem.vawda(a)gmail.com> > date: Sun Jul 31 01:09:04 2011 +0200 > summary: > Issue #11651: Improve Makefile test targets. > > - Use -j0 option by default > - Remove duplicate test run in "make test" and "make testuniversal" That seems very questionable - the rationale for running the test suite twice by default to ensure PYC generation is working correctly still holds. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 4

Re: [Python-Dev] [Python-checkins] cpython: Modernize modulefinder module and tests a bit.
by Ezio Melotti 31 Jul '11

31 Jul '11

Hi, On 29/07/2011 15.35, eric.araujo wrote: > http://hg.python.org/cpython/rev/1521d9837d16 > changeset: 71569:1521d9837d16 > user: Éric Araujo<merwok(a)netwok.org> > date: Thu Jul 28 23:35:29 2011 +0200 > summary: > Modernize modulefinder module and tests a bit. > > The tests don’t use an internal distutils function anymore, and use > regular assertEqual with sorted lists instead of a convoluted manual > diff. > > files: > Lib/modulefinder.py | 15 ++---- > Lib/test/test_modulefinder.py | 48 +++++++++++----------- > 2 files changed, 31 insertions(+), 32 deletions(-) > > > diff --git a/Lib/modulefinder.py b/Lib/modulefinder.py > --- a/Lib/modulefinder.py > +++ b/Lib/modulefinder.py > @@ -1,6 +1,5 @@ > """Find modules used by a script, using introspection.""" > > -from __future__ import generators > import dis > import imp > import marshal > @@ -9,8 +8,6 @@ > import types > import struct > > -READ_MODE = "rU" > - > # XXX Clean up once str8's cstor matches bytes. > LOAD_CONST = bytes([dis.opname.index('LOAD_CONST')]) > IMPORT_NAME = bytes([dis.opname.index('IMPORT_NAME')]) > @@ -29,8 +26,7 @@ > > # A Public interface > def AddPackagePath(packagename, path): > - paths = packagePathMap.get(packagename, []) > - paths.append(path) > + paths = packagePathMap.setdefault(packagename, []).append(path) I'm assuming that packagePathMap is a dict that might contain or not a *packagename* key that maps to a list of paths. Now, unless I'm missing something, the old code assigned to *paths* the list of paths or [] if it wasn't there, and then appended *path* to it. AFAICS, the new code introduced two changes: 1) the packagename key is added to the dict if it was missing -- and this seems reasonable; 2) append is now on the same line, it returns None, and None is assigned to *paths* -- and this seems wrong; > packagePathMap[packagename] = paths Also this is not necessary anymore if you use setdefault. > replacePackageMap = {} > @@ -106,14 +102,14 @@ > > [...] Best Regards, Ezio Melotti

1 0

Re: [Python-Dev] cpython: we can call singleton types now
by Georg Brandl 31 Jul '11

31 Jul '11

On 07/30/11 17:03, benjamin.peterson wrote: > http://hg.python.org/cpython/rev/4a07b772f0e0 > changeset: 71639:4a07b772f0e0 > user: Benjamin Peterson <benjamin(a)python.org> > date: Sat Jul 30 10:03:09 2011 -0500 > summary: > we can call singleton types now > > files: > Doc/library/stdtypes.rst | 8 +++++--- > 1 files changed, 5 insertions(+), 3 deletions(-) > > > diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst > --- a/Doc/library/stdtypes.rst > +++ b/Doc/library/stdtypes.rst > @@ -2706,7 +2706,7 @@ > > This object is returned by functions that don't explicitly return a value. It > supports no special operations. There is exactly one null object, named > -``None`` (a built-in name). > +``None`` (a built-in name). Calling ``type(None)`` produces the same singleton. I know this is technically correct, but it will look like you mean "calling type with None as argument" (same for the other singletons). Probably better to say something like "``type(None)()`` produces ...". Georg

1 0

Re: [Python-Dev] cpython: make the types of None and Ellipsis callable
by Georg Brandl 31 Jul '11

31 Jul '11

Am 30.07.2011 01:20, schrieb benjamin.peterson: > http://hg.python.org/cpython/rev/84c3be27b4c7 > changeset: 71614:84c3be27b4c7 > parent: 71611:a6afd26caa8a > user: Benjamin Peterson <benjamin(a)python.org> > date: Fri Jul 29 18:19:43 2011 -0500 > summary: > make the types of None and Ellipsis callable > > files: > Lib/test/test_builtin.py | 7 +++++ > Misc/NEWS | 3 ++ > Objects/object.c | 34 ++++++++++++++++++++++++++++ > Objects/sliceobject.c | 29 +++++++++++++++++++++++ > 4 files changed, 73 insertions(+), 0 deletions(-) Shouldn't there be a matching Doc change somewhere? Georg

3 3

Re: [Python-Dev] [Python-checkins] cpython: Issue 12647: Add __bool__() method to the None object.
by Brett Cannon 31 Jul '11

31 Jul '11

On Thu, Jul 28, 2011 at 09:55, raymond.hettinger <python-checkins(a)python.org > wrote: > http://hg.python.org/cpython/rev/ccce01988603 > changeset: 71542:ccce01988603 > user: Raymond Hettinger <python(a)rcn.com> > date: Thu Jul 28 09:55:13 2011 -0700 > summary: > Issue 12647: Add __bool__() method to the None object. > > files: > Lib/test/test_descr.py | 3 - > Misc/NEWS | 4 ++ > Objects/object.c | 46 ++++++++++++++++++++++++++++- > 3 files changed, 48 insertions(+), 5 deletions(-) > > > diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py > --- a/Lib/test/test_descr.py > +++ b/Lib/test/test_descr.py > @@ -2068,9 +2068,6 @@ > # Two essentially featureless objects, just inheriting stuff from > # object. > self.assertEqual(dir(NotImplemented), dir(Ellipsis)) > - if support.check_impl_detail(): > - # None differs in PyPy: it has a __nonzero__ > - self.assertEqual(dir(None), dir(Ellipsis)) > > Wasn't this change only in 3.3 where __nonzero__ doesn't exist? So when PyPy eventually supports Python 3 they will have to update to support __bool__ on None but this test won't exercise that for them. IOW I think the guard is wrong and should go. -Brett > # Nasty test case for proxied objects > class Wrapper(object): > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -13,6 +13,10 @@ > - Verify the types of AST strings and identifiers provided by the user > before > compiling them. > > +- Issue #12647: The None object now has a __bool__() method that returns > False. > + Formerly, bool(None) returned False only because of special case logic > + in PyObject_IsTrue(). > + > - Issue #12579: str.format_map() now raises a ValueError if used on a > format string that contains positional fields. Initial patch by > Julian Berman. > diff --git a/Objects/object.c b/Objects/object.c > --- a/Objects/object.c > +++ b/Objects/object.c > @@ -1255,7 +1255,7 @@ > } > > /* > -None is as a non-NULL undefined value. > +None is a non-NULL undefined value. > There is (and should be!) no way to create other objects of this type, > so there is exactly one (which is indestructible, by the way). > */ > @@ -1277,6 +1277,48 @@ > Py_FatalError("deallocating None"); > } > > +static int > +none_bool(PyObject *v) > +{ > + return 0; > +} > + > +static PyNumberMethods none_as_number = { > + 0, /* nb_add */ > + 0, /* nb_subtract */ > + 0, /* nb_multiply */ > + 0, /* nb_remainder */ > + 0, /* nb_divmod */ > + 0, /* nb_power */ > + 0, /* nb_negative */ > + 0, /* nb_positive */ > + 0, /* nb_absolute */ > + (inquiry)none_bool, /* nb_bool */ > + 0, /* nb_invert */ > + 0, /* nb_lshift */ > + 0, /* nb_rshift */ > + 0, /* nb_and */ > + 0, /* nb_xor */ > + 0, /* nb_or */ > + 0, /* nb_int */ > + 0, /* nb_reserved */ > + 0, /* nb_float */ > + 0, /* nb_inplace_add */ > + 0, /* nb_inplace_subtract */ > + 0, /* nb_inplace_multiply */ > + 0, /* nb_inplace_remainder */ > + 0, /* nb_inplace_power */ > + 0, /* nb_inplace_lshift */ > + 0, /* nb_inplace_rshift */ > + 0, /* nb_inplace_and */ > + 0, /* nb_inplace_xor */ > + 0, /* nb_inplace_or */ > + 0, /* nb_floor_divide */ > + 0, /* nb_true_divide */ > + 0, /* nb_inplace_floor_divide */ > + 0, /* nb_inplace_true_divide */ > + 0, /* nb_index */ > +}; > > static PyTypeObject PyNone_Type = { > PyVarObject_HEAD_INIT(&PyType_Type, 0) > @@ -1289,7 +1331,7 @@ > 0, /*tp_setattr*/ > 0, /*tp_reserved*/ > none_repr, /*tp_repr*/ > - 0, /*tp_as_number*/ > + &none_as_number, /*tp_as_number*/ > 0, /*tp_as_sequence*/ > 0, /*tp_as_mapping*/ > 0, /*tp_hash */ > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > >

2 1

Convention on functions that shadow existing stdlib functions
by Eli Bendersky 31 Jul '11

31 Jul '11

Some background: I'm working (on and off) on issue 11015 - documenting the public functions in test.support Some of the functions in test.support (for example unlink, rmtree) simply shadow existing & popular stdlib functions, with the aim of swallowing the exceptions these may throw. This is confusing, IMHO. For example, grepping 'unlink' on Lib/test/test_*.py files doesn't say much about which unlink is being used. A couple of options to handle this are: 1. Remove these functions altogether, trying to use existing constructs (such as the ignore_errors parameter in rmtree). 2. Adapt a naming convention for such functions, for instance rmtree_silent and unlink_silent (or a similar convention, if one exists) Opinions? Eli

13 60

Draft PEP: "Simplified Package Layout and Partitioning"
by P.J. Eby 30 Jul '11

30 Jul '11

So, over on the Import-SIG, we were talking about the implementation and terminology for PEP 382, and it became increasingly obvious that things were, well, not entirely okay in the "implementation is easy to explain" department. Anyway, to make a long story short, we came up with an alternative implementation plan that actually solves some other problems besides the one that PEP 382 sets out to solve, and whose implementation a bit is easier to explain. (In fact, for users coming from various other languages, it hardly needs any explanation at all.) However, for long-time users of Python, the approach may require a bit more justification, which is why roughly 2/3rds of the PEP consists of a detailed rationale, specification overview, rejected alternatives, and backwards-compatibility discussion... which is still a lot less verbiage than reading through the lengthy Import-SIG threads that led up to the proposal. ;-) (The remaining 1/3rd of the PEP is the short, sweet, and easy-to-explain implementation detail.) Anyway, the PEP has already been discussed on the Import-SIG, and is proposed as an alternative to PEP 382 ("Namespace packages"). We expect, however, that many people will be interested in it for reasons having little to do with the namespace packaging use case. So, we would like to submit this for discussion, hole-finding, and eventual Pronouncement. As Barry put it, "I think it's certainly worthy of posting to python-dev to see if anybody else can shoot holes in it, or come up with useful solutions to open questions. I'll be very interested to see Guido's reaction to it. :)" So, without further ado, here it is: PEP: XXX Title: Simplified Package Layout and Partitioning Version: $Revision$ Last-Modified: $Date$ Author: P.J. Eby Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jul-2011 Python-Version: 3.3 Post-History: Replaces: 382 Abstract ======== This PEP proposes an enhancement to Python's package importing to: * Surprise users of other languages less, * Make it easier to convert a module into a package, and * Support dividing packages into separately installed components (ala "namespace packages", as described in PEP 382) The proposed enhancements do not change the semantics of any currently-importable directory layouts, but make it possible for packages to use a simplified directory layout (that is not importable currently). However, the proposed changes do NOT add any performance overhead to the importing of existing modules or packages, and performance for the new directory layout should be about the same as that of previous "namespace package" solutions (such as ``pkgutil.extend_path()``). The Problem =========== .. epigraph:: "Most packages are like modules. Their contents are highly interdependent and can't be pulled apart. [However,] some packages exist to provide a separate namespace. ... It should be possible to distribute sub-packages or submodules of these [namespace packages] independently." -- Jim Fulton, shortly before the release of Python 2.3 [1]_ When new users come to Python from other languages, they are often confused by Python's packaging semantics. At Google, for example, Guido received complaints from "a large crowd with pitchforks" [2]_ that the requirement for packages to contain an ``__init__`` module was a "misfeature", and should be dropped. In addition, users coming from languages like Java or Perl are sometimes confused by a difference in Python's import path searching. In most other languages that have a similar path mechanism to Python's ``sys.path``, a package is merely a namespace that contains modules or classes, and can thus be spread across multiple directories in the language's path. In Perl, for instance, a ``Foo::Bar`` module will be searched for in ``Foo/`` subdirectories all along the module include path, not just in the first such subdirectory found. Worse, this is not just a problem for new users: it prevents *anyone* from easily splitting a package into separately-installable components. In Perl terms, it would be as if every possible ``Net::`` module on CPAN had to be bundled up and shipped in a single tarball! For that reason, various workarounds for this latter limitation exist, circulated under the term "namespace packages". The Python standard library has provided one such workaround since Python 2.3 (via the ``pkgutil.extend_path()`` function), and the "setuptools" package provides another (via ``pkg_resources.declare_namespace()``). The workarounds themselves, however, fall prey to a *third* issue with Python's way of laying out packages in the filesystem. Because a package *must* contain an ``__init__`` module, any attempt to distribute modules for that package must necessarily include that ``__init__`` module, if those modules are to be importable. However, the very fact that each distribution of modules for a package must contain this (duplicated) ``__init__`` module, means that OS vendors who package up these module distributions must somehow handle the conflict caused by several module distributions installing that ``__init__`` module to the same location in the filesystem. This led to the proposing of PEP 382 ("Namespace Packages") - a way to signal to Python's import machinery that a directory was importable, using unique filenames per module distribution. However, there was more than one downside to this approach. Performance for all import operations would be affected, and the process of designating a package became even more complex. New terminology had to be invented to explain the solution, and so on. As terminology discussions continued on the Import-SIG, it soon became apparent that the main reason it was so difficult to explain the concepts related to "namespace packages" was because Python's current way of handling packages is somewhat underpowered, when compared to other languages. That is, in other popular languages with package systems, no special term is needed to describe "namespace packages", because *all* packages generally behave in the desired fashion. Rather than being an isolated single directory with a special marker module (as in Python), packages in other languages are typically just the *union* of appropriately-named directories across the *entire* import or inclusion path. In Perl, for example, the module ``Foo`` is always found in a ``Foo.pm`` file, and a module ``Foo::Bar`` is always found in a ``Foo/Bar.pm`` file. (In other words, there is One Obvious Way to find the location of a particular module.) This is because Perl considers a module to be *different* from a package: the package is purely a *namespace* in which other modules may reside, and is only *coincidentally* the name of a module as well. In current versions of Python, however, the module and the package are more tightly bound together. ``Foo`` is always a module -- whether it is found in ``Foo.py`` or ``Foo/__init__.py`` -- and it is tightly linked to its submodules (if any), which *must* reside in the exact same directory where the ``__init__.py`` was found. On the positive side, this design choice means that a package is quite self-contained, and can be installed, copied, etc. as a unit just by performing an operation on the package's root directory. On the negative side, however, it is non-intuitive for beginners, and requires a more complex step to turn a module into a package. If ``Foo`` begins its life as ``Foo.py``, then it must be moved and renamed to ``Foo/__init__.py``. Conversely, if you intend to create a ``Foo.Bar`` module from the start, but have no particular module contents to put in ``Foo`` itself, then you have to create an empty and seemingly-irrelevant ``Foo/__init__.py`` file, just so that ``Foo.Bar`` can be imported. (And these issues don't just confuse newcomers to the language, either: they annoy many experienced developers as well.) So, after some discussion on the Import-SIG, this PEP was created as an alternative to PEP \382, in an attempt to solve *all* of the above problems, not just the "namespace package" use cases. And, as a delightful side effect, the solution proposed in this PEP does not affect the import performance of ordinary modules or self-contained (i.e. ``__init__``-based) packages. The Solution ============ In the past, various proposals have been made to allow more intuitive approaches to package directory layout. However, most of them failed because of an apparent backward-compatibility problem. That is, if the requirement for an ``__init__`` module were simply dropped, it would open up the possibility for a directory named, say, ``string`` on ``sys.path``, to block importing of the standard library ``string`` module. Paradoxically, however, the failure of this approach does *not* arise from the elimination of the ``__init__`` requirement! Rather, the failure arises because the underlying approach takes for granted that a package is just ONE thing, instead of two. In truth, a package comprises two separate, but related entities: a module (with its own, optional contents), and a *namespace* where *other* modules or packages can be found. In current versions of Python, however, the module part (found in ``__init__``) and the namespace for submodule imports (represented by the ``__path__`` attribute) are both initialized at the same time, when the package is first imported. And, if you assume this is the *only* way to initialize these two things, then there is no way to drop the need for an ``__init__`` module, while still being backwards-compatible with existing directory layouts. After all, as soon as you encounter a directory on ``sys.path`` matching the desired name, that means you've "found" the package, and must stop searching, right? Well, not quite. A Thought Experiment -------------------- Let's hop into the time machine for a moment, and pretend we're back in the early 1990s, shortly before Python packages and ``__init__.py`` have been invented. But, imagine that we *are* familiar with Perl-like package imports, and we want to implement a similar system in Python. We'd still have Python's *module* imports to build on, so we could certainly conceive of having ``Foo.py`` as a parent ``Foo`` module for a ``Foo`` package. But how would we implement submodule and subpackage imports? Well, if we didn't have the idea of ``__path__`` attributes yet, we'd probably just search ``sys.path`` looking for ``Foo/Bar.py``. But we'd *only* do it when someone actually tried to *import* ``Foo.Bar``. NOT when they imported ``Foo``. And *that* lets us get rid of the backwards-compatibility problem of dropping the ``__init__`` requirement, back here in 2011. How? Well, when we ``import Foo``, we're not even *looking* for ``Foo/`` directories on ``sys.path``, because we don't *care* yet. The only point at which we care, is the point when somebody tries to actually import a submodule or subpackage of ``Foo``. That means that if ``Foo`` is a standard library module (for example), and I happen to have a ``Foo`` directory on ``sys.path`` (without an ``__init__.py``, of course), then *nothing breaks*. The ``Foo`` module is still just a module, and it's still imported normally. Self-Contained vs. "Virtual" Packages ------------------------------------- Of course, in today's Python, trying to ``import Foo.Bar`` will fail if ``Foo`` is just a ``Foo.py`` module (and thus lacks a ``__path__`` attribute). So, this PEP proposes to *dynamically* create a ``__path__``, in the case where one is missing. That is, if I try to ``import Foo.Bar`` the proposed change to the import machinery will notice that the ``Foo`` module lacks a ``__path__``, and will therefore try to *build* one before proceeding. And it will do this by making a list of all the existing ``Foo/`` subdirectories of the directories listed in ``sys.path``. If the list is empty, the import will fail with ``ImportError``, just like today. But if the list is *not* empty, then it is saved in a new ``Foo.__path__`` attribute, making the module a "virtual package". That is, because it now has a valid ``__path__``, we can proceed to import submodules or subpackages in the normal way. Now, notice that this change does not affect "classic", self-contained packages that have an ``__init__`` module in them. Such packages already *have* a ``__path__`` attribute (initialized at import time) so the import machinery won't try to create another one later. This means that (for example) the standard library ``email`` package will not be affected in any way by you having a bunch of unrelated directories named ``email`` on ``sys.path``. (Even if they contain ``*.py`` files.) But it *does* mean that if you want to turn your ``Foo`` module into a ``Foo`` package, all you have to do is add a ``Foo/`` directory somewhere on ``sys.path``, and start adding modules to it. But what if you only want a "namespace package"? That is, a package that is *only* a namespace for various separately-distributed submodules and subpackages? For example, if you're Zope Corporation, distributing dozens of separate tools like ``zc.buildout``, each in packages under the ``zc`` namespace, you don't want to have to make and include an empty ``zc.py`` in every tool you ship. (And, if you're a Linux or other OS vendor, you don't want to deal with the package installation conflicts created by trying to install ten copies of ``zc.py`` to the same location!) No problem. All we have to do is make one more minor tweak to the import process: if the "classic" import process fails to find a self-contained module or package (e.g., if ``import zc`` fails to find a ``zc.py`` or ``zc/__init__.py``), then we once more try to build a ``__path__`` by searching for all the ``zc/`` directories on ``sys.path``, and putting them in a list. If this list is empty, we raise ``ImportError``. But if it's non-empty, we create an empty ``zc`` module, and put the list in ``zc.__path__``. Congratulations: ``zc`` is now a namespace-only, "pure virtual" package! It has no module contents, but you can still import submodules and subpackages from it, regardless of where they're located on ``sys.path``. (By the way, both of these additions to the import protocol (i.e. the dynamically-added ``__path__``, and dynamically-created modules) apply recursively to child packages, using the parent package's ``__path__`` in place of ``sys.path`` as a basis for generating a child ``__path__``. This means that self-contained and virtual packages can contain each other without limitation, with the caveat that if you put a virtual package inside a self-contained one, it's gonna have a really short ``__path__``!) Backwards Compatibility and Performance --------------------------------------- Notice that these two changes *only* affect import operations that today would result in ``ImportError``. As a result, the performance of imports that do not involve virtual packages is unaffected, and potential backward compatibility issues are very restricted. Today, if you try to import submodules or subpackages from a module with no ``__path__``, it's an immediate error. And of course, if you don't have a ``zc.py`` or ``zc/__init__.py`` somewhere on ``sys.path`` today, ``import zc`` would likewise fail. Thus, the only potential backwards-compatibility issues are: 1. Tools that expect package directories to have an ``__init__`` module, that expect directories without an ``__init__`` module to be unimportable, or that expect ``__path__`` attributes to be static, will not recognize virtual packages as packages. (In practice, this just means that tools will need updating to support virtual packages, e.g. by using ``pkgutil.walk_modules()`` instead of using hardcoded filesystem searches.) 2. Code that *expects* certain imports to fail may now do something unexpected. This should be fairly rare in practice, as most sane, non-test code does not import things that are expected not to exist! The biggest likely exception to the above would be when a piece of code tries to check whether some package is installed by importing it. If this is done *only* by importing a top-level module (i.e., not checking for a ``__version__`` or some other attribute), *and* there is a directory of the same name as the sought-for package on ``sys.path`` somewhere, *and* the package is not actually installed, then such code could *perhaps* be fooled into thinking a package is installed that really isn't. However, even in the rare case where all these conditions line up to happen at once, the failure is more likely to be annoying than damaging. In most cases, after all, the code will simply fail a little later on, when it actually tries to DO something with the imported (but empty) module. (And code that checks ``__version__`` attributes or for the presence of some desired function, class, or module in the package will not see a false positive result in the first place.) Meanwhile, tools that expect to locate packages and modules by walking a directory tree can be updated to use the existing ``pkgutil.walk_modules()`` API, and tools that need to inspect packages in memory should use the other APIs described in the `Standard Library Changes/Additions`_ section below. Specification ============= Two changes are made to the existing import process. First, the built-in ``__import__`` function must not raise an ``ImportError`` when importing a submodule of a module with no ``__path__``. Instead, it must attempt to *create* a ``__path__`` attribute for the parent module first, as described in `__path__ creation`_, below. Second, if searching ``sys.meta_path`` and ``sys.path`` (or a parent package ``__path__``) fails to find a module being imported, the import process must attempt to create a ``__path__`` attribute for the missing module. If the attempt succeeds, an empty module is created and its ``__path__`` is set. Otherwise, importing fails. In both of the above cases, if a non-empty ``__path__`` is created, the name of the module whose ``__path__`` was created is added to ``sys.virtual_packages`` -- an initially-empty ``set()`` of package names. (This way, code that extends ``sys.path`` at runtime can find out what virtual packages are currently imported, and thereby add any new subdirectories to those packages' ``__path__`` attributes. See `Standard Library Changes/Additions`_ below for more details.) Conversely, if an empty ``__path__`` results, an ``ImportError`` is immediately raised, and the module is not created or changed, nor is its name added to ``sys.virtual_packages``. ``__path__`` Creation --------------------- A virtual ``__path__`` is created by obtaining a PEP 302 "importer" object for each of the path entries found in ``sys.path`` (for a top-level module) or the parent ``__path__`` (for a submodule). (Note: because ``sys.meta_path`` importers are not associated with ``sys.path`` or ``__path__`` entry strings, such importers do *not* participate in this process.) Each importer is checked for a ``get_subpath()`` method, and if present, the method is called with the full name of the module/package the ``__path__`` is being constructed for. The return value is either a string representing a subdirectory for the requested package, or ``None`` if no such subdirectory exists. The strings returned by the importers are added to the ``__path__`` being built, in the same order as they are found. (``None`` values and missing ``get_subpath()`` methods are simply skipped.) In Python code, the algorithm would look something like this:: def get_virtual_path(modulename, parent_path=None): if parent_path is None: parent_path = sys.path path = [] for entry in parent_path: # Obtain a PEP 302 importer object - see pkgutil module importer = pkgutil.get_importer(entry) if hasattr(importer, 'get_subpath'): subpath = importer.get_subpath(modulename) if subpath is not None: path.append(subpath) return path And a function like this one should be exposed in the standard library as e.g. ``imp.get_virtual_path()``, so that people creating ``__import__`` replacements or ``sys.meta_path`` hooks can reuse it. Standard Library Changes/Additions ---------------------------------- The ``pkgutil`` module should be updated to handle this specification appropriately, including any necessary changes to ``extend_path()``, ``iter_modules()``, etc. Specifically the proposed changes and additions to ``pkgutil`` are: * A new ``extend_virtual_paths(path_entry)`` function, to extend existing, already-imported virtual packages' ``__path__`` attributes to include any portions found in a new ``sys.path`` entry. This function should be called by applications extending ``sys.path`` at runtime, e.g. when adding a plugin directory or an egg to the path. The implementation of this function does a simple top-down traversal of ``sys.virtual_packages``, and performs any necessary ``get_subpath()`` calls to identify what path entries need to be added to each package's ``__path__``, given that `path_entry` has been added to ``sys.path``. (Or, in the case of sub-packages, adding a derived subpath entry, based on their parent namespace's ``__path__``.) * A new ``iter_virtual_packages(parent='')`` function to allow top-down traversal of virtual packages in ``sys.virtual_packages``, by yielding the child virtual packages of `parent`. For example, calling ``iter_virtual_packages("zope")`` might yield ``zope.app`` and ``zope.products`` (if they are imported virtual packages listed in ``sys.virtual_packages``), but **not** ``zope.foo.bar``. (This function is needed to implement ``extend_virtual_paths()``, but is also potentially useful for other code that needs to inspect imported virtual packages.) * ``ImpImporter.iter_modules()`` should be changed to also detect and yield the names of modules found in virtual packages. In addition to the above changes, the ``zipimport`` importer should have its ``iter_modules()`` implementation similarly changed. (Note: current versions of Python implement this via a shim in ``pkgutil``, so technically this is also a change to ``pkgutil``.) Last, but not least, the ``imp`` module (or ``importlib``, if appropriate) should expose the algorithm described in the `__path__ creation`_ section above, as a ``get_virtual_path(modulename, parent_path=None)`` function, so that creators of ``__import__`` replacements can use it. Implementation Notes -------------------- For users, developers, and distributors of virtual packages: * While virtual packages are easy to set up and use, there is still a time and place for using self-contained packages. While it's not strictly necessary, adding an ``__init__`` module to your self-contained packages lets users of the package (and Python itself) know that *all* of the package's code will be found in that single subdirectory. In addition, it lets you define ``__all__``, expose a public API, provide a package-level docstring, and do other things that make more sense for a self-contained project than for a mere "namespace" package. * ``sys.virtual_packages`` is allowed to contain non-existent or not-yet-imported package names; code that uses its contents should not assume that every name in this set is also present in ``sys.modules`` or that importing the name will necessarily succeed. * If you are changing a currently self-contained package into a virtual one, it's important to note that you can no longer use its ``__file__`` attribute to locate data files stored in a package directory. Instead, you must search ``__path__`` or use the ``__file__`` of a submodule adjacent to the desired files, or of a self-contained subpackage that contains the desired files. (Note: this caveat is already true for existing users of "namespace packages" today. That is, it is an inherent result of being able to partition a package, that you must know *which* partition the desired data file lives in. We mention it here simply so that *new* users converting from self-contained to virtual packages will also be aware of it.) * XXX what is the __file__ of a "pure virtual" package? ``None``? Some arbitrary string? The path of the first directory with a trailing separator? No matter what we put, *some* code is going to break, but the last choice might allow some code to accidentally work. Is that good or bad? For those implementing PEP \302 importer objects: * Importers that support the ``iter_modules()`` method (used by ``pkgutil`` to locate importable modules and packages) and want to add virtual package support should modify their ``iter_modules()`` method so that it discovers and lists virtual packages as well as standard modules and packages. To do this, the importer should simply list all immediate subdirectory names in its jurisdiction that are valid Python identifiers. XXX This might list a lot of not-really-packages. Should we require importable contents to exist? If so, how deep do we search, and how do we prevent e.g. link loops, or traversing onto different filesystems, etc.? Ick. * "Meta" importers (i.e., importers placed on ``sys.meta_path``) do not need to implement ``get_subpath()``, because the method is only called on importers corresponding to ``sys.path`` entries and ``__path__`` entries. If a meta importer wishes to support virtual packages, it must do so entirely within its own ``find_module()`` implementation. Unfortunately, it is unlikely that any such implementation will be able to merge its package subpaths with those of other meta importers or ``sys.path`` importers, so the meaning of "supporting virtual packages" for a meta importer is currently undefined! (However, since the intended use case for meta importers is to replace Python's normal import process entirely for some subset of modules, and the number of such importers currently implemented is quite small, this seems unlikely to be a big issue in practice.) References ========== .. [1] "namespace" vs "module" packages (mailing list thread) (http://mail.zope.org/pipermail/zope3-dev/2002-December/004251.html) .. [2] "Dropping __init__.py requirement for subpackages" (http://mail.python.org/pipermail/python-dev/2006-April/064400.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

13 30

Re: [Python-Dev] cpython: Fix closes Issue11281 - smtplib.STMP gets source_address parameter, which adds
by Antoine Pitrou 30 Jul '11

30 Jul '11

Hi Senthil, > + if source_address: self.source_address = source_address Could you try to follow PEP 8? (I know PEP 8 is not always followed in old code, but there's no reason not to follow it in code that we add to the stdlib) > + SMTP.__init__(self, host, port, local_hostname = local_hostname, > + source_address = source_address) Ditto here (and other similar occurrences). > + def testSourceAddress(self): > + # connect > + smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3, > + source_address=('127.0.0.1', 19876)) > + self.assertEqual(smtp.source_address, ('127.0.0.1', 19876)) > + self.assertEqual(smtp.local_hostname, 'localhost') Unless this test is also using some kind of mock socket (it doesn't seem to), this can break as soon as port 19876 is already in use. There are utilities in test.support to help with this, such as bind_port() or (less reliable) find_unused_port(). Actually, this can be triggered quite easily by running the test a couple of times in parallel: $ ./python -m test -m testSourceAddress -Fv -j3 test_smtplib [...] [ 14/4] test_smtplib testSourceAddress (test.test_smtplib.GeneralTests) ... ok testSourceAddress (test.test_smtplib.DebuggingServerTests) ... ERROR ====================================================================== ERROR: testSourceAddress (test.test_smtplib.DebuggingServerTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/cpython/default/Lib/test/test_smtplib.py", line 220, in testSourceAddress source_address=('127.0.0.1', 19876)) File "/home/antoine/cpython/default/Lib/smtplib.py", line 238, in __init__ (code, msg) = self.connect(host, port) File "/home/antoine/cpython/default/Lib/smtplib.py", line 313, in connect self.sock = self._get_socket(host, port, self.timeout) File "/home/antoine/cpython/default/Lib/smtplib.py", line 287, in _get_socket self.source_address) File "/home/antoine/cpython/default/Lib/socket.py", line 407, in create_connection raise err File "/home/antoine/cpython/default/Lib/socket.py", line 397, in create_connection sock.bind(source_address) socket.error: [Errno 98] Address already in use ---------------------------------------------------------------------- > + print(dir(smtp)) Usually, we avoid printing anything in tests, except when support.verbose is True. Regards Antoine.

2 1

the history of tests being inside Lib/ in Python
by Eli Bendersky 30 Jul '11

30 Jul '11

The other thread had some claims (*) that made me wonder - why are the tests in Python kept in Lib/ at all? AFAIK, this is rather an unusual project structure. Tests usually have a top-level directory of their own, in parallel to Lib/, Doc/ and others. Some effects of this in other projects: * The tests usually aren't even installed. The user can run them during installation, but once it goes through, tests are not copied into /usr/whatever... * Tests naturally become "developer-domain", removed from the "user-domain". No sane user would even consider using code from inside the Tests/ directory and somehow expect it to keep working in later versions. In addition, tests are then usually documented in special "hacking guides" and "developer docs" instead of in the official documentation of the project. This mail can appear as if advocating the transfer of Lib/test into Tests/, but this is not my intention here. Honest :-) I'm just trying to understand the history and rationale behind this structure in the CPython project. Eli (*) I refer to this reasoning someone raised: "test.support is part of the tests" + "tests are part of stdlib" --> "test.support must be documented where the rest of stdlib is"

5 6