[Import-SIG] PEP 420: Implicit Namespace Packages

[PJ Eby]

On Sun, Apr 22, 2012 at 1:26 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Sun, Apr 22, 2012 at 11:06 AM, Eric V. Smith <eric at trueblade.com>
> wrote:
> > Furthermore, given how __path__ is built, by one-at-a-time remembering
> > the path entries that have a foo directory but no foo/__init__.py, I'm
> > not sure how you'd turn that into some auto-updating iterable.
>
> You just have to remember all your namespace packages somewhere and
> then use a list subclass that triggers a rescan whenever the contents
> change.
>

Not necessary if you set __path__ to an iterable that caches a tuple of its
parent package __path__ (or sys.path), and compares that against the
current value before iterating.  If it's changed, you walk the parent and
rescan, otherwise iterate over your cached value.  I posted a sketch to
Python-Dev the first time 402 discussion happened there.

The consequences of making namespace package __path__ iterable are less
problematic, I believe, than changing the type of sys.path: almost no code
manipulates __path__ as anything but an iterable, and code that does is
broken for namespace packages anyway, because accessing specific offsets
won't give you what you think you're looking for.  So you get noisy
breakage instead of quiet breakage in such cases (as would happen with
using lists for __path__).

If for some reason you want to explicitly change a namespace package's
__path__, you could just reset __path__ to list(__path__), and proceed from
there -- which is the recommended idiom for using extend_path, anyway.


Personally, I'm happier with the basic behaviour being that
> dynamically updating sys.path while the program is running can be a
> bit hit-or-miss in terms of what recognises the change.
>

pkg_resources supports dynamic updating today, so the idea here was to make
it possible to do away with that.  (It only supports updating if it's the
one doing the sys.path manipulation, however.)

I think there should be *some* blessed API(s) to force the updating,
though, even if it's not automatic or dynamic.  extend_path() really isn't
the right tool for the job.

The main argument in favor of automatic updating is that it more closely
matches naive expectations of users coming from other languages.  (Although
to be honest I'm not 100% certain that those other languages actually do
change their lookups that dynamically.)

Anyway, the sketch (using PEP 402's importer protocol; not updated for 420)
was something like:

class VirtualPath:
    __slots__ = ('__name__', '_parent', '_last_seen', '_path')

    def __init__(self, name, parent_path):
        self.__name__ = name
        self._parent = parent_path
        self._path = self._last_seen = ()

    def _fail(self, *args, **kw):
        raise TypeError(self.__name__+" is a virtual package")

    __getitem__ = __setitem__ = __delitem__ = append = extend = insert =
_fail

    def _calculate(self):
        with _ImportLockContext():
            parent = tuple(self._parent)
            if parent != self._last_seen:
                items = []
                name = self.__name__
                for entry in parent:
                    importer = get_importer(entry)
                    if hasattr(importer, 'get_subpath'):
                        item = importer.get_subpath(name)
                        if item is not None:
                            items.append(item)
                self._last_seen = parent
                self._path = tuple(items)
            return self._path

    def __iter__(self):
        return iter(self._calculate())

    def __len__(self):
        return len(self._calculate())

    def __repr__(self):
        return "VirtualPath" + repr((self.__name__, self._parent))

    def __contains__(self, item):
        return item in self._calculate()


Using these objects in place of lists for __path__ objects would then do
the trick.

(And of course, you'd want to change "Virtual" to "Namespace" throughout, I
suppose.  ;-) )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20120422/d5f4ae8a/attachment.html>