At 11:52 AM 7/21/2011 +1000, Nick Coghlan wrote:

Trying to change how packages are identified at the Python level makes PEP 382 sound positively appealing. __path__ needs to stay :)

In which case, it should be a list, not a sentinel. ;-)

Even better would be for these (and sys.path) to be list subclasses that did the right thing under the hood as Glenn suggested. Code that *replaces* rather than modifies these attributes would still potentially break virtual packages, but code that modifies them in place would do the right thing automatically. (Note that all code that manipulates sys.path and __path__ attributes requires explicit calls to correctly support current namespace package mechanisms, so this would actually be an improvement on the status quo rather than making anything worse).

I think the simplest thing, if we're keeping __path__ (and on reflection, I think we should), would be to simply call extend_virtual_paths() automatically on new path entries found in sys.path when an import is performed, relative to the previous value of sys.path. That is, we save an "old" copy of sys.path somewhere, and whenever __import__() is called (well, once it gets past checking if the target is already in sys.modules, anyway), it checks the current sys.path against it, and calls extend_virtual_paths() on any sys.path entries that weren't in the "old" sys.path. This is not the most efficient thing in the world, as it will cause a bunch of stat calls to happen against the new directories, in the middle of a possibly-entirely-unrelated import operation, but it would certainly address the issue in the Simplest Way That Could Possibly Work. A stricter (safer) version of the same thing would be one where we only update __path__ values that are unchanged since we created them, and rather than only appending new entries, we replace the __path__ with a newly-computed one. This version is safer because it avoids corner cases like "I imported foo.bar while foo.baz 1.1 was on my path, then I prepended a directory to sys.path that has foo.baz 1.2, but I still get foo.baz 1.1 when I import." But it loses in cases where people do direct __path__ manipulation. On the other hand, it's a lot easier to say "you break it, you bought it" where __path__ manipulation is concerned, so I'm actually pretty inclined towards using the strict version. Hey... here's a crazy idea. Suppose that a virtual package __path__ is a *tuple* instead of a list? Now, in order to change it, you *have* to replace it. And we can cache the tuple we initially set it to in sys.virtual_package_paths, so we can do an 'is' check before replacing it. Voila: __path__ still exists and is still a sequence for a virtual path, but you have to explicitly replace it if you want to do anything funky -- at which point you're responsible for maintaining it. I'm tempted to say, "well, why not use a list-subclass proxy, then?", but that means more work for no real difference. I just went through dozens of examples of __path__ usage (found via Google), and I found exactly two examples of code that modifies a __path__ that is not: 1. In the __init__.py whose __path__ it is (i.e., code that'll still have a list), or 2. Modifying the __path__ of an explicitly-named self-contained package that's part of the same distribution. The two examples are from Twisted, and Google AppEngine. In the Twisted case, it's some sort of namespace package-like plugin chicanery, and in the AppEngine case, well, I'm not sure what the heck it's doing, but it seems to be making sure that you can still import stuff that has the same name as stdlib stuff, or something. The Twisted case (and an apparent copy of the same code in a project called "flumotion") uses ihooks, though, so I'm not sure it'll even get executed for virtual packages. The Google case loops over everything in sys.modules, in a function by the name of appengine.dist.fix_paths()... but I wasn't able to find out who calls this function, when and why. So, pretty much, except for these bits of "nosy" code, the vast majority of code out there seems to only mess with its own self-contained paths, making the use of tuples seem like a pretty safe choice. (Oh, and all the code I found that reads paths without modifying them only use tuple-safe operations.) So, if we implement automatic __path__ updates for virtual packages, I'm currently leaning towards the strict approach using tuples, but could possibly be persuaded towards read-only list-proxies instead. Side note: it looks like a *lot* of code out there abuses __path__[0] to find data files, so I probably need to add a note to the PEP about not doing that when you convert a self-contained package to a virtual one. Of course, I suppose using a sentinel could address *that* problem, or an iteration-only proxy. The main concern here is that using __path__[0] will *seem* to work when you first use it with a virtual package, because it'll be the right directory. But it'll be wrong long-term. This seems to lean in favor of making a simple reiterable wrapper type for the __path__, that only allows you to take the length and iterate over it. With an appropriate design, it could actually update itself automatically, given a subname and a parent __path__/sys.path. That is, it could keep a tuple copy of the last-seen parent path, and before iteration, compare tuple(self.parent_path) to self.last_seen_path. If they're different, it rebuilds the value to be iterated over. Voila: transparent updating of all virtual __path__ values from sys.path changes (or modifications to self-contained __path__ parents, btw), and trying to change it (or read an item from it positionally) will not create any silent failures. Alright... *if* we support automatic updates to virtual __paths__, this is probably how we should do it. (It will require, though, that imp.find_module be changed to use a different iteration method than PyList_GetItem, as it's quite possible a virtual __path__ will get passed into it.) Also, we *long* ago passed the point where any of this can be sanely backported to Python 2.x with a simple shim, alas. For my purposes at least, needing a full importlib for the implementation is a no-go. :-( Still, for the future of Python, this all makes good sense. I just wish we'd thought of all this in 2006 when the discussion came up before: we maybe could've had this in Python 2.6. Where's that damn time machine when you *really* need it? ;-)