[Import-SIG] One last try: "virtual packages"
pje at telecommunity.com
Tue Jul 12 18:02:40 CEST 2011
At 11:03 AM 7/12/2011 -0400, Barry Warsaw wrote:
>It's a very interesting idea that is worth exploring. A few things come to
>- Under this scheme it's possible for names in a module to "suddenly" appear.
Bear in mind that you still have to actually *import* those names, so
it's not like they really "suddenly" appear. And when you do import
them, they'll be *modules*, not functions or classes or constants or anything.
> E.g. I could install packages that extend existing top level modules like
> `time` or `string`. This might be a good thing in that it gives 3rd party
> folks a more natural place to add things, but it could also open up a
> land-grab type collision if lots of people want to publish their
> packages as
> subpackage extensions to existing modules.
True -- an ironic side-effect, given our intent to make it easier to
*avoid* such collisions. ;-) However, given that this feature will
probably NOT be available on versions <3.3 by default (see discussion
below), it probably won't get *too* far out of hand.
Also, because you can't add new module *contents*, there's little
benefit to doing this anyway. Your users would have to do "from
string.foobar import bizbaz" or "import string.foobar as foobar",
anyway, so why not just make a "foobar.string" module and call it a day?
I also don't think we should really advertise the ability to extend
other people's packages, except maybe to say, "don't do it."
We could also shut down the capability by requiring virtual packages
to be declared in the module, if there is a defining module. That
would actually work well with cross-version compatibility (see below)
but would add an extra step when turning a module into a package.
>- It's unfortunate that this will be more difficult to back port to Python 2.
Well, I'm not that bothered by it. Python 2 still has its two
existing ways to do this, and it's not *that* terribly hard to make
an __import__ wrapper. But there are some things that can be done to
make it easier.
>- It sounds like it will be more difficult to have a single code base that
> supports Python 2, Python3 <= 3.2, and Python 3.3. This is because
> __init__.py is required in the first two, but does the wrong thing (I think
> ;) in a post-PEP 382 Python 3.3. Adding a .pyp file that's ignored in
> anything that doesn't support PEP 382 would make it easier to support
> multiple Pythons.
There's a straightforward way to solve this. Suppose we have a
module called 'pep382', with a function
'make_virtual(packagename)'. In Python 2.x, setuptools will make
"distributionname-version-nspkg.pth" files that just say 'import
pep382; pep382.make_virtual("toplevelnamespace")', and the same
solution would work for Python 3 through 3.2. (In the .egg based
install case, __init__.py gets used and the older API is called, but
in future setuptools that'll be a wrapper over the pep382 module.)
For Python 3.3, these APIs don't need to be used, but they'll still
work. They just won't be doing anything significant. You can drop
use of the APIs as you drop support for older Pythons, and code
targeted to 3.3+ can just do whatever.
For Python < 3.3, you have to get the pep382 module installed and
activated somehow in order to use the feature. However, once you do,
you can use "pure virtual" packages without an __import__ hook,
because a meta_path importer can catch an otherwise-failed import and
set up an empty module with a __path__.
IOW, the difficult part of implementing this on 2.x is only the part
where you allow transitioning from a 'foo' module to a 'foo' package
without changing the module. If you're using namespaces the way
people mostly do now on 2.x, it works without an __import__ hook.
For this reason, I suggest that the default for the
backwards-compatibility module be to only handle pure-virtual and
declared-virtual packages, not module-extension virtual
packages. That way, the overhead remains low. (Writing __import__
in Python adds overhead to *every* import statement, vs. the
relatively small and infrequent overheads added by PEP 302 hooks.)
>Let's see the PEP!
Martin said something about working one up along similar lines
himself; I'm curious to see what his proposal is.
More information about the Import-SIG