[Python-Dev] PEP 382: Namespace Packages

[M.-A. Lemburg]

On 2009-04-07 16:05, P.J. Eby wrote:
> At 02:30 PM 4/7/2009 +0200, M.-A. Lemburg wrote:
>> >> Wouldn't it be better to stick with a simpler approach and look for
>> >> "__pkg__.py" files to detect namespace packages using that O(1)
>> check ?
>> >
>> > Again - this wouldn't be O(1). More importantly, it breaks system
>> > packages, which now again have to deal with the conflicting file names
>> > if they want to install all portions into a single location.
>>
>> True, but since that means changing the package infrastructure, I think
>> it's fair to ask distributors who want to use that approach to also take
>> care of looking into the __pkg__.py files and merging them if
>> necessary.
>>
>> Most of the time the __pkg__.py files will be empty, so that's not
>> really much to ask for.
> 
> This means your proposal actually doesn't add any benefit over the
> status quo, where you can have an __init__.py that does nothing but
> declare the package a namespace.  We already have that now, and it
> doesn't need a new filename.  Why would we expect OS vendors to start
> supporting it, just because we name it __pkg__.py instead of __init__.py?

I lost you there.

Since when do we support namespace packages in core Python without
the need to add some form of magic support code to __init__.py ?

My suggestion basically builds on the same idea as Martin's PEP,
but uses a single __pkg__.py file as opposed to some non-Python
file yaddayadda.pkg.

Here's a copy of the proposal, with some additional discussion
bullets added:

"""
Alternative Approach:
---------------------

Wouldn't it be better to stick with a simpler approach and look for
"__pkg__.py" files to detect namespace packages using that O(1) check ?

This would also avoid any issues you'd otherwise run into if you want
to maintain this scheme in an importer that doesn't have access to a list
of files in a package directory, but is well capable for the checking
the existence of a file.

Mechanism:
----------

If the import mechanism finds a matching namespace package (a directory
with a __pkg__.py file), it then goes into namespace package scan mode and
scans the complete sys.path for more occurrences of the same namespace
package.

The import loads all __pkg__.py files of matching namespace packages
having the same package name during the search.

One of the namespace packages, the defining namespace package, will have
to include a __init__.py file.

After having scanned all matching namespace packages and loading
the __pkg__.py files in the order of the search, the import mechanism
then sets the packages .__path__ attribute to include all namespace
package directories found on sys.path and finally executes the
__init__.py file.

(Please let me know if the above is not clear, I will then try to
follow up on it.)

Discussion:
-----------

The above mechanism allows the same kind of flexibility we already
have with the existing normal __init__.py mechanism.

* It doesn't add yet another .pth-style sys.path extension (which are
difficult to manage in installations).

* It always uses the same naive sys.path search strategy. The strategy
is not determined by some file contents.

* The search is only done once - on the first import of the package.

* It's possible to have a defining package dir and add-one package
dirs.

* The search does not depend on the order of directories in sys.path.
There's no requirement for the defining package to appear first
on sys.path.

* Namespace packages are easy to recognize by testing for a single
resource.

* There's no conflict with existing files using the .pkg extension
such as Mac OS X installer files or Solaris packages.

* Namespace __pkg__.py modules can provide extra meta-information,
logging, etc. to simplify debugging namespace package setups.

* It's possible to freeze such setups, to put them into ZIP files,
or only have parts of it in a ZIP file and the other parts in the
file-system.

* There's no need for a package directory scan, allowing the
mechanism to also work with resources that do not permit to
(easily and efficiently) scan the contents of a package "directory",
e.g. frozen packages or imports from web resources.

Caveats:

* Changes to sys.path will not result in an automatic rescan for
additional namespace packages, if the package was already loaded.
However, we could have a function to make such a rescan explicit.
"""

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 07 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2009-03-19: Released mxODBC.Connect 1.0.1      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/