[Python-3000] Should package __init__ files include pkgutil.extend_path?

[Phillip J. Eby]

At 12:38 PM 9/9/2008 -0700, Brett Cannon wrote:
>On Tue, Sep 9, 2008 at 10:06 AM, Phillip J. Eby <pje at telecommunity.com> wrote:
> > At 03:28 PM 9/6/2008 -0700, Brett Cannon wrote:
> >>
> >> On Sat, Sep 6, 2008 at 2:06 PM,  <skip at pobox.com> wrote:
> >> > I'm trying to figure out how to install this dbm.sqlite module I have
> >> > without overwriting the basic install.  My thought was to create a dbm
> >> > package in site-packages then copy sqlite.py there.  That doesn't work
> >> > though.  Modifying dbm.__init__.py to include this does:
> >> >
> >> >    import pkgutil
> >> >    __path__ = pkgutil.extend_path(__path__, __name__)
> >> >
> >> > I'm wondering if all the core packages in 3.x should include the above
> >> > in
> >> > their __init__.py files.
> >> >
> >>
> >> Well, a side-effect of this is that all package imports will suddenly
> >> spike the number of stat calls linearly to the number of entries on
> >> sys.path.
> >
> > "All package imports"?  "Spike"?
> >
>
>pkgutil.extend_path() would be executed for every package imported by
>the fact that is code at the global level of the module.

Each package that uses it and that is imported, yes.

>  And if you
>look at the implementation of extend_path(), there is a
>os.path.isdir() call for every entry on sys.path, and if that succeeds
>there is os.path.isfile() call. Plus there is also an os.path.isfile()
>call for every sys.path entry as well.

Note, btw, that that could be greatly reduced by use of 
sys.path_importer_cache; only entries that are missing or None need 
to have the subdirectory check.

>I call that a "spike" in "all package imports" in terms of stat calls
>if this was added to all packages as suggested. And that can be
>painful on systems where stat calls are expensive (e.g., NFS). At
>least extend_path() appends so the new entries are put at the back of
>the list.

...which actually negates the entire point of the proposal, which was 
somebody wanting to be able to install an override/upgrade of a 
module in a stdlib package.


>Yes, it's a trade-off depending on what penalty cost you would prefer
>to pay. But as I said, I don't like the idea of letting people inject
>into the stdlib namespace like this in the first place so I don't want
>this to happen in any official capacity.

IIUC, the OP was requesting the ability to *upgrade* a 
stdlib-provided module, not add items to the namespace.