[Python-3000] Should package __init__ files include pkgutil.extend_path?

[Brett Cannon]

On Tue, Sep 9, 2008 at 10:06 AM, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 03:28 PM 9/6/2008 -0700, Brett Cannon wrote:
>>
>> On Sat, Sep 6, 2008 at 2:06 PM,  <skip at pobox.com> wrote:
>> > I'm trying to figure out how to install this dbm.sqlite module I have
>> > without overwriting the basic install.  My thought was to create a dbm
>> > package in site-packages then copy sqlite.py there.  That doesn't work
>> > though.  Modifying dbm.__init__.py to include this does:
>> >
>> >    import pkgutil
>> >    __path__ = pkgutil.extend_path(__path__, __name__)
>> >
>> > I'm wondering if all the core packages in 3.x should include the above
>> > in
>> > their __init__.py files.
>> >
>>
>> Well, a side-effect of this is that all package imports will suddenly
>> spike the number of stat calls linearly to the number of entries on
>> sys.path.
>
> "All package imports"?  "Spike"?
>

pkgutil.extend_path() would be executed for every package imported by
the fact that is code at the global level of the module. And if you
look at the implementation of extend_path(), there is a
os.path.isdir() call for every entry on sys.path, and if that succeeds
there is os.path.isfile() call. Plus there is also an os.path.isfile()
call for every sys.path entry as well.

I call that a "spike" in "all package imports" in terms of stat calls
if this was added to all packages as suggested. And that can be
painful on systems where stat calls are expensive (e.g., NFS). At
least extend_path() appends so the new entries are put at the back of
the list.

>
>> Another option is to use a pth file that imports your module (as like
>> _dbm_sqlite.py or something) and have it, as a side-effect of
>> importing, set itself on dbm.
>
> That adds an import to startup time, whether you use the package or not.  At
> least extend_path will only take effect if you actually import that package.
>

Yes, it's a trade-off depending on what penalty cost you would prefer
to pay. But as I said, I don't like the idea of letting people inject
into the stdlib namespace like this in the first place so I don't want
this to happen in any official capacity.

-Brett