[Distutils] Standardizing distribution of "plugins" for extensible apps

[Thomas Heller]

"Phillip J. Eby" <pje at telecommunity.com> writes:

> At 01:32 PM 12/9/04 +0100, Thomas Heller wrote:
>> >> I'd prefer to be able to use a plugin archive (.par, anyone?)
>> >> directly with zipimport in the case that it's a pure Python archive
>> >> (or if it's on some hypothetical platform that can load extensions
>> >> from a zipfile).  Ideally, also, one should also be able to unzip a
>> >> .par directly into site-packages or a subdirectory thereof and use
>> >> it.  Thus, I'd prefer to see an internal layout that's something
>> >> more like:
>> > <zip layout compatible with zipimport>
>> >
>> > Uh, why does it matter if zipimport can do something with it if we're
>> > going to need a custom importer anyway?
>>
>>If you use a custom importer, it could extract extension modules to the
>>file system on demand (for those non-hypotetical platforms where it's
>>required).  And it may even be possible with zipimporter, if the archive
>>has some custom extension loaders.
>
> That's actually a good point; py2exe does basically that now, with a
> pure Python stub replacing the extension.

Mark does the same in the win32 extensions; pythoncom24.dll and
pywintypes24.dll are also loaded via python stubs.

> Of course, py2exe relies on the extension and its libraries being
> outside the archive.
>
> Hm.  Does the normal Python search look for extensions first or
> second?  I'm just wondering if unzipping an archive with such stubs
> would automatically work when Python does its normal import mechanism.
> That is, would the extension take precedence over the stub?

Yes, .pyd is preferred if a .py is available in the same place.  I don't
think this is by accident, and probably the order is determined by what
imp.getsuffixes() returns.

> If so, then that approach is a real winner.  bdist_plugin (or whatever
> it's called) could create stubs that provide extension loading
> support, and which would be ignored if the archive was unpacked.  If
> need be, a support module could be included in the archive's root, so
> that the stubs don't have to be large.

This may even be an idea for single-file py2exe: Let the stub unpack the
extension, maybe in a TMP directory, open the file with
FILE_FLAG_DELETE_ON_CLOSE (so we don't have to clean up manually), and
then let the stub load the extension.

(Off-topic: I've always been interested in the question if it is
possible to emulate LoadLibrary with user code)

> In fact, if the support module allows controlling policy for how/where
> to unpack extensions for dynamic loading, then an application that
> wants to control that stuff can just include and import the support
> module directly, configuring it for its needs before any plugins with
> extensions get loaded.
>
> For later Python versions (2.5), the support module would be part of
> the stdlib, and would no longer need to be distributed inside the
> archives produced.  We'd only distribute it for plugins produced with
> older (<2.5) Pythons, and then only if the archive contains extensions.