[Distutils] Standardizing distribution of "plugins" for extensible apps

[Bob Ippolito]

On Dec 7, 2004, at 10:40 PM, Phillip J. Eby wrote:

> Many applications need or want to be able to support extension via 
> dynamic installation of "plugin" code, such as Zope Products, Chandler 
> Parcels, and WSGI servers' "application" objects.  Frequently, these 
> plugins may require access to other plugins or to libraries of other 
> Python code.  Currently, application platforms must create their own 
> binary formats to address their specific requirements, but these 
> formats are of course specific to the platform and not portable, so 
> it's not possible to package a third-party module just once, and 
> deploy it in any Python application platform.
>
> Although each platform may have its own additional requirements for 
> the contents of such a "plugin", the minimum basis for such plugins is 
> that they include Python modules and other files, and import and 
> export selected modules.  Although the standard distutils pattern is 
> to use a platform's own packaging system, this really only makes sense 
> if you are dealing with Python as a language, and not as an 
> application platform.  Platform packaging doesn't make sense for 
> applications that are end-user programmable, because even if the core 
> application can be installed in one location, each instance of use of 
> that application (e.g. per user on a multi-user system) may have its 
> own plugins installed.

Is anything you're proposing NOT applicable to Python packages also?  A 
lot of the issues you're proposing to solve are problems we have for 
more than just plugins.  Though, I suppose it's a lot easier to propose 
a new plugin mechanism than a major overhaul to the module system ;)

> Therefore, I would like to propose the creation of:
>
> * a binary package format optimized for "plugin" use (probably just a 
> specific .zip layout)
>
> * a distutils "bdist" command to package a set of modules in this 
> format
>
> * A PEP 302 importer that supports running of code directly from 
> plugin distributions, and which can be configured to install or 
> extract C extension modules.  (This will serve as a base for platform 
> developers to create their platform-specific plugin registries, 
> installation mechanisms, etc., although there will be opportunity for 
> standardization/code sharing at this level, too.)

This importer also needs to grow methods so that these plugins can 
introspect themselves and ask for things (metadata that may be in the 
zip, etc.).  Something smarter than zipimport's get_data would be nice 
too.

Also, if you allow extraction of C extension modules, you'll probably 
also have to allow extraction of dependent dlls and whatnot.. which is 
a real mess.  For dependent dynamic libraries on Darwin, this is a 
*real* mess, because the runtime linker is only affected by environment 
variables at process startup time.  I can only think of two solutions 
to this for Darwin:
(a) build an executable bundle on and execve it on process startup, 
drop the dependent libraries inside that executable bundle
(b) have some drop location for dependent libraries and C extensions 
and rewrite the load commands in them before loading (which may fail if 
there isn't enough wiggle room in the header)

For Darwin, py2app would be more or less required at runtime to do 
either...  Note that either can be done without a compiler handy, 
because py2app can read and write enough of the Mach-O object file 
format (equivalent to ELF or PE).

> * additions to setup() metadata to support declaration of:
>
>   * modules required/provided, with version information

Does this mean sys.modules keys will also have a version in them?  How 
do you import cross-plugin or cross-package in this scenario?

>   * platform metadata for C extensions contained in the plugin 
> distribution

How do you build one of these that has C extensions for multiple 
platforms?  Multiple variants of the same platform?  Multiple Python 
runtime versions?  It's not always possible to cross-compile from 
platform A to B for arbitrary A and B.

>   * ability to specify metadata files to be included in the 
> distribution, that are specific to the target application platform 
> (e.g. Zope config files, Chandler parcel schemas, WSGI deployment 
> configuration, etc.)
>
> (This is actually only "level 1" of the standardization that I'd like 
> to do; levels 2 and 3 would address runtime issues like 
> startup/shutdown of plugins, automatic dependency resolution, 
> isolation between plugins, and mediated service discovery and service 
> management across plugins.  However, these other levels are distinct 
> deliverables that don't necessarily relate to the distutils, except 
> insofar as those levels may influence requirements for the first 
> level.  Also, it's important to note that even without those higher 
> levels of standardization, the availability of a "plug and play" 
> distribution format should be beneficial to the Python community, in 
> making it easier for applications to bundle arbitrary libraries.  
> Indeed, tools like py2exe and py2app might grow the ability to 
> automatically "link" bundles needed by an application, and there will 
> likely be many other spin-off uses of this technology once it's 
> available.)
>
> My main idea is to expand the existing PKG-INFO format, adding some 
> formally-defined fields for system processing, that are supplied by 
> the setup script or in additional files.  The additional metadata 
> should be syntactically validated (and semantically, to the extent 
> possible), and the distutils should not knowingly produce a plugin 
> package with invalid execution metadata.  The specific kinds of 
> metadata needed (that I know of currently) are:
>
> * Static imports (module names and specification of compatible 
> versions)
> * Dynamic imports (wildcard masks)
> * Exports (module names and versions)
> * Platform requirements for included extension modules
> * Other platform requirements
> * Entry point for dynamic integration (technically a level 2 feature, 
> but it'd be nice to reserve the header and define its syntax)

What do you mean by "entry point" and "dynamic integration"?

> * Update URL (for obtaining the "latest" version of the plugin)

How about some kind of public key as well, so that if you visit the 
update URL you will know if the new package was provided by the same 
author or not?

> There are several issues that would have to be hammered out, here.  
> Versioning, for example, both in the sense of formats and qualifiers, 
> and in the sense of versioning a module/package versus versioning a 
> distribution.  Gathering information about imports is also tricky.  
> Tools like py2exe try to gather some of this information 
> automatically, but that info doesn't include version requirements.  
> (It may be that we can get by without version requirements, taking the 
> default state to mean, "any version will do.")

If we require that modules specify a __version__ that is a "constant", 
it would be easy to parse...  When you "link" this bundle, it could 
automatically say that it requires a version >= to the version it saw 
when it was scanned for dependencies (unless explicitly specified to be 
more or less specific).

In any case, I think versioning is a really hard problem, especially in 
Python due to sys.modules and the import mechanism, so I think that 
this task should be deferred.  If Python developers really felt 
strongly about this, we'd probably see more packages with version 
numbers in their names :)

> Platform specs are another dicey issue, since we're talking about 
> trying to define binary compatibility here.  This includes the issue 
> that it might be necessary to include shared libraries other than the 
> C extensions themselves.  (For example, like the wxWidgets libraries 
> that ship with wxPython.)

The only reasonable solution I've found to this issue is to just shove 
every dependency in the package.. any less than that and it's just too 
hard to deal with.

> While researching a deployment strategy for WSGI, I discovered the 
> OSGi specifications for Java, which address all of these issues and 
> more.  Where OSGi's solutions are directly usable, I'd like to apply 
> them, rather than re-inventing wheels... not to mention axles, brakes, 
> and transmissions!  However, their syntax for these things is often 
> weird and verbose, and could probably use some "Pythonizing".
>
> Anyway, I would see the deliverables here as being a PEP documenting 
> the format, and a prototype implementation in setuptools (for current 
> Python versions), that would then migrate to an official 
> implementation in the distutils for Python 2.5.  I'd like to find out 
> "who's with me" at this stage in having an interest in any aspect of 
> this project, especially if you have requirements or issues I haven't 
> thought of.  Thanks!

I'm +100 on this venture, but I could definitely see it happening in 
pieces, and much of it being applied to packages in general rather than 
just plugins.

-bob