[Distutils] Standardizing distribution of "plugins" for extensible apps

[Phillip J. Eby]

Many applications need or want to be able to support extension via dynamic 
installation of "plugin" code, such as Zope Products, Chandler Parcels, and 
WSGI servers' "application" objects.  Frequently, these plugins may require 
access to other plugins or to libraries of other Python code.  Currently, 
application platforms must create their own binary formats to address their 
specific requirements, but these formats are of course specific to the 
platform and not portable, so it's not possible to package a third-party 
module just once, and deploy it in any Python application platform.

Although each platform may have its own additional requirements for the 
contents of such a "plugin", the minimum basis for such plugins is that 
they include Python modules and other files, and import and export selected 
modules.  Although the standard distutils pattern is to use a platform's 
own packaging system, this really only makes sense if you are dealing with 
Python as a language, and not as an application platform.  Platform 
packaging doesn't make sense for applications that are end-user 
programmable, because even if the core application can be installed in one 
location, each instance of use of that application (e.g. per user on a 
multi-user system) may have its own plugins installed.

Therefore, I would like to propose the creation of:

* a binary package format optimized for "plugin" use (probably just a 
specific .zip layout)

* a distutils "bdist" command to package a set of modules in this format

* A PEP 302 importer that supports running of code directly from plugin 
distributions, and which can be configured to install or extract C 
extension modules.  (This will serve as a base for platform developers to 
create their platform-specific plugin registries, installation mechanisms, 
etc., although there will be opportunity for standardization/code sharing 
at this level, too.)

* additions to setup() metadata to support declaration of:

   * modules required/provided, with version information

   * platform metadata for C extensions contained in the plugin distribution

   * ability to specify metadata files to be included in the distribution, 
that are specific to the target application platform (e.g. Zope config 
files, Chandler parcel schemas, WSGI deployment configuration, etc.)

(This is actually only "level 1" of the standardization that I'd like to 
do; levels 2 and 3 would address runtime issues like startup/shutdown of 
plugins, automatic dependency resolution, isolation between plugins, and 
mediated service discovery and service management across plugins.  However, 
these other levels are distinct deliverables that don't necessarily relate 
to the distutils, except insofar as those levels may influence requirements 
for the first level.  Also, it's important to note that even without those 
higher levels of standardization, the availability of a "plug and play" 
distribution format should be beneficial to the Python community, in making 
it easier for applications to bundle arbitrary libraries.  Indeed, tools 
like py2exe and py2app might grow the ability to automatically "link" 
bundles needed by an application, and there will likely be many other 
spin-off uses of this technology once it's available.)

My main idea is to expand the existing PKG-INFO format, adding some 
formally-defined fields for system processing, that are supplied by the 
setup script or in additional files.  The additional metadata should be 
syntactically validated (and semantically, to the extent possible), and the 
distutils should not knowingly produce a plugin package with invalid 
execution metadata.  The specific kinds of metadata needed (that I know of 
currently) are:

* Static imports (module names and specification of compatible versions)
* Dynamic imports (wildcard masks)
* Exports (module names and versions)
* Platform requirements for included extension modules
* Other platform requirements
* Entry point for dynamic integration (technically a level 2 feature, but 
it'd be nice to reserve the header and define its syntax)
* Update URL (for obtaining the "latest" version of the plugin)

There are several issues that would have to be hammered out, 
here.  Versioning, for example, both in the sense of formats and 
qualifiers, and in the sense of versioning a module/package versus 
versioning a distribution.  Gathering information about imports is also 
tricky.  Tools like py2exe try to gather some of this information 
automatically, but that info doesn't include version requirements.  (It may 
be that we can get by without version requirements, taking the default 
state to mean, "any version will do.")

Platform specs are another dicey issue, since we're talking about trying to 
define binary compatibility here.  This includes the issue that it might be 
necessary to include shared libraries other than the C extensions 
themselves.  (For example, like the wxWidgets libraries that ship with 
wxPython.)

While researching a deployment strategy for WSGI, I discovered the OSGi 
specifications for Java, which address all of these issues and more.  Where 
OSGi's solutions are directly usable, I'd like to apply them, rather than 
re-inventing wheels... not to mention axles, brakes, and 
transmissions!  However, their syntax for these things is often weird and 
verbose, and could probably use some "Pythonizing".

Anyway, I would see the deliverables here as being a PEP documenting the 
format, and a prototype implementation in setuptools (for current Python 
versions), that would then migrate to an official implementation in the 
distutils for Python 2.5.  I'd like to find out "who's with me" at this 
stage in having an interest in any aspect of this project, especially if 
you have requirements or issues I haven't thought of.  Thanks!