[Distutils] API for finding plugins

[Phillip J. Eby]

At 06:59 AM 2/7/2006 -0500, Kevin Dangoor wrote:
>On 2/7/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> > My question at this point is, is this algorithm sane?  For example, is it
> > reasonable to fall back to older versions of plugins in the case of missing
> > dependencies or version conflicts, or would it be better to just disable
> > those plugins altogether?  I can also imagine that some environments might
> > want to have the option of "failing safe" by switching back to a known-good
> > prior configuration, or else failing altogether, rather than have arbitrary
> > drops or version fallbacks.
>
>To date, I've just been hunting among the installed packages for
>things that satisfy particular entry points. I'm not certain I
>understand the intersection between plugins here and entry points. (If
>this is already documented in the setuptools documentation, feel free
>to say that...)

The difference is that plugins of this kind are installed into an 
application instance-specific directory or directories, rather than to the 
default sys.path.  Chandler plugins, for example, might be installed in a 
user's profile directory, much like Firefox extensions.

The key here is that an application wants to scan its plugin directory or 
directories and figure out what eggs can safely be added to sys.path, thus 
making their entry points available.  In theory, if the application uses 
easy_install to set up and configure its plugins, everything would be fine 
of course, but an application's plugin directory isn't normally a "site" 
directory; i.e., it's not going to support .pth files and so can't have a 
"default" version set for each package.

Thus, a more dynamic approach is usually needed.  Currently, the Zope 
"basket" product and the Trac plugin system scan a plugin directory in this 
fashion, looking for suitable eggs and adding them to sys.path.  It's 
possible of course to do this selection by looking for application-specific 
entry points, and in fact some of these programs do that.  Indeed, I 
started out with that idea in my first draft code for Chandler.  What I 
soon realized, however, is that there is going to be other metadata besides 
entry points.  For example, we're expecting to supply i18n metadata listing 
message catalogs and other localized resources, so that you can translate 
one plugin using data in another.  A plugin that consists solely of 
translations isn't going to contain any entry points, so using entry points 
as a filter isn't a great idea, and neither is scanning eggs for an 
ever-increasing list of possible metadata that might identify a Chandler 
plugin.

Thus, it became clear to me that I needed to have some way to just activate 
everything in the plugin directory, and look for metadata and entry points 
later by scanning the working_set.  But "everything" needs to be a 
consistently-ordered and "guaranteed" usable set of distributions.

Thinking over the plugin API some more, at this point I'm leaning towards 
just adding a 'fallback=True' flag, so that if version fallback is 
undesirable it can be turned off.  Systems wanting to implement an even 
stricter fallback mechanism (i.e., always fall back to a known working set 
of plugins if a new set can't be initialized) could then just check the 
error_info dictionary and resort to their fallback plan if there's anything 
in it.

(The reason why different fallback plans may be desirable is that some 
applications may have user data that depends on a particular version of a 
plugin; arbitrary upgrade or downgrade of a plugin could conceivably cause 
data corruption in such scenarios.)