[Distutils] API for finding plugins

[Jim Fulton]

Phillip J. Eby wrote:
> I recently started work on adding egg support to Chandler ( 
> http://chandler.osafoundation.org/ ), and ran into some interesting issues 
> with respect to plugin discovery.  Specifically, it's not easy to do it 
> well with the APIs that pkg_resources currently offers.  I suspect that 
> others who've worked on plugin loading for application environments like 
> Zope and Trac have probably run into similar issues.
> 
> I'm proposing, therefore, to add a new API to pkg_resources to make 
> plugin-finding easier.  Among the requirements:
> 
> * It should not cause anything to be imported before it actually needs to 
> be used by the application
> 
> * It should be able to deal gracefully with multiple installed versions of 
> a project in designated plugin directories, falling back to older versions 
> if a newer version's requirements can't be met
> 
> * It should not actually add anything to sys.path until all plugins have 
> been analyzed.
> 
> The proposed API would be a 'find_plugins()' method added to the WorkingSet 
> class, as follows:
> 
> ==========
> 
> find_plugins(plugin_env, full_env=None)
>     Scan `plugin_env` and identify which distributions could be added to 
> this working set without version conflicts or missing requirements.
> 
>     Example usage::
> 
>         distributions, errors = working_set.find_plugins(
>             Environment(plugin_dirlist)
>         )
>         map(working_set.add, distributions)  # add plugins+libs to sys.path
>         print "Couldn't load", errors        # display errors
> 
>     The `plugin_env` should be an ``Environment`` instance that contains 
> only distributions that are in the project's "plugin directory" or 
> directories.  The `full_env`, if supplied, should be an ``Environment`` 
> instance that contains all usable distributions, *including* those listed 
> in `plugin_env`.  If `full_env` is not supplied, one is created 
> automatically from the ``WorkingSet`` this method is called on, which will 
> typically mean that every directory on ``sys.path`` will be scanned.
> 
>     This method returns a 2-tuple: (`distributions`, `error_info`), where 
> `distributions` is a list of the distributions found in `plugin_env` that 
> were loadable, along with any other distributions that are needed to 
> resolve their dependencies.  `error_info` is a dictionary mapping 
> unloadable plugin distributions to an exception instance describing the 
> error that occurred.  Usually this will be a ``DistributionNotFound`` or 
> ``VersionConflict`` instance.
> 
>     Most applications will use this method mainly on the master 
> ``working_set`` instance in ``pkg_resources``, and then immediately add the 
> returned distributions to the working set so that they are available on 
> sys.path.  This will make it possible to find any entry points, and allow 
> any other metadata tracking and hooks to be activated.
> 
>     The resolution algorithm used by ``find_plugins()`` is as 
> follows.  First, the project names of the distributions present in 
> `plugin_env` are sorted.  Then, each project's eggs are tried in descending 
> version order (i.e., newest version first).  An attempt is made to resolve 
> that egg's dependencies.  If the attempt is successful, the egg and its 
> dependencies are added to the output list and to a temporary copy of the 
> working set.  The process continues with the next project, and no older 
> eggs for that project are tried. If the resolution attempt fails, however, 
> the error is added to the error dictionary and the next older version of 
> the plugin is tried, until a working version is found.
> 
>     Note that this algorithm gives precedence to satisfying the 
> dependencies of alphabetically prior project names in case of version 
> conflicts.  If two projects named "AaronsPlugin" and "ZekesPlugin" both 
> need different versions of "TomsLibrary", then "AaronsPlugin" will win and 
> "ZekesPlugin" will be disabled due to version conflict.
> 
> ==========
> 
> My question at this point is, is this algorithm sane? 

Depends on what you mean by "sane". :)

There seems to be an assumption that things get into the plugin
directory outside of the application's control.  Is that a good idea?

Perhaps a better model would be for users of the application to install
plugins one by one.  The application can advise them of sucess or
failure, let them know about conflicts and possible remedies and
let the user decide what to do.  I think this would be a better model.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org