I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues. I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements: * It should not cause anything to be imported before it actually needs to be used by the application * It should be able to deal gracefully with multiple installed versions of a project in designated plugin directories, falling back to older versions if a newer version's requirements can't be met * It should not actually add anything to sys.path until all plugins have been analyzed. The proposed API would be a 'find_plugins()' method added to the WorkingSet class, as follows: ========== find_plugins(plugin_env, full_env=None) Scan `plugin_env` and identify which distributions could be added to this working set without version conflicts or missing requirements. Example usage:: distributions, errors = working_set.find_plugins( Environment(plugin_dirlist) ) map(working_set.add, distributions) # add plugins+libs to sys.path print "Couldn't load", errors # display errors The `plugin_env` should be an ``Environment`` instance that contains only distributions that are in the project's "plugin directory" or directories. The `full_env`, if supplied, should be an ``Environment`` instance that contains all usable distributions, *including* those listed in `plugin_env`. If `full_env` is not supplied, one is created automatically from the ``WorkingSet`` this method is called on, which will typically mean that every directory on ``sys.path`` will be scanned. This method returns a 2-tuple: (`distributions`, `error_info`), where `distributions` is a list of the distributions found in `plugin_env` that were loadable, along with any other distributions that are needed to resolve their dependencies. `error_info` is a dictionary mapping unloadable plugin distributions to an exception instance describing the error that occurred. Usually this will be a ``DistributionNotFound`` or ``VersionConflict`` instance. Most applications will use this method mainly on the master ``working_set`` instance in ``pkg_resources``, and then immediately add the returned distributions to the working set so that they are available on sys.path. This will make it possible to find any entry points, and allow any other metadata tracking and hooks to be activated. The resolution algorithm used by ``find_plugins()`` is as follows. First, the project names of the distributions present in `plugin_env` are sorted. Then, each project's eggs are tried in descending version order (i.e., newest version first). An attempt is made to resolve that egg's dependencies. If the attempt is successful, the egg and its dependencies are added to the output list and to a temporary copy of the working set. The process continues with the next project, and no older eggs for that project are tried. If the resolution attempt fails, however, the error is added to the error dictionary and the next older version of the plugin is tried, until a working version is found. Note that this algorithm gives precedence to satisfying the dependencies of alphabetically prior project names in case of version conflicts. If two projects named "AaronsPlugin" and "ZekesPlugin" both need different versions of "TomsLibrary", then "AaronsPlugin" will win and "ZekesPlugin" will be disabled due to version conflict. ========== My question at this point is, is this algorithm sane? For example, is it reasonable to fall back to older versions of plugins in the case of missing dependencies or version conflicts, or would it be better to just disable those plugins altogether? I can also imagine that some environments might want to have the option of "failing safe" by switching back to a known-good prior configuration, or else failing altogether, rather than have arbitrary drops or version fallbacks. Ergo, this API is only a preliminary proposal. I'd like to hear from folks who've tried to implement their own plugin finders, or who would like to have one, as to their thoughts on what kind of fallback approaches seem best. That way, I can refine the API better, and perhaps get the benefit of some of your field experience with previously-tried approaches.
On 2/7/06, Phillip J. Eby <pje@telecommunity.com> wrote:
My question at this point is, is this algorithm sane? For example, is it reasonable to fall back to older versions of plugins in the case of missing dependencies or version conflicts, or would it be better to just disable those plugins altogether? I can also imagine that some environments might want to have the option of "failing safe" by switching back to a known-good prior configuration, or else failing altogether, rather than have arbitrary drops or version fallbacks.
To date, I've just been hunting among the installed packages for things that satisfy particular entry points. I'm not certain I understand the intersection between plugins here and entry points. (If this is already documented in the setuptools documentation, feel free to say that...) Kevin
Phillip J. Eby wrote:
I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues.
I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements:
I don't fully understand the goal here. From later discussion, I think you envision a model where people drop eggs into some directory and an application should be able to analyze this directory to determine which ones to use, meaning which to include in some working set. In addition to automatically determining the working set, the application should be able to find the entry points in that working set. Does that capture what you want to do? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
At 07:15 AM 2/9/2006 -0500, Jim Fulton wrote:
Phillip J. Eby wrote:
I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues. I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements:
I don't fully understand the goal here. From later discussion, I think you envision a model where people drop eggs into some directory and an application should be able to analyze this directory to determine which ones to use, meaning which to include in some working set. In addition to automatically determining the working set, the application should be able to find the entry points in that working set.
Does that capture what you want to do?
Yes, similar to the Zope 2 Basket product or Trac's plugin facility. It also relates to non-entrypoint metadata, like the resource system that Ian and I have been discussing. The goal there is to allow the located plugin eggs to offer translations, localizations, skins, etc. for other eggs to use. I expect this will be relevant to Zope also, so I'd appreciate your input there as well. I've been away from Zope 3 a little too long to know whether the ideas we're discussing can be made to work for it as well.
Phillip J. Eby wrote:
At 07:15 AM 2/9/2006 -0500, Jim Fulton wrote:
Phillip J. Eby wrote:
I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues. I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements:
I don't fully understand the goal here. From later discussion, I think you envision a model where people drop eggs into some directory and an application should be able to analyze this directory to determine which ones to use, meaning which to include in some working set. In addition to automatically determining the working set, the application should be able to find the entry points in that working set.
Does that capture what you want to do?
Yes, similar to the Zope 2 Basket product or Trac's plugin facility.
OK, well I'll share the following intuition. I think it's important to distinguish, as I think you have, between application construction and pluggins. (I think Zope 2 blurred these too much.) I think these activities are performed by different people with different concerns. I think the people who install plugins are idiots, in the best sense of the word. :) Meaning people who are not technically savy or not technically interested. Therefore, I think plugins should be as simple as possible, providing well constrained functionality. As mentioned elsewhere, I think, pluggin installation should be an explicit operation under the control of the application. (This is not what Zope 2 does, btw.)
It also relates to non-entrypoint metadata, like the resource system that Ian and I have been discussing. The goal there is to allow the located plugin eggs to offer translations, localizations, skins, etc. for other eggs to use. I expect this will be relevant to Zope also, so I'd appreciate your input there as well. I've been away from Zope 3 a little too long to know whether the ideas we're discussing can be made to work for it as well.
I've read the thread a couple of times and am not entirely clear what your use cases are. I think it's all a bit too abstract for me. I suspect that Zope's component architecture is close to what you might want for a "plugin registry". It is serving our needs in this area very well. A package advertises the faclities it provides through xml configuration files (ZCML). People installing a package elect whether to use the package configuration, or provide their own. Often, they do both, since ZCML supports overriding. This system is far from perfect and continues to evolve. We do have a lot of valuable experience in this area though. This sounds like a good topic for an Open Space discussion at PyCon. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Phillip J. Eby wrote:
I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues.
I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements:
* It should not cause anything to be imported before it actually needs to be used by the application
* It should be able to deal gracefully with multiple installed versions of a project in designated plugin directories, falling back to older versions if a newer version's requirements can't be met
* It should not actually add anything to sys.path until all plugins have been analyzed.
The proposed API would be a 'find_plugins()' method added to the WorkingSet class, as follows:
==========
find_plugins(plugin_env, full_env=None) Scan `plugin_env` and identify which distributions could be added to this working set without version conflicts or missing requirements.
Example usage::
distributions, errors = working_set.find_plugins( Environment(plugin_dirlist) ) map(working_set.add, distributions) # add plugins+libs to sys.path print "Couldn't load", errors # display errors
The `plugin_env` should be an ``Environment`` instance that contains only distributions that are in the project's "plugin directory" or directories. The `full_env`, if supplied, should be an ``Environment`` instance that contains all usable distributions, *including* those listed in `plugin_env`. If `full_env` is not supplied, one is created automatically from the ``WorkingSet`` this method is called on, which will typically mean that every directory on ``sys.path`` will be scanned.
This method returns a 2-tuple: (`distributions`, `error_info`), where `distributions` is a list of the distributions found in `plugin_env` that were loadable, along with any other distributions that are needed to resolve their dependencies. `error_info` is a dictionary mapping unloadable plugin distributions to an exception instance describing the error that occurred. Usually this will be a ``DistributionNotFound`` or ``VersionConflict`` instance.
Most applications will use this method mainly on the master ``working_set`` instance in ``pkg_resources``, and then immediately add the returned distributions to the working set so that they are available on sys.path. This will make it possible to find any entry points, and allow any other metadata tracking and hooks to be activated.
The resolution algorithm used by ``find_plugins()`` is as follows. First, the project names of the distributions present in `plugin_env` are sorted. Then, each project's eggs are tried in descending version order (i.e., newest version first). An attempt is made to resolve that egg's dependencies. If the attempt is successful, the egg and its dependencies are added to the output list and to a temporary copy of the working set. The process continues with the next project, and no older eggs for that project are tried. If the resolution attempt fails, however, the error is added to the error dictionary and the next older version of the plugin is tried, until a working version is found.
Note that this algorithm gives precedence to satisfying the dependencies of alphabetically prior project names in case of version conflicts. If two projects named "AaronsPlugin" and "ZekesPlugin" both need different versions of "TomsLibrary", then "AaronsPlugin" will win and "ZekesPlugin" will be disabled due to version conflict.
==========
My question at this point is, is this algorithm sane?
Depends on what you mean by "sane". :) There seems to be an assumption that things get into the plugin directory outside of the application's control. Is that a good idea? Perhaps a better model would be for users of the application to install plugins one by one. The application can advise them of sucess or failure, let them know about conflicts and possible remedies and let the user decide what to do. I think this would be a better model. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
participants (3)
-
Jim Fulton -
Kevin Dangoor -
Phillip J. Eby