[Distutils] API for finding plugins

[Ian Bicking]

Kevin Dangoor wrote:
> On 2/7/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> 
>>At 06:59 AM 2/7/2006 -0500, Kevin Dangoor wrote:
>>
>>>To date, I've just been hunting among the installed packages for
>>>things that satisfy particular entry points. I'm not certain I
>>>understand the intersection between plugins here and entry points. (If
>>>this is already documented in the setuptools documentation, feel free
>>>to say that...)
>>
>>The difference is that plugins of this kind are installed into an
>>application instance-specific directory or directories, rather than to the
>>default sys.path.  Chandler plugins, for example, might be installed in a
>>user's profile directory, much like Firefox extensions.
>>
>>The key here is that an application wants to scan its plugin directory or
>>directories and figure out what eggs can safely be added to sys.path, thus
>>making their entry points available.  In theory, if the application uses
>>easy_install to set up and configure its plugins, everything would be fine
>>of course, but an application's plugin directory isn't normally a "site"
>>directory; i.e., it's not going to support .pth files and so can't have a
>>"default" version set for each package.
> 
> 
> Ahh, OK. Your explanation makes the difference clear. I'm pretty sure
> that TurboGears doesn't need plugins (in this terminology) yet, but I
> could imagine that some TurboGears apps may like to use plugins (much
> like Trac does).

I actually think there's some intersection between what you and I are 
doing, and what Trac and Chandler are doing.  In that, I'm a little 
worried that what we're doing is going to have scaling problems once a 
rich set of plugins is available.  The opt-in nature of a plugin 
directory solves some of these problems; and much of what Phillip is 
describing (in terms of being able to scan for lots of kinds of content 
and other things provided by plugins) does not seem so far out of what I 
am doing with eggs and entry points.  *Except*, that with what we're 
doing there's not an ownership relationship.  If Paste and TG provide 
things to each other, one of them doesn't own the other -- TG wouldn't 
go in Paste's plugin dir, nor vice versa.  I think even if the packages 
were refactored to be more granular, it still wouldn't work out that way.

At the same time, I feel like opt-in plugins -- explicitly enumerated in 
some way -- are going to be necessary for us.  Right now if an egg is 
not activated by default, you won't see its entry points when scanning 
by entry point group.  This has confused me before, which I figure is a 
good predictor that other people are going to be confused by this too.

Right now in Paste there's a couple ways you opt-in.  One is with 
middleware, which doesn't get picked up implicitly, but instead gets 
specifically activated and configured.  The other is with PasteScript 
plugins, where a list of applicable plugins is put in your .egg-info 
directory for the project.  Those generally work pretty well.  But I'm 
not sure how to apply that in other areas, like PasteScript commands 
that aren't project-local.  I also worry -- but have not actually 
profiled -- that the scanning is causing Paste Script to be slow on 
startup (Paste might be too, but I wouldn't notice as much).  It would 
be good to look into this more closely; but it would also be really good 
to catch any problems now -- and fix that -- instead of later when it 
might lead to people bitching about it and talking down setuptools as a 
result.

I also want content plugins, like the internationalization plugins (but 
it might be Javascript or templates or whatever).  And certainly that 
applies to TG as well.  I've been doing this at work, but using entry 
points, where the entry points points to an object that is a string that 
points to a resource that is then fetched with pkg_resources.  Something 
more direct would be appreciated ;)  I like entry point groups, and 
applying the same idea to resources (but without the indirection) would 
be great.

Hmm... and reading over this, I realize that a "plugin directory" is not 
that different from the virtual python setups, and I think that kind of 
setup is really the only good way to deploy web applications.  "Working 
set" -- which is already an idea in setuptools -- is probably the 
concept that encompasses both uses.  As a developer, I'd like it to be 
easier and safer to change my working set -- right now I have hacks in 
sitecustomize and distutils to make that easy.  This switching of 
working sets is basically what Jim wants in a Zope Instance as well.

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org