[Distutils] API for finding plugins

[Ian Bicking]

Phillip J. Eby wrote:
> At 11:31 AM 2/7/2006 -0600, Ian Bicking wrote:
> 
>> At the same time, I feel like opt-in plugins -- explicitly enumerated 
>> in some way -- are going to be necessary for us.  Right now if an egg 
>> is not activated by default, you won't see its entry points when 
>> scanning by entry point group.  This has confused me before, which I 
>> figure is a good predictor that other people are going to be confused 
>> by this too.
> 
> 
> Others have been confused too, but it's also quite easy to understand 
> once you've bumped into it once or twice.  Especially since you really 
> don't want the entry points for *every* version of a project to show up 
> in the list.  Explicit activation on sys.path is the poor man's global 
> "plugin directory", so now I'm finally following up on having a clean 
> way to have a *real* plugin directory.

Indeed -- and I'm thinking we all need to go that way, because the poor 
man's plugin directory is already showing some problems.

>> Right now in Paste there's a couple ways you opt-in.  One is with 
>> middleware, which doesn't get picked up implicitly, but instead gets 
>> specifically activated and configured.  The other is with PasteScript 
>> plugins, where a list of applicable plugins is put in your .egg-info 
>> directory for the project.  Those generally work pretty well.  But I'm 
>> not sure how to apply that in other areas, like PasteScript commands 
>> that aren't project-local.
> 
> 
> Configuration files, perhaps?

Yes, but I've avoiding those so far.  I'd have to locate them (in 
/etc/paster/*?), and then provide options to specify them, and so forth. 
  It will probably be necessary... but I feel like there's something 
useful in defering that too.  But maybe that is irrational.

>> I also worry -- but have not actually profiled -- that the scanning is 
>> causing Paste Script to be slow on startup (Paste might be too, but I 
>> wouldn't notice as much).  It would be good to look into this more 
>> closely; but it would also be really good to catch any problems now -- 
>> and fix that -- instead of later when it might lead to people bitching 
>> about it and talking down setuptools as a result.
> 
> 
> FYI, the best way to minimize scan overhead is to avoid repeatedly 
> creating Environment instances; this is the slowest thing you can 
> possibly do.  Note that most pkg_resources APIs were designed to be 
> called once at startup and that's it, so they usually are able to create 
> a default Environment if one isn't passed in.  However, if you have an 
> API that gets called "at runtime" and it invokes pkg_resources APIs that 
> take Environment instances, you may want to create one early and stash 
> it away for future use.  This prevents the most expensive scanning from 
> taking place repeatedly.

I guess the problem is that for Paste Script, the startup overhead and 
the runtime overhead are the same thing.  Like, 8 seconds to complete 
"paster help" (which gives a listing of all available commands).  After 
the first run it warms up and is much faster (disk caching?), but that 
still doesn't feel good to me.  Now that it's warmed up I can't tell how 
slow a normal command would be -- possible faster, since "help" 
necessarily looks at all packages.

> Note that Distribution objects cache their entry point information, so 
> looking for entry points repeatedly on the same WorkingSet doesn't do 
> any disk I/O.
> 
> 
>> I also want content plugins, like the internationalization plugins 
>> (but it might be Javascript or templates or whatever).  And certainly 
>> that applies to TG as well.  I've been doing this at work, but using 
>> entry points, where the entry points points to an object that is a 
>> string that points to a resource that is then fetched with 
>> pkg_resources.  Something more direct would be appreciated ;)
> 
> 
> Um, you do realize that this is *exactly* what I was campaigning for on 
> the Web-SIG when I said, "we need a resource deployment standard", don't 
> you?  :)  

Yes, and I completely agreed with you!  Though I think it's a little 
larger of a problem than can be solved with eggs alone, unless everyone 
creates eggs for all their content, even instance-specific content.  The 
developer overhead for that seems too large, since much of the content 
won't be produced by developers.  It's also something that isn't even 
Python specific (kind of by definition -- this is what distinguishes 
content from code).

> This is relevant for me for Chandler too; I'd like for us to 
> be able to release our coming egg translations and resources library as 
> an implementation of a standard, rather than as a Chandler-only thing.  
> I suppose it technically goes beyond the Web-SIG at that point and 
> should maybe go to i18n-SIG, or maybe here if distutils/setuptools is 
> part of it.  OTOH, web framework and template system authors are among 
> the most important stakeholders for such an effort, since they are the 
> most readily identified cases of existing systems that would need to 
> implement such a standard.
> 
> However, it may be that it's a big enough discussion with a diverse 
> enough set of stakeholders that it deserves its own (hopefully time 
> limited) SIG.  Anyway, I have more cycles that I can devote to a 
> resource deployment standard than I can to a templating standard; the 
> former has direct impact on my "day job" work, and the latter does not.

Right now I suspect many possible stakeholders won't see how it applies 
to their cases, or feel ready to take that discussion on.  It's still 
too abstract, unless perhaps it can be phrased in terms of some existing 
standard (which may not be a Python-specific standard at all).  Kind of 
like how WSGI was phrased in terms of CGI.

I am not familiar enough with this stuff to have any idea what such 
standards exist.  Seems like internationalization is the most common 
such case, but even there people aren't utilizing that functionality to 
the degree they could (e.g., internationalization should provide hooks 
for customizing the text of an application, but it isn't used for that 
very often).

>> I like entry point groups, and applying the same idea to resources 
>> (but without the indirection) would be great.
> 
> 
> Again, this is precisely what I wanted to do with a resource deployment 
> standard.  (And note also that such a standard would make a more 
> standardized approach to the find_template issue much more 
> possible/practical -- another reason I thought we should tackle 
> deployment before rendering.)

My hope is that find_template can be implemented with such a resource 
deployment standard, even if the template stuff happens first.  And if 
that is possible, that would give the resource deployment standard a 
kind of instance applicability to a real problem, which is a bonus.

>> Hmm... and reading over this, I realize that a "plugin directory" is 
>> not that different from the virtual python setups, and I think that 
>> kind of setup is really the only good way to deploy web applications.  
>> "Working set" -- which is already an idea in setuptools -- is probably 
>> the concept that encompasses both uses.  As a developer, I'd like it 
>> to be easier and safer to change my working set -- right now I have 
>> hacks in sitecustomize and distutils to make that easy.  This 
>> switching of working sets is basically what Jim wants in a Zope 
>> Instance as well.
> 
> 
> Ideas welcome, especially ones with working implementations.  ;)

Yeah, I'm still trying to figure out what I want ;)  Heck, I'm only half 
way into articulating what I see as the current outstanding problems.

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org