[Distutils] API for finding plugins

[Phillip J. Eby]

At 12:43 PM 2/7/2006 -0600, Ian Bicking wrote:
>I guess the problem is that for Paste Script, the startup overhead and the 
>runtime overhead are the same thing.  Like, 8 seconds to complete "paster 
>help" (which gives a listing of all available commands).  After the first 
>run it warms up and is much faster (disk caching?), but that still doesn't 
>feel good to me.  Now that it's warmed up I can't tell how slow a normal 
>command would be -- possible faster, since "help" necessarily looks at all 
>packages.

I'm assuming here that the problem is needing to import each command to get 
its description and display it?


>>>I also want content plugins, like the internationalization plugins (but 
>>>it might be Javascript or templates or whatever).  And certainly that 
>>>applies to TG as well.  I've been doing this at work, but using entry 
>>>points, where the entry points points to an object that is a string that 
>>>points to a resource that is then fetched with pkg_resources.  Something 
>>>more direct would be appreciated ;)
>>
>>Um, you do realize that this is *exactly* what I was campaigning for on 
>>the Web-SIG when I said, "we need a resource deployment standard", don't 
>>you?  :)
>
>Yes, and I completely agreed with you!

Um, suuuure you did, Mr. "Objections only on the Spec and not the Scope".  ;-)


>   Though I think it's a little larger of a problem than can be solved 
> with eggs alone,

I don't need the spec to be egg-specific, only egg-*compatible*.


>>However, it may be that it's a big enough discussion with a diverse 
>>enough set of stakeholders that it deserves its own (hopefully time 
>>limited) SIG.  Anyway, I have more cycles that I can devote to a resource 
>>deployment standard than I can to a templating standard; the former has 
>>direct impact on my "day job" work, and the latter does not.
>
>Right now I suspect many possible stakeholders won't see how it applies to 
>their cases, or feel ready to take that discussion on.  It's still too 
>abstract, unless perhaps it can be phrased in terms of some existing 
>standard (which may not be a Python-specific standard at all).  Kind of 
>like how WSGI was phrased in terms of CGI.
>
>I am not familiar enough with this stuff to have any idea what such 
>standards exist.  Seems like internationalization is the most common such 
>case, but even there people aren't utilizing that functionality to the 
>degree they could (e.g., internationalization should provide hooks for 
>customizing the text of an application, but it isn't used for that very often).

I'm not aware of any existing standards, either, but I'm thinking that 
what's needed is more of an API for resource retrieval keyed on some set of 
simple values or wildcards, with some way to aggregate search results from 
multiple sources (so that e.g. databases, eggs, and directory trees) can 
all "offer" resources on demand.

I'm thinking that you would call this API (at the low level) via something 
like:

      my_page_source = resource_set.find_resource(
          resource=['my_page'], for_project=['MyProject'],
          locale=['en','de'], layer=['some_layer', 'other_layer'],
      ).as_string()

Of course, this would in most cases be wrapped by some higher-level API 
that eliminates most of the parameters from needing to be specified (e.g. 
by a framework that knows what locales and skin layers are in effect and 
what project the requesting code is calling from).  For performance, you 
could extract subset resource sets and use them instead of querying a 
top-level resource set.

There are a lot of things that would still have to be worked out, of 
course, like precedence among parameters, wildcards, and speed/space 
efficiencies.  But the basic idea is that you just do queries on a 
relatively simple dataset.  Since providers determine where the resources 
are and how they're retrieved, this wouldn't actually require everybody to 
agree on where things are or how they're organized for input, but would 
instead only require agreeing on an abstract addressing scheme, not unlike 
an LDAP directory schema.  (Representing such a directory with good 
speed/space tradeoffs is of course one of the biggest hurdles, 
implementation-wise.)

With respect to eggs specifically, I expect that setup() would grow an 
argument to accept an input data structure of some kind, with various 
utility routines to generate the data directly from common directory 
structure layouts.  This is basically what we'd planned to do for Chandler 
message catalogs and localizable resources, but with a more simplistic data 
structure.


>My hope is that find_template can be implemented with such a resource 
>deployment standard, even if the template stuff happens first.  And if 
>that is possible, that would give the resource deployment standard a kind 
>of instance applicability to a real problem, which is a bonus.

Well, I've already got a real use case in Chandler, so it's not as though 
the real-world needs don't exist.  :)