[Python-Dev] PEP 376 proposed changes for basic plugins support

[Michael Foord]

On 02/08/2010 20:36, M.-A. Lemburg wrote:
> Michael Foord wrote:
>    
>> On 02/08/2010 13:31, exarkun at twistedmatrix.com wrote:
>>      
>>> On 12:21 pm, mal at egenix.com wrote:
>>>        
>>>> Tarek Ziad� wrote:
>>>>          
>>>>> On Mon, Aug 2, 2010 at 3:06 AM, P.J. Eby<pje at telecommunity.com>  wrote:
>>>>> ..
>>>>>            
>>>>>> So without specific examples of why this is a problem, it's hard to
>>>>>> see why
>>>>>> a special Python-specific set of configuration files is needed to
>>>>>> resolve
>>>>>> it, vs. say, encouraging application authors to use the available
>>>>>> alternatives for doing plugin directories, config files, etc.
>>>>>>              
>>>>> I don't have a specific example in mind, and I must admit that if an
>>>>> application does the right thing
>>>>> (provide the right configuration file), this activate feature is not
>>>>> useful at all. So it seems to be a bad idea.
>>>>>
>>>>> I propose that we drop the PLUGINS file idea and we add a new metadata
>>>>> field called Provides-Plugin
>>>>> in PEP 345, which will contain the info I've described minus the state
>>>>> field. This will allow us to expose
>>>>> plugins at PyPI.
>>>>>
>>>>> IOW, have entry points like setuptools provides, but in a metadata
>>>>> field instead of a entry_points.txt file.
>>>>>            
>>>> Do we really need to make Python packaging even more complicated by
>>>> adding support for application-specific plugin mechanisms ?
>>>>
>>>> Packages can already work as application plugins by simply defining
>>>> a plugins namespace package and then placing the plugin packages
>>>> into that namespace.
>>>>
>>>> See Zope for an example of how well this simply mechanism works out in
>>>> practice: it simply scans the "Products" namespace for sub-packages and
>>>> then loads each sub-package it finds to have it register itself with
>>>> Zope.
>>>>          
>>> This is also roughly how Twisted's plugin system works.  One drawback,
>>> though, is that it means potentially executing a large amount of
>>> Python in order to load plugins.  This can build up to a significant
>>> performance issue as more and more plugins are installed.
>>>
>>>        
>> unittest will solve this problem by having plugins explicitly enabled in
>> its own configuration system, and possibly managed through a separate
>> tool like a plugins subcommand. The full package list will *only* need
>> to be scanned when managing plugins, not during normal execution.
>>
>> Having this distutils2 supported "plugin declaration and discovery" will
>> be extremely useful for the unittest plugin system. Given that plugins
>> may need configuring after installation, and tools that handle both
>> activation and configuration can be provided, it doesn't seem a heavy cost.
>>
>> The downside to this is that installing and activating plugins are two
>> separate steps. Given that each project can have a different set of
>> plugins enabled I don't see a way round it.
>>      
> You might want to take a look at the Trac plugin system which
> works in more or less the same way:
>
> http://trac.edgewall.org/wiki/TracPlugins
>
>
>    

Ouch. I really don't want to emulate that system. For installing a 
plugin for a single project the recommended technique is:

     * Unpack the source. It should provide a setup.py.
     * Run:

       $ python setup.py bdist_egg

     Then you will have a *.egg file. Examine the output of running 
python to find where this was created.

     Once you have the plugin archive, you need to copy it into the 
plugins directory of the project environment

For global plugins it just uses entry points, which is similar to the 
functionality we are suggesting adding... However note:

     Unlike plugins installed per-environment, you'll have to explicitly 
enable globally installed plugins via trac.ini.

Really this sounds *astonishingly* like the system we are proposing. :-) 
(Global discovery with per-application choice about whether or not 
installed plugins are actually used).

> Since applications tend to have a rather diverse set of needs for
> plugins, I don't think we should add plugins support to PEP 376.
>    

We are really just suggesting adding entry points.
> Users of applications will not want to edit a single configuration
> file to maintain plugins of many different applications

This we are not proposing. Nor were we ever proposing it. The single 
file that was proposed (and in my understanding is no longer proposed) 
was to be maintained by distutils2 *anyway*.

> (they might
> break some other application doing so) and sys admins
> will have trouble with such a setup as well (they usually want to
> have control over which plugins get used for various reasons).
>
> In the end, you'd have a system wide plugin configuration (maintained
> by the sys admin), a per user one (with local customizations) and a
> per application one (providing application-specific defaults) -
> which only increases complexity and doesn't really solve anything.
>
>    

We simply provide information about the availability of plugins. System 
administrators or users can control the use of this information (and the 
plugins) as per their own policies.

> Instead, I'd suggest to let each application do its own little thing
> to manage plugins, in a complex or simple way, with or without
> configuration, and have them all live happily side-by-side.
>
> The stdlib should really only provide tools to applications and
> make useful suggestions, not try to enforce application design
> choices. I think that's simply out of scope for the stdlib
>
>    
Well, a tool for application developers is pretty much all that is being 
proposed.

All the best,

Michael Foord

> Tarek:
>
> What you might want to do is add new type fields to PEP 345,
> making it easier to identify and list packages that work as
> plugins for applications, e.g.
>
> Type: Plugin for MyCoolApp
>
> The MyCoolApp could then use the Type-field to identify all
> installed plugins, get their installation directories, etc.
> and work on from there.
>
> Whether or not to use an installed plugin is really not
> without the scope of Python's packaging system. This is
> something the application must provide in its config file,
> together with possible additional sections to configure
> a particular plugin.
>
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.