[Python-Dev] PEP 376 proposed changes for basic plugins support

[M.-A. Lemburg]

Michael Foord wrote:
> On 02/08/2010 20:36, M.-A. Lemburg wrote:
>> Michael Foord wrote:
>>   
>>> On 02/08/2010 13:31, exarkun at twistedmatrix.com wrote:
>>>     
>>>> On 12:21 pm, mal at egenix.com wrote:
>>>>       
>>>>> Tarek Ziad� wrote:
>>>>>         
>>>>>> On Mon, Aug 2, 2010 at 3:06 AM, P.J. Eby<pje at telecommunity.com> 
>>>>>> wrote:
>>>>>> ..
>>>>>>           
>>>>>>> So without specific examples of why this is a problem, it's hard to
>>>>>>> see why
>>>>>>> a special Python-specific set of configuration files is needed to
>>>>>>> resolve
>>>>>>> it, vs. say, encouraging application authors to use the available
>>>>>>> alternatives for doing plugin directories, config files, etc.
>>>>>>>              
>>>>>> I don't have a specific example in mind, and I must admit that if an
>>>>>> application does the right thing
>>>>>> (provide the right configuration file), this activate feature is not
>>>>>> useful at all. So it seems to be a bad idea.
>>>>>>
>>>>>> I propose that we drop the PLUGINS file idea and we add a new
>>>>>> metadata
>>>>>> field called Provides-Plugin
>>>>>> in PEP 345, which will contain the info I've described minus the
>>>>>> state
>>>>>> field. This will allow us to expose
>>>>>> plugins at PyPI.
>>>>>>
>>>>>> IOW, have entry points like setuptools provides, but in a metadata
>>>>>> field instead of a entry_points.txt file.
>>>>>>            
>>>>> Do we really need to make Python packaging even more complicated by
>>>>> adding support for application-specific plugin mechanisms ?
>>>>>
>>>>> Packages can already work as application plugins by simply defining
>>>>> a plugins namespace package and then placing the plugin packages
>>>>> into that namespace.
>>>>>
>>>>> See Zope for an example of how well this simply mechanism works out in
>>>>> practice: it simply scans the "Products" namespace for sub-packages
>>>>> and
>>>>> then loads each sub-package it finds to have it register itself with
>>>>> Zope.
>>>>>          
>>>> This is also roughly how Twisted's plugin system works.  One drawback,
>>>> though, is that it means potentially executing a large amount of
>>>> Python in order to load plugins.  This can build up to a significant
>>>> performance issue as more and more plugins are installed.
>>>>
>>>>        
>>> unittest will solve this problem by having plugins explicitly enabled in
>>> its own configuration system, and possibly managed through a separate
>>> tool like a plugins subcommand. The full package list will *only* need
>>> to be scanned when managing plugins, not during normal execution.
>>>
>>> Having this distutils2 supported "plugin declaration and discovery" will
>>> be extremely useful for the unittest plugin system. Given that plugins
>>> may need configuring after installation, and tools that handle both
>>> activation and configuration can be provided, it doesn't seem a heavy
>>> cost.
>>>
>>> The downside to this is that installing and activating plugins are two
>>> separate steps. Given that each project can have a different set of
>>> plugins enabled I don't see a way round it.
>>>      
>> You might want to take a look at the Trac plugin system which
>> works in more or less the same way:
>>
>> http://trac.edgewall.org/wiki/TracPlugins
>>
>>
>>    
> 
> Ouch. I really don't want to emulate that system. For installing a
> plugin for a single project the recommended technique is:
> 
>     * Unpack the source. It should provide a setup.py.
>     * Run:
> 
>       $ python setup.py bdist_egg
> 
>     Then you will have a *.egg file. Examine the output of running
> python to find where this was created.
> 
>     Once you have the plugin archive, you need to copy it into the
> plugins directory of the project environment
> 
> For global plugins it just uses entry points, which is similar to the
> functionality we are suggesting adding... However note:
> 
>     Unlike plugins installed per-environment, you'll have to explicitly
> enable globally installed plugins via trac.ini.
> 
> Really this sounds *astonishingly* like the system we are proposing. :-)
> (Global discovery with per-application choice about whether or not
> installed plugins are actually used).

The difference being that Trac is usually hosted using a
separate Python installation, so the pre-application choice is
really a per-Trac instance choice.

But yes, the system you are proposing does sound a lot like
what Trac uses and it works well - for that one application.

>> Since applications tend to have a rather diverse set of needs for
>> plugins, I don't think we should add plugins support to PEP 376.
> 
> We are really just suggesting adding entry points.

Tarek's email sounded a lot like the attempt to come up
with a universal plugin system, both in terms of managing
installed plugins and their configuration.

Perhaps I've just missed some twist in the thread :-)

>> Users of applications will not want to edit a single configuration
>> file to maintain plugins of many different applications
> 
> This we are not proposing. Nor were we ever proposing it. The single
> file that was proposed (and in my understanding is no longer proposed)
> was to be maintained by distutils2 *anyway*.

Sorry, I was refering to the plugins.cfg file used for
enabling the plugins, not the PLUGINS file used by
installers.

http://mail.python.org/pipermail/python-dev/2010-August/102627.html

>> (they might
>> break some other application doing so) and sys admins
>> will have trouble with such a setup as well (they usually want to
>> have control over which plugins get used for various reasons).
>>
>> In the end, you'd have a system wide plugin configuration (maintained
>> by the sys admin), a per user one (with local customizations) and a
>> per application one (providing application-specific defaults) -
>> which only increases complexity and doesn't really solve anything.
> 
> We simply provide information about the availability of plugins. System
> administrators or users can control the use of this information (and the
> plugins) as per their own policies.
> 
>> Instead, I'd suggest to let each application do its own little thing
>> to manage plugins, in a complex or simple way, with or without
>> configuration, and have them all live happily side-by-side.
>>
>> The stdlib should really only provide tools to applications and
>> make useful suggestions, not try to enforce application design
>> choices. I think that's simply out of scope for the stdlib
>>
>>    
> Well, a tool for application developers is pretty much all that is being
> proposed.

Right, but one which has consequences for users of applications
relying on the feature.

setuptools was also "just" a tool for application developers,
but one which had some serious side-effects for users.

Let's please not play the same trick again and be more careful
about the user side of things, e.g. plugin configuration should
not be part of a Python packaging system.

Plugin discovery is useful, but doesn't really require yet another
lookup file. The few bits of extra information could easily be
placed into the distribution meta-data of PEP 345.

Perhaps the main motivation behind adding a new PLUGINS file is to
reduce the overhead of having to scan dozens of meta-data .dist-info
directories ?!

If that's the case, then it would be better to come up with an
idea of how to make access to that meta-data available in a less
I/O intense way, e.g. by having pip or other package managers update
a central SQLite database cache of the data found on disk.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 02 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/