[Python-Dev] PEP 376 proposed changes for basic plugins support

[M.-A. Lemburg]

Michael Foord wrote:
> On 03/08/2010 09:28, M.-A. Lemburg wrote:
>> P.J. Eby wrote:
>>   
>>> At 10:37 PM 8/2/2010 +0200, M.-A. Lemburg wrote:
>>>     
>>>> If that's the case, then it would be better to come up with an
>>>> idea of how to make access to that meta-data available in a less
>>>> I/O intense way, e.g. by having pip or other package managers update
>>>> a central SQLite database cache of the data found on disk.
>>>>        
>>> Don't forget system packaging tools like .deb, .rpm, etc., which do not
>>> generally take kindly to updating such things.  For better or worse, the
>>> filesystem *is* our "central database" these days.
>>>      
>> I don't think that's a problem: the SQLite database would be a cache
>> like e.g. a font cache or TCSH command cache, not a replacement of
>> the meta files stored in directories.
>>
>> Such a database would solve many things at once: faster access to
>> the meta-data of installed packages, fewer I/O calls during startup,
>> more flexible ways of doing queries on the meta-data, needed for
>> introspection and discovery, etc.
>>    
> 
> Sounds good as an "optional extra" (i.e. it should be safe to completely
> delete the cache file and still have everything work) to me. If the API
> for using the feature is worked out well first this could be done as a
> behind the scenes optimisation...

True, but is also allows more freedom in using existing resources:

The data put into the PLUGINS file is essentially not needed, since
this can be had from the meta-data (provided this gets some extra
fields for describing the plugin nature).

>>> Btw, while adding PLUGINS to PEP 376 is a new proposal, it's essentially
>>> another spelling of the existing entry_points.txt used by eggs; it
>>> changes the format to csv instead of .ini, and adds "description" and
>>> "type" fields, but drops requirements information and I'm not sure if it
>>> can point to arbitrary objects the way entry_points.txt can.
>>>
>>> Anyway, entry_points.txt has been around enough years in the field that
>>> the concept itself can't really be called "new" - it's actually quite
>>> proven.  Checking
>>> http://nullege.com/codes/search/pkg_resources.iter_entry_points/call , I
>>> find 187 modules using just that one entry points API.
>>>
>>> Some projects do have more than one module loading plugins, but the
>>> majority of those 187 appear to be different projects.
>>>
>>> Note that that's modules *loading plugins*, not plugins being
>>> provided...  so the total number of PyPI projects using entry points in
>>> some way is likely much higher, once you add in the plugins that these
>>> 187 lookups are, well, looking up.
>>>      
>> setuptools entry points are just one way of doing plugins. There
>> are other such systems that work well and which do not require
>> any special administration or setup, simply because the application
>> using the plugins defines the plugin protocol.
>>    
> Right, and those won't magically stop working if this proposal is
> implemented.

Right, but the proposal does add an extra burden on the package
manager tools and it codifies one specific way of implementing
plugins.

>> Since you are into comparing numbers, you might want to count
>> the number of Zope plugins that are available on PyPI and its plugin
>> system has been around much longer than setuptools has been.
>> I don't think that proves anything, though.
>>
>> I simply don't see a good reason to complicate the Python
>> packaging system by trying to add a particular plugin support
>> to it.
>>
>> Plugins are application scope features and should be treated
>> as such.
>>
>>    
> The fact is that entry points *are* widely used and solve a problem that
> you *can't* solve without some feature like this. The success of entry
> points demonstrates their utility (and you talk vaguely about 'problems'
> setuptools caused without any concrete examples - do you know of any
> *specific* difficulties with entry points?).

Not specific to entry points, but I do know a lot about of problems
that setuptools has introduced (and didn't want to start yet another
flame war based on these ;-).

> I doubt I will change your mind, but the bottom line is that if you
> don't like this feature you don't have to use it. For applications that
> want it (like the unittest plugin system) it will be *enormously* useful
> and *reduce* complexity for the user (by allowing simpler plugin
> management tools).

Sure and those tools can use such a system. No question there :-)

Maybe I just have to spell things out more clearly:

I support the idea of adding better query tools to the
installed package meta-data to make it easier to write plugin
systems or simplify existing ones.

I don't support the idea of using central configuration files
for plugins that span multiple applications (this reminds me
a lot of the early Windows win.ini file days and all the problems
this caused back then). It's better to have per application
configurations, implemented in a way that is suitable for
the application (e.g. some applications might want to store
plugin data in a database, provide GUI tools to enable/disable
plugins, etc.).

I also don't think it's necessary to complicate things to get this
extra functionality:

If you look at the proposal, it is really just about adding a
new data store to manage a certain package type called "plugins".
Next time around, someone will want to see support for "skins" or
"themes". Then perhaps identify "script" packages, or
"application" packages, or "namespace" packages, or "stubs", etc.
All this can be had by providing this kind of extra
meta-information in the already existing format.

If we add a new extra file to be managed by the package
managers every time someone comes up with a new use case,
we'd just clutter up the disk with more and more CSV file
extracts and make PEP 376 more and more complex.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 03 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/