[Python-Dev] PEP 376 proposed changes for basic plugins support

[P.J. Eby]

At 01:53 PM 8/2/2010 +0000, exarkun at twistedmatrix.com wrote:
>On 01:27 pm, mal at egenix.com wrote:
>>exarkun at twistedmatrix.com wrote:
>>>This is also roughly how Twisted's plugin system works.  One drawback,
>>>though, is that it means potentially executing a large amount of Python
>>>in order to load plugins.  This can build up to a significant
>>>performance issue as more and more plugins are installed.
>>
>>I'd say that it's up to the application to deal with this problem.
>>
>>An application which requires lots and lots of plugins could
>>define a registration protocol that does not require loading
>>all plugins at scanning time.

Just for the record, solving this problem is precisely what entry 
points are for: they provide a discovery mechanism that doesn't 
require importing anything until you actually need it.

>It's not fixable at the application level, at least in Twisted's 
>plugin system.  It sounds like Zope's system has the same problem, 
>but all I know of that system is what you wrote above.

I don't know about Zope in general, but there are certainly Zope 
corp. projects that use entry points instead of namespaces (buildout, 
for one), and I believe that there's been a long time push to move 
third-party code out of the common namespace package.  i.e., AFAIK, 
Zope 3 doesn't use package namespaces as a primary method of extension.


>   The cost increases with the number of plugins installed on the 
> system, not the number of plugins the application wants to load.

Pretty much any plugin discovery system is going to scale that way, 
but entry points only require file reads rather than imports, and 
have a shared cache for all code in use by the application.  So if, 
say, Twisted uses entry points and an application running on Twisted 
also uses entry points, the loading cost is only paid once for both 
sets of entry points inspected.