At 01:53 PM 8/2/2010 +0000, exarkun@twistedmatrix.com wrote:

On 01:27 pm, mal@egenix.com wrote:

...

exarkun@twistedmatrix.com wrote:

...

This is also roughly how Twisted's plugin system works. One drawback, though, is that it means potentially executing a large amount of Python in order to load plugins. This can build up to a significant performance issue as more and more plugins are installed.

I'd say that it's up to the application to deal with this problem.

An application which requires lots and lots of plugins could define a registration protocol that does not require loading all plugins at scanning time.

Just for the record, solving this problem is precisely what entry points are for: they provide a discovery mechanism that doesn't require importing anything until you actually need it.

It's not fixable at the application level, at least in Twisted's plugin system. It sounds like Zope's system has the same problem, but all I know of that system is what you wrote above.

I don't know about Zope in general, but there are certainly Zope corp. projects that use entry points instead of namespaces (buildout, for one), and I believe that there's been a long time push to move third-party code out of the common namespace package. i.e., AFAIK, Zope 3 doesn't use package namespaces as a primary method of extension.

The cost increases with the number of plugins installed on the system, not the number of plugins the application wants to load.

Pretty much any plugin discovery system is going to scale that way, but entry points only require file reads rather than imports, and have a shared cache for all code in use by the application. So if, say, Twisted uses entry points and an application running on Twisted also uses entry points, the loading cost is only paid once for both sets of entry points inspected.