[Distutils] Twisted plugin system and Python packaging

[Tristan Seligmann]

This conversation originally started in a bug report against pip[1],
but I'm moving it here since it seems this might be a better venue;
you may wish to read the bug log for interest, but I'll summarize the
issue from scratch anyway.

Twisted has a plugin system (twisted.plugin[2]) which is used by
Twisted itself, as well as other projects (such as Axiom[3] and
Dosage[4]) to allow for pluggable extensibility. The implementation
looks for modules in a plugins package and then looks in those modules
for objects providing (in the zope.interface sense) the IPlugin
interface. For the sake of simplifying this explanation, I will only
refer to twisted.plugins in the rest of this mail, but assume that
everything applies similarly to plugin packages in other projects (eg.
axiom.plugins and dosage.plugins).

twisted/plugins/__init__.py sets __path__ as follows:

from twisted.plugin import pluginPackagePaths
__path__.extend(pluginPackagePaths(__name__))

The implementation of pluginPackagePaths[5] loops through the
directories on sys.path, and includes DIR/twisted/plugins/ for every
DIR on sys.path, *except* when DIR/twisted/plugins/__init__.py exists.
This means that if you drop a project dir into PYTHONPATH while
developing, the plugins from your project will be picked up, even
though they're not installed into the copy of Twisted you're using;
you can also have a project shipping plugins installed in
~/.local/lib/pythonX.Y/site-packages even though you're using a
site-wide install of Twisted in /usr/lib/pythonX.Y/site-packages.
However, if you have multiple versions of Twisted on sys.path (because
you have a newer version installed locally overriding the site-wide
one, for example), plugins from a different version of Twisted won't
be accidentally picked up.

Given the way __path__ is set, if you have a project that ships
Twisted plugins, it would be possible to install it to
.../site-packages/MyProject/myproject/... +
.../site-packages/MyProject/twisted/plugins/myproject_plugin.py, and
then include this in a .../site-packages/MyProject.pth. However, the
usual way to install your project is to install it to
.../site-packages/myproject/... +
.../site-packages/twisted/plugins/myproject_plugin.py; in other words,
directly into the existing Twisted installation. If you include
"twisted.plugins" in your setup.py, then this works fine with
distutils "setup.py install" as well as "pip install"; setuptools
"setup.py install" will install everything into an egg, which will
also work due to the way __path__ is set. However, since "twisted"
ends up in top_files.txt in the egg-info, "pip uninstall" will blow
away your whole Twisted install when uninstalling a project shipping
Twisted plugins that was installed with "pip install". Ironically, if
the project was installed with setuptools "setup.py install", there is
no problem, since pip just removes the egg that was installed, and
nothing else.

In the pip bug report, Carl Meyer suggests using the setuptools
"namespace packages" feature; this works around the immediate problem
(pip uninstall doesn't blow away your Twisted install anymore), but
causes somer othe problems of its own. First of all, code in
__init__.py is not functional for a namespace package, so the __path__
functionality described above ceases to function. Additionally,
twisted.plugins cannot be declared as a namespace package without also
declaring twisted as a namespace package, which thus also affects
twisted/__init__.py; the code in this module exports an __version__
attribute (identifying the version of Twisted) as well as importing
twisted.python.compat to ensure that various backwards compatibility
monkeypatches are installed.

So, how should Twisted and Twisted-related projects be packaged in
order to avoid these issues? Please bear in mind that the current
plugin system in Twisted was first introduced around March 2005
(replacing the even older plugin system in use at the time, I
believe), thus there are quite a number of users relying on this code;
any changes would need to be backwards-compatible to avoid causing
problems for all of the existing projects and users relying on the
functionality.

[1] https://github.com/pypa/pip/issues/355

[2] http://twistedmatrix.com/documents/current/core/howto/plugin.html

[3] https://launchpad.net/divmod-axiom

[4] https://launchpad.net/dosage

[5] http://twistedmatrix.com/trac/browser/trunk/twisted/plugin.py#L225
-- 
mithrandi, i Ainil en-Balandor, a faer Ambar