[Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Jean-Paul Calderone]

On Tue, 07 Oct 2008 15:19:57 +0200, Josselin Mouette <joss at debian.org> wrote:
>Le mardi 07 octobre 2008 à 08:58 -0400, Jean-Paul Calderone a écrit :
>> I don't think the details of the plugin system are relevant to the topic
>> under discussion here.  The installation requirements are not unusual for
>> the most part - that a directory full of .py files be copied to the install
>> location, just as any package would be.  The one unusual thing is that
>> non-Twisted package X might want to copy files into twisted/plugins/ (if it
>> is providing plugins).
>
>This is not unusual, but other frameworks that do the same don’t break
>when you add namespace packages. In fact, the very fact that you can’t
>add namespace packages shows that you are abusing the python modules
>directory: you are shipping a twisted/plugins/foo/foo.py file which is
>not meant to be available as a twisted.plugins.foo.foo module.

I think you meant "shipping a twisted/plugins/foo.py file which is not
meant to be available as a twisted.plugins.foo module", since as far as
I know, no one is putting packages (as opposed to modules) into the
twisted.plugins package (and I don't think it works, either).

>The ability to install several incompatible versions of Twisted on the
>system is the reason I was given by glyph; if there is another reason,
>I’d be happy to hear the explanation, but the result is the same: we had
>to code a special case for Twisted.
>

Above correction aside, your understanding of the system is still flawed
(I am fully aware that this is likely due to our failure to document the
system sufficiently, so please don't interpret that as an attack).  I
think that you also misunderstood the explanation that Glyph provided.

In your earlier message, you wrote this:

> The reason is to allow plugins for several incompatible Twisted versions
> to be installed in the *same* directory.

However, no such feature is offered by the Twisted plugin system. :)

The issue which arose when Debian packaged Twisted and (for example) a
new version of Nevow which provides a plugin to Twisted is that Nevow's
plugin was installed to a location which was not the location intended
or expected by the Twisted developers or the Nevow developers.  The
expectation of the Twisted developers was that the installed directory
structure for Twisted files would look like this:

  /usr/lib/python2.5/site-packages/twisted/
  /usr/lib/python2.5/site-packages/twisted/__init__.py
  /usr/lib/python2.5/site-packages/twisted/plugins/
  /usr/lib/python2.5/site-packages/twisted/plugins/__init__.py

(obviously this is only part of what a Twisted installation would look like)

The expectations of the Nevow developers was that a file included in Nevow,
nevow_widget.py, would be installed to

  /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py

On Debian, difficulties were encountered because, while Twisted was
installed more or less to the expected location, the file in Nevow
was instead installed to

  /var/lib/python-support/python2.5/twisted/plugins/nevow_widget.py

and the infrastructure for installing the file to this location also
creates

  /var/lib/python-support/python2.5/twisted/
  /var/lib/python-support/python2.5/twisted/__init__.py
  /var/lib/python-support/python2.5/twisted/plugins/
  /var/lib/python-support/python2.5/twisted/plugins/__init__.py

This is problematic because there are now two Twisted packages in the
module search path.  Python's module system can't do anything with the
second one (which one is second depends on the order of sys.path).  If
the version created for nevow_widget.py came first, the Twisted package
itself would basically be completely broken.  Fortunately, it doesn't,
so that doesn't happen.  However, since nevow_widget.py is in the second
version, the only way it can be imported is if some extra help is supplied
to the import system.

In previous releases of Twisted, this help was given by twisted/plugins/
__init__.py.  It included an extremely primitive implementation of the
concept of namespace packages (_extremely_ primitive).  It would find all
directories named "twisted/plugins" which were in the module search path
and add them to its own __path__.  The purpose of this feature wasn't to
support installations, however.  The purpose of this feature was as a
developer convenience.  It allows developers to edit plugins in a non-
installed location (for example, their home directory) and then use them
without copying them to the installed Twisted's twisted/plugins/ directory.

In a more recent release of Twisted, this feature was tweaked slightly.
The reason for the change is that there was a significant bug in the old
implementation of namespace packages.  If a developer was using their own
copy of Twisted (for example, an SVN checkout in their home directory)
instead of the installed version, the namespace package implementation
would still load plugins from the *installed* version of Twisted.  In the
best possible case, this gave you two copies of every plugin.  In the
worst case, the installed version of Twisted wasn't compatible with the
other version of Twisted and the plugins would raise exceptions.

This is the failure condition which I believe you misinterpreted as meaning
we were trying to support any kind of versioning features.  We're not.  We're
only trying to not run code that the user (/developer) doesn't want run.

So, to fix this, we tweaked the namespace package setup code so that it will
skip any "twisted/plugins/" directory which contains an "__init__.py", since
that is a relatively strong hint that it's a real package, not one of the
developer-convenience plugin locations (which are not real packages).

Debian's Nevow package then broke, since Debian installed an __init__.py
when installing Nevow's nevow_widget.py file.

The ideal fix, from my perspective as a Twisted developer, is to install
the nevow_widget.py file into

  /usr/lib/python2.5/site-packages/twisted/plugins/

along with all the other Twisted plugin files (perhaps using a symlink, I
can't think of any reason why that would make a difference, particularly
since all the other .py files in that area are symlinks).

In other words, Debian took advantage of a feature which was intended to be
used only as a convenience for development and was never intended to assist
installation.

Have I succeeded in explaining why the Twisted plugin system isn't making
any unusual requirements of the installation system nor using the Python
module system in an unusual way?  Can I clarify any parts of this further?
Can I expect to see any improvements to the Debian packaging of Twisted and
projects which supply plugins to Twisted? :)

Jean-Paul