[Distutils] Symlinks vs API -- question for developers

[Toshio Kuratomi]

So I have a question for all the developers on this list.  Philip thinks
that using symlinks will drive adoption better than an API to access
package data.  I think an API will have better adoption than a symlink
hack.  But the real question is what do people who maintain packages
think?  Since Philip's given his reasoning, here's mine:

1) Philip says that with symlinks distributions will likely have to
submit patches to the build scripts to tag various files as belonging to
certain categories.  If you, as an upstream are going to accept a patch
to your build scripts to place files in a different place wouldn't you
also accept a patch to your source code to use a well defined API to
pull files from a different source?  This is a distribution's bread and
butter and if there's a small, useful, well-liked, standard API for
accessing data files you will start receiving patches from distributions
that want to help you help them.

2) Symlinks cannot be used universally.  Although it might not be common
to want an FHS style install in such an environment, it isn't unheard
of.  At one time in the distant past I had to use cygwin so I know that
while this may be a corner case, it does exist.

3) The primary argument for symlinks is that symlinks are compatible
with __file__.  But this compatibility comes at a cost -- symlinks can't
do anything extra.  In a different subthread Philip argues that
setuptools provides more than distutils and that's why people switch and
that the next generation tool needs to provide even more than
setuptools.  Symlinks cannot do that.

4) In contrast an API can do more:  It can deal with writable files. On
Unix, persistent, per user storage would go in the user's home
directory, on other OS's it would go somewhere else.  This is
abstractable using an API at runtime but not using symlinks at install time.

5) cross package data.  Using __file__ to detect file location is
inherently not suitable for crossing package boundaries.  Egg
Translations would not be able to use a symlink based backend to do its
work for this reason.

6) zipped eggs.  These require an API.  So moving to symlinks is
actually a regression.

7) Philip says that the reason pkg_resources does not see widespread
adoption is that the developer cost of using an API is too high compared
to __file__.  I don't believe that the difference between file and API
is that great.  An example of using an API could be something like this:

Symlinks::
  import os
  icondirectory = os.path.join(os.path.basename(__file__), 'icons')

API::
  import pkgdata
  icondirectory = pkgdata.resource(pkg='setuptools', \
      category='icon', resource='setuptools.png')

Instead I think the data handling portion of pkg_resources is not more
widely adopted for these reasons:

* pkg_resources's package handling is painful for the not-infrequent
corner cases.  So people who have encountered the problems with
require() not overriding a default or not selecting the proper version
when multiple packages specify overlapping version ranges already have a
negative impression of the library before they even get to the data
handling portion.

* pkg_resources does too much: loading libraries by version really has
nothing to do with loading data for use by a library.  This is a
drawback because people think of and promote pkg_resources as a way to
enable easy_install rather than a way to enable abstraction of data
location.

* The only benefit (at least, being promoted in the documentation) is to
allow zipped eggs to work.  Distributions have no reason to create
zipped eggs so they have no reason to submit patches to upstream to
support the pkg_resources api.

* Distributions, further, don't want to install all-in-one egg
directories on the system.  The pkg_resources API just gets in the way
of doing things correctly in a distribution.  I've had to patch code to
not use pkg_resources if data is installed in the FHS mandated areas.
Far from encouraging distributions to send patches upstream to make
modules use pkg_resources this makes distributions actively discourage
upstreams from using it.

* The API isn't flexible enough.  EggTranslations places its data within
the metadata store of eggs instead of within the data store.  This is
because the metadata is able to be read outside of the package in which
it is included while the package data can only be accessed from within
the package.


8) To a distribution, symlinks are just a hack.  We use them for things
like php web apps when the web application is hardcoded to accept only
one path for things (like the writable state files being intermixed with
the program code).  Managing a symlink farm is not something
distributions are going to get excited over so adoption by distributions
that this is the way to work with files won't happen until upstreams
move on their own.

Further, since the install tool is being proposed as a separate project
from the metadata to mark files, the expectation is that the
distributions are going to want to write an install tool that manages
this symlink farm.  For that to happen, you have to get distributions to
be much more than simply neutral about the idea of symlinks, you have to
have them enthused enough about using symlinks that they are willing to
spend time writing a tool to do it.


So once again, I think this boils down to these questions: if we have a
small library whose sole purpose is to abstract a data store so you can
find out where a particular non-code file lives on this system will you
use it?  If a distribution packager sends you a patch so the data files
are marked correctly and the code can retrieve their location instead of
hardcoding an offset against __file__ will you commit it?

-Toshio

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20081017/b22150d4/attachment.pgp>