[Distutils] Symlinks vs API -- question for developers

Josselin Mouette joss at debian.org
Mon Oct 20 12:45:54 CEST 2008

Le dimanche 19 octobre 2008 à 19:28 -0400, Phillip J. Eby a écrit :
> >3) The primary argument for symlinks is that symlinks are compatible
> >with __file__.  But this compatibility comes at a cost -- symlinks can't
> >do anything extra.

From my point of view, compatibility with __file__ is something that is
not desired. We should discourage developers from using such a hack
instead of going on supporting it. Yes, there is a cost for this
migration. There is always a cost for developing properly, but it is a
win in the long term.

> I think Ian's already said this, but the API itself has to do 
> something more, and so far nobody's proposed an API that does 
> anything "more" than what setuptools does in this area, from the 
> developer point of view.

Part of the solution is to introduce semantics in this API. Merely
tagging files only allows to move them to a handful of usable locations
(e.g. /usr/share/$package, /usr/share/doc/$package, /usr/share/pixmaps, /etc/$package).  However, depending on the file type and what you want to use for, it may be useful in many more locations, and they will not depend on a mere tag. For example, when you use gettext, the thing you express is “get translations for the current locale”, not “load the LC_MESSAGES/fr/blah.po file”. Similarly for GTK+, the API allows to retrieve an icon at the size of a given widget, the file location being completely abstracted.

As such, the solution is probably to develop an API allowing to access
data in an abstract way, and one that can integrate properly with
existing tools relying on file locations.

Let’s take the icon example. You first need to define what is an icon
file and where it goes. It could be defined in a file looking like this:

<type name="icon">
 <attribute name="theme">
 <attribute name="size" required="true">
 <attribute name="category">
 <location system="unix">

Then, you need to flag your icons; this is what you already proposed, in
a more sophisticated way.
        foo.png: icon size="48x48" category="apps"
        foo.svg: icon size="scalable" category="apps"

Finally, in your code: 
      * if you already use GTK+, you only have to retrieve the icon
        named "foo";
      * otherwise, you need to write a helper function based on the API,
        which will allow to write things like this:
                location=builds.get_path("icon", "foo.png", category="apps", size="48x48")

I don’t think any other build system is going this far; which means it
is not necessary to go this far, but it could be a way to get developers
to abandon their hacks.

> >4) In contrast an API can do more:  It can deal with writable files. On
> >Unix, persistent, per user storage would go in the user's home
> >directory, on other OS's it would go somewhere else.  This is
> >abstractable using an API at runtime but not using symlinks at install time.
> This is all well and good, but it's actually quite orthogonal to most 
> uses of __file__ and resources today.

Not entirely orthogonal, since it goes in the same direction: a common
base to implement standard FHS and freedesktop.org locations. If all
Python applications start using standard home and configuration
directories, it’s a lot of burden that goes away for sysadmins and

> >5) cross package data.  Using __file__ to detect file location is
> >inherently not suitable for crossing package boundaries.  Egg
> >Translations would not be able to use a symlink based backend to do its
> >work for this reason.
> EggTranslations doesn't use __file__, it uses the API, so I don't see 
> how this relates.

I think this was just an example. There will be many cases where you
need to extend an existing package’s resources with your own. Using
__file__ doesn’t allow that and will require that you pass full paths
instead of using an abstract API to access the data.

> It's requiring the use of an API that's a regression wrt 
> developer-side features.

Generally, I call the deprecation of a hack in favor of a clean API an
improvement, not a regression. But that may just be your own

> >7) Philip says that the reason pkg_resources does not see widespread
> >adoption is that the developer cost of using an API is too high compared
> >to __file__.  I don't believe that the difference between file and API
> >is that great.
> It isn't; it's the *switching* cost that's high, and that's the cost 
> that needs to be minimized in order to drive adoption quickly.

You shouldn’t forget another part of the issue. If the only thing the
new specification allows is to make symbolic links to a few pre-defined
locations without any flexibility, it is very unlikely that
distributions will take the time to write any patches to it. Without
anyone to drive this feature forward, it will not be adopted at all. It
is not only a question of making the migration easy, but also on making
people want to push this migration.

> >8) To a distribution, symlinks are just a hack.  We use them for things
> >like php web apps when the web application is hardcoded to accept only
> >one path for things (like the writable state files being intermixed with
> >the program code).  Managing a symlink farm is not something
> >distributions are going to get excited over so adoption by distributions
> >that this is the way to work with files won't happen until upstreams
> >move on their own.
> We need to distinguish between "providing the ability to have a 
> low-cost transition" and "the recommended True Way".

If no one makes the low-cost transition in the end, you are going to
lose your time trying to set it up.

> >Further, since the install tool is being proposed as a separate project
> >from the metadata to mark files, the expectation is that the
> >distributions are going to want to write an install tool that manages
> >this symlink farm.  For that to happen, you have to get distributions to
> >be much more than simply neutral about the idea of symlinks, you have to
> >have them enthused enough about using symlinks that they are willing to
> >spend time writing a tool to do it.
> Well, the question is whether they prefer to have a long, drawn out 
> transition or not.  Maybe they don't care about that part, but my 
> assumption was that a replacement for setuptools/easy_install in this 
> space was desired sooner rather than later.

Currently I think we’d be more reluctant to symlinks than enthusiastic
about them. We already maintain symlink farms to deal with the insanity
of Python modules, moving files to a place that is more neutral for the
FHS (/usr/share/pysomething). The urgency is far behind us since Python
module developers didn’t see it at all at that time.

> Now, if you want to establish a transition timetable for phasing out 
> __file__ usage, deprecation, etc., based on when the API will be 
> available in the stdlib etc., publicize and bless that schedule, 
> etc...  again, these are all good ideas.
> The ONLY thing I object to is requiring it up front from day 1, 
> because then we're just shooting off a giant foot-gun wrt adoption.

No one is asking that.

BTW deprecating __file__ is far from easy, since there are valid uses
for it, and it’s not trivial to distinguish them.

: :' :      We are debian.org. Lower your prices, surrender your code.
`. `'       We will add your hardware and software distinctiveness to
  `-        our own. Resistance is futile.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Ceci est une partie de message num?riquement sign?e
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20081020/510f3e96/attachment.pgp>

More information about the Distutils-SIG mailing list