[Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Phillip J. Eby]

At 12:43 AM 10/8/2008 +0200, Josselin Mouette wrote:
>Le mardi 07 octobre 2008 à 16:51 -0400, Phillip J. Eby a écrit :
> > >If you write a tool to do that, why not make it simply move files
> > >properly and let the code locate them, instead of adding yet another
> > >layer on top of the existing stuff? The tool will not be more
> > >complicated this way.
> >
> > I'm not sure I follow you.  What I'm proposing is that:
> >
> > 1. A BUILDS "manifest" for a project should list the files included
> > in the project by what "kind" of file they are, as well as by
> > inter-file relationships (e.g., "these 3 files are part of Python
> > package 'foo.bar' within this project, and they are static data used
> > at runtime but not read or written").
> >
> > 2. BUILDS-compatible install tools will be free to install those
> > files wherever they like (or create .debs, .rpms, etc. that do), and
> > use symlinks so that the installed project can find them in the
> > places that the developer wanted to find them.
> >
> > Is this what you're asking for?  Because it's what I've been trying
> > to say we should do.
>
>What I am afraid of is that, by adding just another layer, you will
>introduce new problems while fixing existing ones. Currently we already
>have a hard time maintaining a symlink farm, and adding a second symlink
>farm on top of it is not going to make things more understandable
>(currently, package maintainers already have a hard time understanding
>how to deal with Python modules) nor more reliable.

I guess I'm confused a bit here, since the idea is not to have 
maintainers need to understand anything but how to be able to tell 
upstream, "hey, you didn't configure file X correctly".  At least, 
that's what I'd hope would be all that's needed.  :-)


>The first kind of issues I can think of is that dpkg handles symlinks to
>directories very badly, so you’re likely to run into issues that have to
>be dealt with by hand by maintainers. I haven’t thought about it much,
>but I have the feeling that other things will break.

Actually, until I saw the Twisted plugins directory use case, I 
hadn't actually seen any use cases for symlinks to directories that 
couldn't be handled by using a directory of symlinks instead.  Are 
there any other Python packages you know of that would need a symlink 
to a directory?


>In the end, if you are designing a new packaging specification and a new
>tool,

The BUILDS manifest is neither packaging nor a tool; it's just 
information about a bunch of files.  ;-)  We don't want to design *a* 
tool, we want to make it possible for tools to be 
written.  Currently, there are a whole bunch of installation and 
maintenance tools out there built on top of setuptools, but they 
aren't necessarily very compatible with each other, and building 
tools that approach things very differently from setuptools is hard.

The idea of a specification is to allow installation tools to be 
built that apply different policies on different platforms, not to 
build The One True Tool[tm].

In other words, the goal isn't to "do the right thing", but to allow 
various useful things to be done.


> > For example, if Debian needs metadata that's not in the core BUILDS
> > metadata, it should be possible within BUILDS to include it, so that
> > it can be contributed upstream.  Likewise, it should be possible for
> > a BUILDS install tool to use a third-party supplement to get that
> > data, so that when you're initially porting a project, you could
> > write a file with the additional data while doing the initial port,
> > then submit that file upstream.
>
>If there is a need for Debian-specific metadata, it will go in the
>debian/ directory. It looks interesting to integrate it upstream only if
>it is relevant for other distributors.

If I have to explicitly mark my locale files for Debian to be happy, 
then that *is* Debian-specific metadata from my perspective as a 
cross-platform developer.

That is, I didn't say anything about metadata that was unique to 
Debian, just metadata needed by a Debian BUILDS installer to be 
happy, as opposed to a generic BUILDS installer that doesn't care 
about the FHS.  (For example, an Enthought installer for Windows 
that's also installing its own Python interpreter might want to 
consume BUILDS but it sure as heck wouldn't be following FHS 
standards; it'd have its own.)


>There is no use in being compatible with the autotools, the point is
>about stealing good ideas.
>
>The first thing that comes to mind, and that we already discussed at
>large, is the ability to specify at install time where data,
>documentation, manual pages and so on have to go.

The idea of the BUILDS "install locations" (IL) manifest is just that 
it lists the contents of the project distribution, and describes the 
logical locations that things go in.  So, part of the BUILDS spec 
would be what categories exist for the above.

It would then be the responsibility of people building 
BUILDS-consuming install tools to define the actual locations.  I say 
it that way mainly because I'm not volunteering my time to actually 
build such an install tool, nor am I necessarily volunteering to 
build *any* tools.  No sense in replacing me as the setuptools 
bottleneck to be the bottleneck on new tools.  The whole idea is to 
get system packagers and major developers together to agree on how to 
communicate installation requirements to installation tools, and then 
have everybody go back and update their tools to generate or consume 
installation data in accordance with the spec.

Of course, it's likely that rather than everybody creating their own 
install tools, there will probably be some grouping and alliances 
along platform and/or use-case lines.  For the stdlib, I expect there 
will be a simple installer with configurable locations for installing 
things, maybe with some ability to be extended to handle more complex 
policies.  There would probably also need to be BUILDS-to-wininst, 
and BUILDS-to-msi converters.


>What some Python projects using autoconf are doing, and I’d like to see
>in BUILDS, is the generation of a config.py file at installation time.
>It would be useful to specify install-time stuff like installation
>directories, optional features and installation parameters.
>
>For example it could contain:
>         version = "1.2.3"
>         builddate = "Tue 7 Oct 2008, 23:12 +0100"
>         templates_dir = "/usr/share/foobar/templates"
>         has_libamazing = True # _foobar.so is built with libamazing support
>         page_footer = "Powered by foobar"
>
>For Python-only packages, you can easily ship a default config.py file
>that works without installation:
>         version = get_version_from_changelog ()
>         templates_dir =
>         os.path.join(os.path.basedir(__file__),"templates")
>         page_footer = "Foobar (development version)"
>
>By doing such a thing, you are guaranteed to obtain easily the data you
>need from within the Python application:
>         import config
>         template = file(os.path.join(config.templates_dir),
>         "foo.template")
>
>I may be biased because I’m used to work like this, but I find this
>easier for both the developer and the distributor. In my personal
>experience, it is also more reliable.

It looks like a great idea, and I'm all for recommending people use 
it, or something like it.  I'm also fine with BUILDS having support 
for some sort of "install-time editing" like this.  I just don't want 
it to be a requirement that everybody change their ways.  Not 
everybody even uses pkg_resources' API for accessing data files yet, after all.

In my experience with setuptools adoption, anything that makes you 
have to change something outside of the packaging part is a 
significant barrier.  People wonder if they can rely on it, whether 
it will lock them in to something, they need to do testing, etc.

Sure, these things aren't REALLY that big of a hassle, but the 
psychological effect is to create hesitation.  But if you can move 
your code without changing your code, it creates a sense of safety, 
and provides incremental rewards for incremental efforts.  Thus, 
people who first moved to setuptools without needing to change their 
actual package code, were then much more likely to adopt the API 
later.  The same will be true for this.


>Another thing from the autotools that I’d like to see is seamless
>integration with pkg-config, for modules with C extensions. Bonus points
>for being able to use not only compilation flags and installation paths,
>but also macros you need to use inside the code (they could go in
>config.py).

Sorry, what is pkg-config?