Hi all, I had a discussion with Tarek on IRC yesterday and would like to bring some points to your attention and also write down some conclusions/ideas we had yesterday. I think that the distutils module has a serious flaw that need to be addressed before any further work is done on getting PyPi in a better shape. The main issue is, that there are many ways to customise the installation of a distribution but absolutely nothing to retrieve this information within the setup script or within your programs/libraries. tl;dr: save prefixes in .egg-info/PREFIX and create methods for Distribution to retrieve these and data files Install prefixes / Installation schemes --------------------------------------- There are basically five installation schemes: 1. Default 2. User 3. Home 4. Prefix 5. Custom All these installation schemes taken together enable the user to install pure libraries, non-pure libraries, data files and scripts in exactly the places s/he choose. But this also means that it is impossible to make any assumptions about the locations in your code. I think it is of uttermost importance to work on an implementation that enables users to query this information: * at build time (within setup.py) * at run time (within scripts/libraries) The major problem today is that the information where files of a given distribution are placed is *only* available at build time and not saved anywhere. And even at build time you'll have to get the finalised install command to query that information, which is suboptimal and counter-intuitive. Data files ---------- Especially data files are problematic in this respect because the FHS mandates that these are *not* placed within the packages but at exactly defined places in the file system. The API available to retrieve data files in the *stdlib* pkgutil.get_data() can not cope with this and is therefore not usable. This leads to the following problems: * Linux package maintainers have to implement a lot of patches if they want to place data files in the correct places. * Python developers have to either (i) save the various locations at install time in a place where they can retrieve that information later or (ii) accept that their library/program will break if one of the fancier installation schemes is used. * Python distribution installations that violate the FHS * ... Proposal -------- I propose three things: 1. Save installation prefixes for every installed distribution 2. Define standard infixes for typical data file classes 3. Implement an API that is able to query that information/retrieve data files 1 and 3. Discussion The installation prefixes could for example be saved within a suitable file in .egg-info/ which would mandate changes to PEP 376. I could think of a PREFIX file and the following methods/attribute for e.g. the Distribution class * get_install_prefixes() -> returns a dictionary with all prefixes for this distribution * get_data_file(type, name) -> get data file of given type and name (see (see below) 2. Discussion The FHS differentiates between various classes of files and defines proper location for them. We could define platform dependent standard infixes for the following data file classes for distribution foo: * configuration etc * shared data usr/share/foo/ * readme usr/share/foo/README README, TODO, ... could be automatically discovered * examples usr/share/foo/examples * documentation usr/share/foo/doc * man files usr/share/man * variable var/lib/foo * ... There should be a set of platform dependent prefixes for all classes we agree on *and* it should be possible to change/set their default values with, for example, environment variables or command line options. I call them infixes and not prefixes here, because applications like virtualenv might want to define a different root for these files than the system default one. Tarek suggested to use this scheme like this: setup( ... data_files = [ ('$configuration', ['data/my.conf', 'data/default.conf']), ('$examples', ['doc/examples/do_foo.sample']), ('/i/want/file/here', ['custom_placed_file']), ... ] ) The placeholders will then be replaced with their current value at installation time and the locations written to .egg-info/PREFIX. This would allow/ease a couple of things, namely: * automatic placement of typical files in the right locations * much less work for packaging python applications on *nix distributions -> debhelper could take advantage of all these for example * automatic tests/fixers for different file types I think of automatic lintian style tests on PyPi here that will reject a package if it does not conform to a standard or at least warns the user about it * retrieval of certain types of data easily at run time Distribution.get_data_file('$configuration', 'my.cfg') -> file object * ... thanks! -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC