[Distutils] Add additional file categories for distutils, setuptools, wheel

[Daniel Holth]

On Sat, Apr 18, 2015 at 7:26 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 18 April 2015 at 22:24, Daniel Holth <dholth at gmail.com> wrote:
>> with 'data' usually meaning '/' or the root of the virtualenv
>
> On Windows, it means sys.prefix.
>
> In a user install, it means
>
> - %APPDATA%\Python on Windows
> - ~/Library/$PYTHONFRAMEWORK/x.y on Darwin (is that OSX?)
> - ~/.local on Unix
>
> See sysconfig.py for the full details, but it's far more complex than
> just "/ or the root of the virtualenv"...
>
>> This works fairly well for Python libraries, but anyone packaging
>> applications for a Linux distribution is required to follow the
>> filesystem hierarchy standard or FHS.
>
> Serious question - should people packaging applications for a Linux
> distribution be using wheels in the first place? They are a Python
> packaging format. Are there any examples on PyPI of the sort of
> distribution you mean here? Or is this something that would be needed
> by in-house projects, rather than public ones on PyPI?
>
>> It would help Linux distribution package maintainers and Python
>> application (not library) developers if wheel had better support for
>> installing files into the FHS, but it would help everyone else who
>> wanted to generate cross-platform packages if the FHS was not
>> hardcoded as paths in the data/ category. To that end we should come
>> up with some additional categories that map, or do not map, to the FHS
>> based on the end user's platform.
>
> Is this a purely Linux issue? Or do the same problems apply for people
> packaging applications for Windows? Are the packages involved expected
> to be platform-specific? (In other words, would they need to work on
> both Linux and Windows, or can we assume that any project needing this
> feature wouldn't support Windows anyway?)
>
> Again, specific examples of projects needing this functionality would
> help, here.
>
>> Bento has a list based on the GNU paths, and allows new paths to be defined:
>>
>> prefix: install architecture-independent files
>> eprefix: install architecture-dependent files
>> bindir: user executables
>> sbindir: system admin executables
>> libexecdir: program executables
>> sysconfdir: read-only single-machine data
>> sharedstatedir: modifiable architecture-independent data
>> localstatedir: modifiable single-machine data
>> libdir: object code libraries
>> includedir: C header files
>> oldincludedir: C header files for non-gcc
>> datarootdir: read-only arch.-independent data root
>> datadir: read-only architecture-independent data
>> infodir: info documentation
>> localedir: locale-dependent data
>> mandir: man documentation
>> docdir: documentation root
>> htmldir: html documentation
>> dvidir: dvi documentation
>> pdfdir: pdf documentation
>> psdir: ps documentation
>>
>> I would like to add Bento's list to wheel and to setuptools.
>
> These look highly Unix-centric. If this is to solve a purely Linux
> problem, that's fine. But in that case can we please make it clear
> when defining this feature that using it explicitly makes a package
> non-portable to Windows? If it's intended to be cross-platform, I'd
> rather the targets weren't so tied to Unix usage (mandir is
> meaningless on Windows, I can't see Windows users wanting dvidir or
> psdir, but the lack of a chmdir is an oversight, sbindir (which
> probably corresponds to C:\Windows) should be split to cover
> C:\Windows, C:\Windows\System, and C:\Windows\System32. And so on...
>
> Examples of where this functionality is needed would be great. I'm
> worried that we'll over-engineer a solution otherwise.

Remember when distutils-sig didn't believe that Windows users actually
lacked compilers?

graphite-web and carbon are two examples of Python applications with
lots of data files, that want to install config files into an obvious
place so you can edit them and get the applications up and running.
Because Python packaging has such terrible support for applications
that are not libraries, it's unnecessarily hard to get these packages
up and running -- especially if you want to put them in a virtualenv.

Unfortunately data_files has been so broken for so long that most
Python packages, especially the libraries that I actually use, avoid
the feature entirely.