[Distutils] elements of setuptools/wheel/FHS file categories proposal

[Paul Moore]

On 20 April 2015 at 02:22, Daniel Holth <dholth at gmail.com> wrote:
> - Add more install paths (categories) to the set currently used by
> distutils/sysconfig
> https://docs.python.org/3/library/sysconfig.html#installation-paths .
> Define sensible defaults for each platform.

+0. I have no personal use for extra categories, but I gather others do.

> - Allow the paths to be individually overridden for each installed
> package, in the same way that python setup.py install
> --install-purelib=/some-directory can override the categories we have
> now.

-0. I'm not clear this is needed, but if it is I don't mind as long as
it's as rare as it presently is. I am concerned about the implication
that it's package authors that set this, and users won't have control
(note that pip install doesn't offer an interface to this - there's
--install-option but that's sdist-only so not applicable).

> - Record the { category : path } mapping used during installation.

-1. There's no clear use case (except see below) and it implies that
each package having its own mapping is the norm, where I would prefer
a standard mapping for all cases with overrides being a vanishingly
rare need. Basically, like existing paths such as the header
directory.

> - Provide an API mapping (distribution name, category, relative path
> within category) to help applications using data that is only
> accessible via the module loader.
> https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceLoader

+0. Ideally, this should be in the stdlib, but I accept that is
impractical. I'm strongly against "installer runtime support" modules,
it feels too like the setuptools/pkg_resources problem (developer or
end user tool?). If necessary,then this should be in a completely
separate package, just providing the runtime support, and not bundled
with any particular installer. Note that for interoperability, this
implies that the PEP needs to be very clear on all details, to avoid
the API being defined by the implementation. Also, I'd prefer an
interoperability-style "how the data is made available" standard,
allowing projects to write their won API for it, rather than a "you
have to use X to get at your data" style.

The data for the mapping sounds like it's something that should be
stored in the package's metadata 2.0 API (as a metadata extension). We
don't need a custom approach for this data.

> - Provide an API mapping the same to paths on the filesystem.

-0. Like pkg_resources.resource_filename. Donald was proposing an
extension to the pkgutils API to do something similar as well. We
don't need even more of these APIs, it should just be solved once in a
general manner. For files not in package_data, the metadata API should
provide enough to allow projects to extract the information manually.
Someone could provide a simple helper module on PyPI to do that, but
it shouldn't be *necessary*.

> - Make the recorded mapping available in a predictable location, so it
> can be perhaps understood by non-Python code.

-1. Metadata 2.0 extensions seem to me to be the correct place for this.

> - Allow setup.py's setup() call to install files relative to each
> defined category.

Conditional +1. As in, if we do define such categories, setup.py
*must* support them, by definition. I presume here we'd be talking
about setuptools rather than distutils.

> - Extend Python binary package formats to support the new categories,
> so the { category : path } mapping can be set at install time and not
> at build time.

Conditional +0. Binary formats (wheel) must support this if defined,
but setting mappings at install time seems optional to me. The
defaults should be good enough for 99% of use cases, and I'd want
installer support for changing the mappings to be optional (although
clearly a quality-of-implementation issue - I'd expect pip to include
it in some form, although maybe via a variant of something generic
like --install-option).

One further note - when I mention Metadata 2.0 above, I'm thinking of
the runtime data installed into a system with the package, not the
static data exposed by PyPI (although having "which install locations
does this package use" recorded in the static data would be good for
ease of auditing). I'm not sure how well the runtime side of Metadata
2.0 has been thought through yet (in terms of APIs to read the data,
installers adding data at install time) but that doesn't alter the
fact that I think this is where such data should go. Also, I'm not
trying to kill this proposal by making it depend on Metadata 2.0,
rather pointing out that this is the sort of thing we need to push
Metadata 2.0 forward for,so that we don't invent yet more custom and
short-term data formats. I do think it's important to get this right
if we're doing it, and that means implementing the appropriate parts
of Metadata 2.0 to support it, so it is a *partial* dependency (muck
like the packaging library has implemented the version and specifier
parts, so that pip can use them).

Paul