[Distutils] elements of setuptools/wheel/FHS file categories proposal

[Daniel Holth]

On Mon, Apr 20, 2015 at 3:13 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 20 April 2015 at 02:22, Daniel Holth <dholth at gmail.com> wrote:
>> - Add more install paths (categories) to the set currently used by
>> distutils/sysconfig
>> https://docs.python.org/3/library/sysconfig.html#installation-paths .
>> Define sensible defaults for each platform.
>
> +0. I have no personal use for extra categories, but I gather others do.
>
>> - Allow the paths to be individually overridden for each installed
>> package, in the same way that python setup.py install
>> --install-purelib=/some-directory can override the categories we have
>> now.
>
> -0. I'm not clear this is needed, but if it is I don't mind as long as
> it's as rare as it presently is. I am concerned about the implication
> that it's package authors that set this, and users won't have control
> (note that pip install doesn't offer an interface to this - there's
> --install-option but that's sdist-only so not applicable).

To understand the need for this feature it's important to imagine
different installation scenarios.

Imagine a package manager that installs each package into a tree of
~/python-packages/dist_name/version/ ie
~/python-packages/mercurial/3.4/. Each installed distribution exists
in its own independent directory. Console scripts written by this
package manager append the required dependencies to sys.path before
calling the program.

In this setup, each individual application has its own isolated set of
dependencies; upgrading one package does not break any other. You
would get similar isolation to having a virtualenv per package, but
without the overhead of setting up one before being able to use a
useful program.

This is just one example of a potentially useful setup that would be
helped by per-distribution { category : path } mappings. That doesn't
mean there is not another that you might like better.

I feel strongly about better supporting isolated per-package installs
because I can't recommend any Python application to my
non-Python-programmer friends if they have to create a virtualenv
first; they simply will not be successful getting through the
virtualenv step. Non-package-managed installers are too much
potentially platform specific work for the package creator, and --user
installs might break the dependencies of any other application already
installed as --user. If they can simply run 'humane_install
application' and be ready to go then it will be easier to recommend
useful Python applications.

>> - Record the { category : path } mapping used during installation.
>
> -1. There's no clear use case (except see below) and it implies that
> each package having its own mapping is the norm, where I would prefer
> a standard mapping for all cases with overrides being a vanishingly
> rare need. Basically, like existing paths such as the header
> directory.
>
>> - Provide an API mapping (distribution name, category, relative path
>> within category) to help applications using data that is only
>> accessible via the module loader.
>> https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceLoader
>
> +0. Ideally, this should be in the stdlib, but I accept that is
> impractical. I'm strongly against "installer runtime support" modules,
> it feels too like the setuptools/pkg_resources problem (developer or
> end user tool?). If necessary,then this should be in a completely
> separate package, just providing the runtime support, and not bundled
> with any particular installer. Note that for interoperability, this
> implies that the PEP needs to be very clear on all details, to avoid
> the API being defined by the implementation. Also, I'd prefer an
> interoperability-style "how the data is made available" standard,
> allowing projects to write their won API for it, rather than a "you
> have to use X to get at your data" style.
>
> The data for the mapping sounds like it's something that should be
> stored in the package's metadata 2.0 API (as a metadata extension). We
> don't need a custom approach for this data.

The Metadata 2.0 specifically pydata.json is supposed to be static. If
it is static then it would be illegal for the installer to change that
file as part of the installation. Instead, the { category : path }
mapping would be more like RECORD, an additional file in .dist-info
that stores installation data. See
https://www.python.org/dev/peps/pep-0376/

I am against making everyone depend on 'installer_support_package'
too, mirroring the situation that many packages depend on
pkg_resources without declaring that dependency. That is why I
proposed writing the install paths to an importable file in the
package's namespace on request without a new API. This would also
avoid "for path in sys.path: glob(path + '/*.dist-info')" the first
time any install path was required. However if it must be a new API so
be it.

>> - Provide an API mapping the same to paths on the filesystem.
>
> -0. Like pkg_resources.resource_filename. Donald was proposing an
> extension to the pkgutils API to do something similar as well. We
> don't need even more of these APIs, it should just be solved once in a
> general manner. For files not in package_data, the metadata API should
> provide enough to allow projects to extract the information manually.
> Someone could provide a simple helper module on PyPI to do that, but
> it shouldn't be *necessary*.
>
>> - Make the recorded mapping available in a predictable location, so it
>> can be perhaps understood by non-Python code.
>
> -1. Metadata 2.0 extensions seem to me to be the correct place for this.
>
>> - Allow setup.py's setup() call to install files relative to each
>> defined category.
>
> Conditional +1. As in, if we do define such categories, setup.py
> *must* support them, by definition. I presume here we'd be talking
> about setuptools rather than distutils.

If we had a better setup.py replacement, I'd propose supporting the
new categories there, first, as a carrot. But that's a different
discussion.

>> - Extend Python binary package formats to support the new categories,
>> so the { category : path } mapping can be set at install time and not
>> at build time.
>
> Conditional +0. Binary formats (wheel) must support this if defined,
> but setting mappings at install time seems optional to me. The
> defaults should be good enough for 99% of use cases, and I'd want
> installer support for changing the mappings to be optional (although
> clearly a quality-of-implementation issue - I'd expect pip to include
> it in some form, although maybe via a variant of something generic
> like --install-option).

The defaults will be unoffensive and typical for each supported
platform and default install scheme.

> One further note - when I mention Metadata 2.0 above, I'm thinking of
> the runtime data installed into a system with the package, not the
> static data exposed by PyPI (although having "which install locations
> does this package use" recorded in the static data would be good for
> ease of auditing). I'm not sure how well the runtime side of Metadata
> 2.0 has been thought through yet (in terms of APIs to read the data,
> installers adding data at install time) but that doesn't alter the
> fact that I think this is where such data should go. Also, I'm not
> trying to kill this proposal by making it depend on Metadata 2.0,
> rather pointing out that this is the sort of thing we need to push
> Metadata 2.0 forward for,so that we don't invent yet more custom and
> short-term data formats. I do think it's important to get this right
> if we're doing it, and that means implementing the appropriate parts
> of Metadata 2.0 to support it, so it is a *partial* dependency (muck
> like the packaging library has implemented the version and specifier
> parts, so that pip can use them).
>
> Paul