<div dir="ltr"><div>Can you clarify the relationship to PEP426 metadata?<br></div><div>There's no standard for metadata in here other than what's required to run a build hook.</div><div>Does that imply you would have each build tool enforce their own convention for where metadata is found?<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 1, 2015 at 9:53 PM, Nathaniel Smith <span dir="ltr"><<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
We realized that actually as far as we could tell, it wouldn't be that<br>
hard at this point to clean up how sdists work so that it would be<br>
possible to migrate away from distutils. So we wrote up a little draft<br>
proposal.<br>
<br>
The main question is, does this approach seem sound?<br>
<br>
-n<br>
<br>
---<br>
<br>
PEP: ??<br>
Title: Standard interface for interacting with source trees<br>
and source distributions<br>
Version: $Revision$<br>
Last-Modified: $Date$<br>
Author: Nathaniel J. Smith <<a href="mailto:njs@pobox.com">njs@pobox.com</a>><br>
Thomas Kluyver <<a href="mailto:takowl@gmail.com">takowl@gmail.com</a>><br>
Status: Draft<br>
Type: Standards-Track<br>
Content-Type: text/x-rst<br>
Created: 30-Sep-2015<br>
Post-History:<br>
Discussions-To: <<a href="mailto:distutils-sig@python.org">distutils-sig@python.org</a>><br>
<br>
Abstract<br>
========<br>
<br>
Distutils delenda est.<br>
<br>
<br>
Extended abstract<br>
=================<br>
<br>
While ``distutils`` / ``setuptools`` have taken us a long way, they<br>
suffer from three serious problems: (a) they're missing important<br>
features like autoconfiguration and usable build-time dependency<br>
declaration, (b) extending them is quirky, complicated, and fragile,<br>
(c) you are forced to use them anyway, because they provide the<br>
standard interface for installing python packages expected by both<br>
users and installation tools like ``pip``.<br>
<br>
Previous efforts (e.g. distutils2 or setuptools itself) have attempted<br>
to solve problems (a) and/or (b). We propose to solve (c).<br>
<br>
The goal of this PEP is get distutils-sig out of the business of being<br>
a gatekeeper for Python build systems. If you want to use distutils,<br>
great; if you want to use something else, then the more the merrier.<br>
The difficulty of interfacing with distutils means that there aren't<br>
many such systems right now, but to give a sense of what we're<br>
thinking about see `flit <<a href="https://github.com/takluyver/flit" rel="noreferrer" target="_blank">https://github.com/takluyver/flit</a>>`_ or<br>
`bento<br>
<<a href="https://cournape.github.io/Bento/" rel="noreferrer" target="_blank">https://cournape.github.io/Bento/</a>>`_. Fortunately, wheels have now<br>
solved many of the hard problems here -- e.g. it's no longer necessary<br>
that a build system also know about every possible installation<br>
configuration -- so pretty much all we really need from a build system<br>
is that it have some way to spit out standard-compliant wheels.<br>
<br>
We therefore propose a new, relatively minimal interface for<br>
installation tools like ``pip`` to interact with package source trees<br>
and source distributions.<br>
<br>
<br>
Synopsis and rationale<br>
======================<br>
<br>
To limit the scope of our design, we adopt several principles.<br>
<br>
First, we distinguish between a *source tree* (e.g., a VCS checkout)<br>
and a *source distribution* (e.g., an official snapshot release like<br>
``lxml-3.4.4.zip``).<br>
<br>
There isn't a whole lot that *source trees* can be assumed to have in<br>
common. About all you know is that they can -- via some more or less<br>
Rube-Goldbergian process -- produce one or more binary distributions.<br>
In particular, you *cannot* tell via simple static inspection:<br>
- What version number will be attached to the resulting packages (e.g.<br>
it might be determined programmatically by consulting VCS metadata --<br>
I have here a build of numpy version "1.11.0.dev0+4a9ad17")<br>
- What build- or run-time dependencies are required (e.g. these may<br>
depend on arbitrarily complex configuration settings that are<br>
determined via a mix of manual settings and auto-probing)<br>
- Or even how many distinct binary distributions will be produced<br>
(e.g. a source distribution may always produce wheel A, but only<br>
produce wheel B when built on Unix-like systems).<br>
<br>
Therefore, when dealing with source trees, our goal is just to provide<br>
a standard UX for the core operations that are commonly performed on<br>
other people's packages; anything fancier and more developer-centric<br>
we leave at the discretion of individual package developers. So our<br>
source trees just provide some simple hooks to let a tool like<br>
``pip``:<br>
<br>
- query for build dependencies<br>
- run a build, producing wheels as output<br>
- set up the current source tree so that it can be placed on<br>
``sys.path`` in "develop mode"<br>
<br>
and that's it. We teach users that the standard way to install a<br>
package from a VCS checkout is now ``pip install .`` instead of<br>
``python setup.py install``. (This is already a good idea anyway --<br>
e.g., pip can do reliable uninstall / upgrades.)<br>
<br>
Next, we note that pretty much all the operations that you might want<br>
to perform on a *source distribution* are also operations that you<br>
might want to perform on a source tree, and via the same UX. The only<br>
thing you do with source distributions that you don't do with source<br>
trees is, well, distribute them. There's all kind of metadata you<br>
could imagine including in a source distribution, but each piece of<br>
metadata puts an increased burden on source distribution generation<br>
tools, and most operations will still have to work without this<br>
metadata. So we only include extra metadata in source distributions if<br>
it helps solve specific problems that are unique to distribution. If<br>
you want wheel-style metadata, get a wheel and look at it -- they're<br>
great and getting better.<br>
<br>
Therefore, our source distributions are basically just source trees +<br>
a mechanism for signing.<br>
<br>
Finally: we explicitly do *not* have any concept of "depending on a<br>
source distribution". As in other systems like Debian, dependencies<br>
are always phrased in terms of binary distributions (wheels), and when<br>
a user runs something like ``pip install <package>``, then the<br>
long-run plan is that <package> and all its transitive dependencies<br>
should be available as wheels in a package index. But this is not yet<br>
realistic, so as a transitional / backwards-compatibility measure, we<br>
provide a simple mechanism for ``pip install <package>`` to handle<br>
cases where <package> is provided only as a source distribution.<br>
<br>
<br>
Source trees<br>
============<br>
<br>
We retroactively declare the legacy source tree format involving<br>
``setup.py`` to be "version 0". We don't try to specify it further;<br>
its de facto specification is encoded in the source code of<br>
``distutils``, ``setuptools``, ``pip``, and other tools.<br>
<br>
A version 1-or-greater format source tree can be identified by the<br>
presence of a file ``_pypackage/_pypackage.cfg``.<br>
<br>
If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present,<br>
then we have a version 1+ source tree, i.e., ``setup.py`` is ignored.<br>
This is necessary because we anticipate that version 1+ source trees<br>
may want to contain a ``setup.py`` file for backwards compatibility,<br>
e.g.::<br>
<br>
#!/usr/bin/env python<br>
import sys<br>
print("Don't call setup.py directly!")<br>
print("Use 'pip install .' instead!")<br>
print("(You might have to upgrade pip first.)")<br>
sys.exit(1)<br>
<br>
In the current version of the specification, the one file<br>
``_pypackage/_pypackage.cfg`` is where pretty much all the action is<br>
(though see below). The motivation for putting it into a subdirectory<br>
is that:<br>
- the way of all standards is that cruft accumulates over time, so<br>
this way we pre-emptively have a place to put it,<br>
- real-world projects often accumulate build system cruft as well, so<br>
we might as well provide one obvious place to put it too.<br>
<br>
Of course this then creates the possibility of collisions between<br>
standard files and user files, and trying to teach arbitrary users not<br>
to scatter files around willy-nilly never works, so we adopt the<br>
convention that names starting with an underscore are reserved for<br>
official use, and non-underscored names are available for<br>
idiosyncratic use by individual projects.<br>
<br>
The alternative would be to simply place the main configuration file<br>
at the top-level, create the subdirectory only when specifically<br>
needed (most trees won't need it), and let users worry about finding<br>
their own place for their cruft. Not sure which is the best approach.<br>
Plus we can have a nice bikeshed about the names in general (FIXME).<br>
<br>
_pypackage.cfg<br>
--------------<br>
<br>
The ``_pypackage.cfg`` file contains various settings. Another good<br>
bike-shed topic is which file format to use for storing these (FIXME),<br>
but for purposes of this draft I'll write examples using `toml<br>
<<a href="https://github.com/toml-lang/toml" rel="noreferrer" target="_blank">https://github.com/toml-lang/toml</a>>`_, because you'll instantly be<br>
able to understand the semantics, it has similar expressivity to JSON<br>
while being more human-friendly (e.g., it supports comments and<br>
multi-line strings), it's better-specified than ConfigParser, and it's<br>
much simpler than YAML. Rust's package manager uses toml for similar<br>
purposes.<br>
<br>
Here's an example ``_pypackage/_pypackage.cfg``::<br>
<br>
# Version of the "pypackage format" that this file uses.<br>
# Optional. If not present then 1 is assumed.<br>
# All version changes indicate incompatible changes; backwards<br>
# compatible changes are indicated by just having extra stuff in<br>
# the file.<br>
version = 1<br>
<br>
[build]<br>
# An inline requirements file. Optional.<br>
# (FIXME: I guess this means we need a spec for requirements files?)<br>
requirements = """<br>
mybuildtool >= 2.1<br>
special_windows_tool ; sys_platform == "win32"<br>
"""<br>
# The path to an out-of-line requirements file. Optional.<br>
requirements-file = "build-requirements.txt"<br>
# A hook that will be called to query build requirements. Optional.<br>
requirements-dynamic = "mybuildtool:get_requirements"<br>
<br>
# A hook that will be called to build wheels. Required.<br>
build-wheels = "mybuildtool:do_build"<br>
<br>
# A hook that will be called to do an in-place build (see below).<br>
# Optional.<br>
build-in-place = "mybuildtool:do_inplace_build"<br>
<br>
# The "x" namespace is reserved for third-party extensions.<br>
# To use x.foo you should own the name "foo" on pypi.<br>
[x.mybuildtool]<br>
spam = ["spam", "spam", "spam"]<br>
<br>
All paths are relative to the ``_pypackage/`` directory (so e.g. the<br>
build.requirements-file value above refers to a file named<br>
``_pypackage/build-requirements.txt``).<br>
<br>
A *hook* is a Python object that is looked up using the same rules as<br>
traditional setuptools entry_points: a dotted module name, followed by<br>
a colon, followed by a dotted name that is looked up within that<br>
module. *Running a hook* means: first, find or create a python<br>
interpreter which is executing in the current venv, whose working<br>
directory is set to the ``_pypackage/`` directory, and which has the<br>
``_pypackage/`` directory on ``sys.path``. Then, inside this<br>
interpreter, look up the hook object, and call it, with arguments as<br>
specified below.<br>
<br>
A build command like ``pip wheel <source tree>`` performs the following steps:<br>
<br>
1) Validate the ``_pypackage.cfg`` version number.<br>
<br>
2) Create an empty virtualenv / venv, that matches the environment<br>
that the installer is targeting (e.g. if you want wheels for CPython<br>
3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows<br>
venv).<br>
<br>
3) If the build.requirements key is present, then in this venv run the<br>
equivalent of ``pip install -r <a file containing its value>``, using<br>
whatever index settings are currently in effect.<br>
<br>
4) If the build.requirements-file key is present, then in this venv<br>
run the equivalent of ``pip install -r <the named file>``, using<br>
whatever index settings are currently in effect.<br>
<br>
5) If the build.requirements-dynamic key is present, then in this venv<br>
run the hook with no arguments, capture its stdout, and pipe it into<br>
``pip install -r -``, using whatever index settings are currently in<br>
effect. If the hook raises an exception, then abort the build with an<br>
error.<br>
<br>
Note: because these steps are performed in sequence, the<br>
build.requirements-dynamic hook is allowed to use packages that are<br>
listed in build.requirements or build.requirements-file.<br>
<br>
6) In this venv, run the build.build-wheels hook. This should be a<br>
Python function which takes one argument.<br>
<br>
This argument is an arbitrary dictionary intended to contain<br>
user-specified configuration, specified via some install-tool-specific<br>
mechanism. The intention is that tools like ``pip`` should provide<br>
some way for users to specify key/value settings that will be passed<br>
in here, analogous to the legacy ``--install-option`` and<br>
``--global-option`` arguments.<br>
<br>
To make it easier for packages to transition from version 0 to<br>
version 1 sdists, we suggest that ``pip`` and other tools that have<br>
such existing option-setting interfaces SHOULD map them to entries in<br>
this dictionary when -- e.g.::<br>
<br>
pip --global-option=a --install-option=b --install-option=c<br>
<br>
could produce a dict like::<br>
<br>
{"--global-option": ["a"], "--install-option": ["b", "c"]}<br>
<br>
The hook's return value is a list of pathnames relative to the<br>
scratch directory. Each entry names a wheel file created by this<br>
build.<br>
<br>
Errors are signaled by raising an exception.<br>
<br>
When performing an in-place build (e.g. for ``pip install -e .``),<br>
then the same steps are followed, except that instead of the<br>
build.build-wheels hook, we call the build.build-in-place hook, and<br>
instead of returning a list of wheel files, it returns the name of a<br>
directory that should be placed onto ``sys.path`` (usually this will<br>
be the source tree itself, but may not be, e.g. if a build system<br>
wants to enforce a rule where the source is always kept pristine then<br>
it could symlink the .py files into a build directory, place the<br>
extension modules and dist-info there, and return that). This<br>
directory must contain importable versions of the code in the source<br>
tree, along with appropriate .dist-info directories.<br>
<br>
(FIXME: in-place builds are useful but intrinsically kinda broken --<br>
e.g. extensions / source / metadata can all easily get out of sync --<br>
so while I think this paragraph provides a reasonable hack that<br>
preserves current functionality, maybe we should defer specifying them<br>
to until after we've thought through the issues more?)<br>
<br>
When working with source trees, build tools like ``pip`` are<br>
encouraged to cache and re-use virtualenvs for performance.<br>
<br>
<br>
Other contents of _pypackage/<br>
-----------------------------<br>
<br>
_RECORD, _RECORD.jws, _RECORD.p7s: see below.<br>
<br>
_x/<pypi name>/: reserved for use by tools (e.g.<br>
_x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/)<br>
<br>
<br>
Source distributions<br>
====================<br>
<br>
A *source distribution* is a file in a well-known archive format such<br>
as zip or tar.gz, which contains a single directory, and this<br>
directory is a source tree (in the sense defined in the previous<br>
section).<br>
<br>
The ``_pypackage/`` directory in a source distribution SHOULD also<br>
contain a _RECORD file, as defined in PEP 427, and MAY also contain<br>
_RECORD.jws and/or _RECORD.p7s signature files.<br>
<br>
For official releases, source distributions SHOULD be named as<br>
``<package>-<version>.<ext>``, and the directory they contain SHOULD<br>
be named ``<package>-<version>``, and building this source tree SHOULD<br>
produce a wheel named ``<package>-<version>-<compatibility tag>.whl``<br>
(though it may produce other wheels as well).<br>
<br>
(FIXME: maybe we should add that if you want your sdist on PyPI then<br>
you MUST include a proper _RECORD file and use the proper naming<br>
convention?)<br>
<br>
Integration tools like ``pip`` SHOULD take advantage of this<br>
convention by applying the following heuristic: when seeking a package<br>
<package>, if no appropriate wheel can be found, but an sdist named<br>
<package>-<version>.<ext> is found, then:<br>
<br>
1) build the sdist<br>
2) add the resulting wheels to the package search space<br>
3) retry the original operation<br>
<br>
This handles a variety of simple and complex cases -- for example, if<br>
we need a package 'foo', and we find foo-1.0.zip which builds foo.whl<br>
and bar.whl, and foo.whl depends on bar.whl, then everything will work<br>
out. There remain other cases that are not handled, e.g. if we start<br>
out searching for bar.whl we will never discover foo-1.0.zip. We take<br>
the perspective that this is nonetheless sufficient for a transitional<br>
heuristic, and anyone who runs into this problem should just upload<br>
wheels already. If this turns out to be inadequate in practice, then<br>
it will be addressed by future extensions.<br>
<br>
<br>
Examples<br>
========<br>
<br>
**Example 1:** While we assume that installation tools will have to<br>
continue supporting version 0 sdists for the indefinite future, it's a<br>
useful check to make sure that our new format can continue to support<br>
packages using distutils / setuptools as their build system. We assume<br>
that a future version ``pip`` will take its existing knowledge of<br>
distutils internals and expose them as the appropriate hooks, and then<br>
existing distutils / setuptools packages can be ported forward by<br>
using the following ``_pypackage/_pypackage.cfg``::<br>
<br>
[build]<br>
requirements = """<br>
pip >= whatever<br>
wheel<br>
"""<br>
# Applies monkeypatches, then does 'setup.py dist_info' and<br>
# extracts the setup_requires<br>
requirements-dynamic = "pip.pypackage_hooks:setup_requirements"<br>
# Applies monkeypatches, then does 'setup.py wheel'<br>
build-wheels = "pip.pypackage_hooks:build_wheels"<br>
# Applies monkeypatches, then does:<br>
# setup.py dist_info && setup.py build_ext -i<br>
build-in-place = "pip.pypackage_hooks:build_in_place"<br>
<br>
This is also useful for any other installation tools that may want to<br>
support version 0 sdists without having to implement bug-for-bug<br>
compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is<br>
present, they can use this as a default.<br>
<br>
**Example 2:** For packages using numpy.distutils. This is identical<br>
to the distutils / setuptools example above, except that numpy is<br>
moved into the list of static build requirements. Right now, most<br>
projects using numpy.distutils don't bother trying to declare this<br>
dependency, and instead simply error out if numpy is not already<br>
installed. This is because currently the only way to declare a build<br>
dependency is via the ``setup_requires`` argument to the ``setup``<br>
function, and in this case the ``setup`` function is<br>
``numpy.distutils.setup``, which... obviously doesn't work very well.<br>
Drop this ``_pypackage.cfg`` into an existing project like this and it<br>
will become robustly pip-installable with no further changes::<br>
<br>
[build]<br>
requirements = """<br>
numpy<br>
pip >= whatever<br>
wheel<br>
"""<br>
requirements-dynamic = "pip.pypackage_hooks:setup_requirements"<br>
build-wheels = "pip.pypackage_hooks:build_wheels"<br>
build-in-place = "pip.pypackage_hooks:build_in_place"<br>
<br>
**Example 3:** `flit <<a href="https://github.com/takluyver/flit" rel="noreferrer" target="_blank">https://github.com/takluyver/flit</a>>`_ is a tool<br>
designed to make distributing simple packages simple, but it currently<br>
has no support for sdists, and for convenience includes its own<br>
installation code that's redundant with that in pip. These 4 lines of<br>
boilerplate make any flit-using source tree pip-installable, and lets<br>
flit get out of the package installation business::<br>
<br>
[build]<br>
requirements = "flit"<br>
build-wheels = "flit.pypackage_hooks:build_wheels"<br>
build-in-place = "flit.pypackage_hooks:build_in_place"<br>
<br>
<br>
FAQ<br>
===<br>
<br>
**Why is it version 1 instead of version 2?** Because the legacy sdist<br>
format is barely a format at all, and to `remind us to keep things<br>
simple <<a href="https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect" rel="noreferrer" target="_blank">https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect</a>>`_.<br>
<br>
**What about cross-compilation?** Standardizing an interface for<br>
cross-compilation seems premature given how complicated the<br>
configuration required can be, the lack of an existing de facto<br>
standard, and the authors of this PEP's inexperience with<br>
cross-compilation. This would be a great target for future extensions,<br>
though. In the mean time, there's no requirement that<br>
``_pypackage/_pypackage.cfg`` contain the *only* entry points to a<br>
project's build system -- packages that want to support<br>
cross-compilation can still do so, they'll just need to include a<br>
README explaining how to do it.<br>
<br>
**PEP 426 says that the new sdist format will support automatically<br>
creating policy-compliant .deb/.rpm packages. What happened to that?**<br>
Step 1: enhance the wheel format as necessary so that a wheel can be<br>
automatically converted into a policy-compliant .deb/.rpm package (see<br>
PEP 491). Step 2: make it possible to automatically turn sdists into<br>
wheels (this PEP). Step 3: we're done.<br>
<br>
**What about automatically running tests?** Arguably this is another<br>
thing that should be pushed off to wheel metadata instead of sdist<br>
metadata: it's good practice to include tests inside your built<br>
distribution so that end-users can test their install (and see above<br>
re: our focus here being on stuff that end-users want to do, not<br>
dedicated package developers), there are lots of packages that have to<br>
be built before they can be tested anyway (e.g. because of binary<br>
extensions), and in any case it's good practice to test against an<br>
installed version in order to make sure your install code works<br>
properly. But even if we do want this in sdist, then it's hardly<br>
urgent (e.g. there is no ``pip test`` that people will miss), so we<br>
defer that for a future extension to avoid blocking the core<br>
functionality.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Nathaniel J. Smith -- <a href="http://vorpus.org" rel="noreferrer" target="_blank">http://vorpus.org</a><br>
_______________________________________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
</font></span></blockquote></div><br></div>