<div dir="ltr">Thank you for your work on this. We have to kill distutils to make progress in packaging.</div><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 2, 2015 at 10:12 AM Daniel Holth <<a href="mailto:dholth@gmail.com">dholth@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><span style="line-height:1.5">One way to do sdist 2.0 would be to have the package-1.0.dist-info directory in there (most sdists contain setuptools metadata) and to have a flag static-metadata=1 in setup.cfg asserting that setup.py [if present] does not alter the list of dependencies.</span><br></div><div><div><br></div><div>In the old MEBS design the package could suggest a build system, but pip would invoke a list of build plugins to inspect the directory and return True if they were able to build the package. This would allow for ignoring the package's suggested build system. Instead of defining a command line interface for setup.py MEBS would define a set of methods on the build plugin.</div></div><div><br></div><div><div>I thought Robert Collins had a working setup-requires implementation already? I have a worse but backwards compatible one too at <a href="https://bitbucket.org/dholth/setup-requires/src/tip/setup.py" target="_blank">https://bitbucket.org/dholth/setup-requires/src/tip/setup.py</a></div><span style="line-height:1.5"></span></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 2, 2015 at 9:42 AM Marcus Smith <<a href="mailto:qwcode@gmail.com" target="_blank">qwcode@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Can you clarify the relationship to PEP426 metadata?<br></div><div>There's no standard for metadata in here other than what's required to run a build hook.</div></div><div dir="ltr"><div>Does that imply you would have each build tool enforce their own convention for where metadata is found?<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 1, 2015 at 9:53 PM, Nathaniel Smith <span dir="ltr"><<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
We realized that actually as far as we could tell, it wouldn't be that<br>
hard at this point to clean up how sdists work so that it would be<br>
possible to migrate away from distutils. So we wrote up a little draft<br>
proposal.<br>
<br>
The main question is, does this approach seem sound?<br>
<br>
-n<br>
<br>
---<br>
<br>
PEP: ??<br>
Title: Standard interface for interacting with source trees<br>
and source distributions<br>
Version: $Revision$<br>
Last-Modified: $Date$<br>
Author: Nathaniel J. Smith <<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>><br>
Thomas Kluyver <<a href="mailto:takowl@gmail.com" target="_blank">takowl@gmail.com</a>><br>
Status: Draft<br>
Type: Standards-Track<br>
Content-Type: text/x-rst<br>
Created: 30-Sep-2015<br>
Post-History:<br>
Discussions-To: <<a href="mailto:distutils-sig@python.org" target="_blank">distutils-sig@python.org</a>><br>
<br>
Abstract<br>
========<br>
<br>
Distutils delenda est.<br>
<br>
<br>
Extended abstract<br>
=================<br>
<br>
While ``distutils`` / ``setuptools`` have taken us a long way, they<br>
suffer from three serious problems: (a) they're missing important<br>
features like autoconfiguration and usable build-time dependency<br>
declaration, (b) extending them is quirky, complicated, and fragile,<br>
(c) you are forced to use them anyway, because they provide the<br>
standard interface for installing python packages expected by both<br>
users and installation tools like ``pip``.<br>
<br>
Previous efforts (e.g. distutils2 or setuptools itself) have attempted<br>
to solve problems (a) and/or (b). We propose to solve (c).<br>
<br>
The goal of this PEP is get distutils-sig out of the business of being<br>
a gatekeeper for Python build systems. If you want to use distutils,<br>
great; if you want to use something else, then the more the merrier.<br>
The difficulty of interfacing with distutils means that there aren't<br>
many such systems right now, but to give a sense of what we're<br>
thinking about see `flit <<a href="https://github.com/takluyver/flit" rel="noreferrer" target="_blank">https://github.com/takluyver/flit</a>>`_ or<br>
`bento<br>
<<a href="https://cournape.github.io/Bento/" rel="noreferrer" target="_blank">https://cournape.github.io/Bento/</a>>`_. Fortunately, wheels have now<br>
solved many of the hard problems here -- e.g. it's no longer necessary<br>
that a build system also know about every possible installation<br>
configuration -- so pretty much all we really need from a build system<br>
is that it have some way to spit out standard-compliant wheels.<br>
<br>
We therefore propose a new, relatively minimal interface for<br>
installation tools like ``pip`` to interact with package source trees<br>
and source distributions.<br>
<br>
<br>
Synopsis and rationale<br>
======================<br>
<br>
To limit the scope of our design, we adopt several principles.<br>
<br>
First, we distinguish between a *source tree* (e.g., a VCS checkout)<br>
and a *source distribution* (e.g., an official snapshot release like<br>
``lxml-3.4.4.zip``).<br>
<br>
There isn't a whole lot that *source trees* can be assumed to have in<br>
common. About all you know is that they can -- via some more or less<br>
Rube-Goldbergian process -- produce one or more binary distributions.<br>
In particular, you *cannot* tell via simple static inspection:<br>
- What version number will be attached to the resulting packages (e.g.<br>
it might be determined programmatically by consulting VCS metadata --<br>
I have here a build of numpy version "1.11.0.dev0+4a9ad17")<br>
- What build- or run-time dependencies are required (e.g. these may<br>
depend on arbitrarily complex configuration settings that are<br>
determined via a mix of manual settings and auto-probing)<br>
- Or even how many distinct binary distributions will be produced<br>
(e.g. a source distribution may always produce wheel A, but only<br>
produce wheel B when built on Unix-like systems).<br>
<br>
Therefore, when dealing with source trees, our goal is just to provide<br>
a standard UX for the core operations that are commonly performed on<br>
other people's packages; anything fancier and more developer-centric<br>
we leave at the discretion of individual package developers. So our<br>
source trees just provide some simple hooks to let a tool like<br>
``pip``:<br>
<br>
- query for build dependencies<br>
- run a build, producing wheels as output<br>
- set up the current source tree so that it can be placed on<br>
``sys.path`` in "develop mode"<br>
<br>
and that's it. We teach users that the standard way to install a<br>
package from a VCS checkout is now ``pip install .`` instead of<br>
``python setup.py install``. (This is already a good idea anyway --<br>
e.g., pip can do reliable uninstall / upgrades.)<br>
<br>
Next, we note that pretty much all the operations that you might want<br>
to perform on a *source distribution* are also operations that you<br>
might want to perform on a source tree, and via the same UX. The only<br>
thing you do with source distributions that you don't do with source<br>
trees is, well, distribute them. There's all kind of metadata you<br>
could imagine including in a source distribution, but each piece of<br>
metadata puts an increased burden on source distribution generation<br>
tools, and most operations will still have to work without this<br>
metadata. So we only include extra metadata in source distributions if<br>
it helps solve specific problems that are unique to distribution. If<br>
you want wheel-style metadata, get a wheel and look at it -- they're<br>
great and getting better.<br>
<br>
Therefore, our source distributions are basically just source trees +<br>
a mechanism for signing.<br>
<br>
Finally: we explicitly do *not* have any concept of "depending on a<br>
source distribution". As in other systems like Debian, dependencies<br>
are always phrased in terms of binary distributions (wheels), and when<br>
a user runs something like ``pip install <package>``, then the<br>
long-run plan is that <package> and all its transitive dependencies<br>
should be available as wheels in a package index. But this is not yet<br>
realistic, so as a transitional / backwards-compatibility measure, we<br>
provide a simple mechanism for ``pip install <package>`` to handle<br>
cases where <package> is provided only as a source distribution.<br>
<br>
<br>
Source trees<br>
============<br>
<br>
We retroactively declare the legacy source tree format involving<br>
``setup.py`` to be "version 0". We don't try to specify it further;<br>
its de facto specification is encoded in the source code of<br>
``distutils``, ``setuptools``, ``pip``, and other tools.<br>
<br>
A version 1-or-greater format source tree can be identified by the<br>
presence of a file ``_pypackage/_pypackage.cfg``.<br>
<br>
If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present,<br>
then we have a version 1+ source tree, i.e., ``setup.py`` is ignored.<br>
This is necessary because we anticipate that version 1+ source trees<br>
may want to contain a ``setup.py`` file for backwards compatibility,<br>
e.g.::<br>
<br>
#!/usr/bin/env python<br>
import sys<br>
print("Don't call setup.py directly!")<br>
print("Use 'pip install .' instead!")<br>
print("(You might have to upgrade pip first.)")<br>
sys.exit(1)<br>
<br>
In the current version of the specification, the one file<br>
``_pypackage/_pypackage.cfg`` is where pretty much all the action is<br>
(though see below). The motivation for putting it into a subdirectory<br>
is that:<br>
- the way of all standards is that cruft accumulates over time, so<br>
this way we pre-emptively have a place to put it,<br>
- real-world projects often accumulate build system cruft as well, so<br>
we might as well provide one obvious place to put it too.<br>
<br>
Of course this then creates the possibility of collisions between<br>
standard files and user files, and trying to teach arbitrary users not<br>
to scatter files around willy-nilly never works, so we adopt the<br>
convention that names starting with an underscore are reserved for<br>
official use, and non-underscored names are available for<br>
idiosyncratic use by individual projects.<br>
<br>
The alternative would be to simply place the main configuration file<br>
at the top-level, create the subdirectory only when specifically<br>
needed (most trees won't need it), and let users worry about finding<br>
their own place for their cruft. Not sure which is the best approach.<br>
Plus we can have a nice bikeshed about the names in general (FIXME).<br>
<br>
_pypackage.cfg<br>
--------------<br>
<br>
The ``_pypackage.cfg`` file contains various settings. Another good<br>
bike-shed topic is which file format to use for storing these (FIXME),<br>
but for purposes of this draft I'll write examples using `toml<br>
<<a href="https://github.com/toml-lang/toml" rel="noreferrer" target="_blank">https://github.com/toml-lang/toml</a>>`_, because you'll instantly be<br>
able to understand the semantics, it has similar expressivity to JSON<br>
while being more human-friendly (e.g., it supports comments and<br>
multi-line strings), it's better-specified than ConfigParser, and it's<br>
much simpler than YAML. Rust's package manager uses toml for similar<br>
purposes.<br>
<br>
Here's an example ``_pypackage/_pypackage.cfg``::<br>
<br>
# Version of the "pypackage format" that this file uses.<br>
# Optional. If not present then 1 is assumed.<br>
# All version changes indicate incompatible changes; backwards<br>
# compatible changes are indicated by just having extra stuff in<br>
# the file.<br>
version = 1<br>
<br>
[build]<br>
# An inline requirements file. Optional.<br>
# (FIXME: I guess this means we need a spec for requirements files?)<br>
requirements = """<br>
mybuildtool >= 2.1<br>
special_windows_tool ; sys_platform == "win32"<br>
"""<br>
# The path to an out-of-line requirements file. Optional.<br>
requirements-file = "build-requirements.txt"<br>
# A hook that will be called to query build requirements. Optional.<br>
requirements-dynamic = "mybuildtool:get_requirements"<br>
<br>
# A hook that will be called to build wheels. Required.<br>
build-wheels = "mybuildtool:do_build"<br>
<br>
# A hook that will be called to do an in-place build (see below).<br>
# Optional.<br>
build-in-place = "mybuildtool:do_inplace_build"<br>
<br>
# The "x" namespace is reserved for third-party extensions.<br>
# To use x.foo you should own the name "foo" on pypi.<br>
[x.mybuildtool]<br>
spam = ["spam", "spam", "spam"]<br>
<br>
All paths are relative to the ``_pypackage/`` directory (so e.g. the<br>
build.requirements-file value above refers to a file named<br>
``_pypackage/build-requirements.txt``).<br>
<br>
A *hook* is a Python object that is looked up using the same rules as<br>
traditional setuptools entry_points: a dotted module name, followed by<br>
a colon, followed by a dotted name that is looked up within that<br>
module. *Running a hook* means: first, find or create a python<br>
interpreter which is executing in the current venv, whose working<br>
directory is set to the ``_pypackage/`` directory, and which has the<br>
``_pypackage/`` directory on ``sys.path``. Then, inside this<br>
interpreter, look up the hook object, and call it, with arguments as<br>
specified below.<br>
<br>
A build command like ``pip wheel <source tree>`` performs the following steps:<br>
<br>
1) Validate the ``_pypackage.cfg`` version number.<br>
<br>
2) Create an empty virtualenv / venv, that matches the environment<br>
that the installer is targeting (e.g. if you want wheels for CPython<br>
3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows<br>
venv).<br>
<br>
3) If the build.requirements key is present, then in this venv run the<br>
equivalent of ``pip install -r <a file containing its value>``, using<br>
whatever index settings are currently in effect.<br>
<br>
4) If the build.requirements-file key is present, then in this venv<br>
run the equivalent of ``pip install -r <the named file>``, using<br>
whatever index settings are currently in effect.<br>
<br>
5) If the build.requirements-dynamic key is present, then in this venv<br>
run the hook with no arguments, capture its stdout, and pipe it into<br>
``pip install -r -``, using whatever index settings are currently in<br>
effect. If the hook raises an exception, then abort the build with an<br>
error.<br>
<br>
Note: because these steps are performed in sequence, the<br>
build.requirements-dynamic hook is allowed to use packages that are<br>
listed in build.requirements or build.requirements-file.<br>
<br>
6) In this venv, run the build.build-wheels hook. This should be a<br>
Python function which takes one argument.<br>
<br>
This argument is an arbitrary dictionary intended to contain<br>
user-specified configuration, specified via some install-tool-specific<br>
mechanism. The intention is that tools like ``pip`` should provide<br>
some way for users to specify key/value settings that will be passed<br>
in here, analogous to the legacy ``--install-option`` and<br>
``--global-option`` arguments.<br>
<br>
To make it easier for packages to transition from version 0 to<br>
version 1 sdists, we suggest that ``pip`` and other tools that have<br>
such existing option-setting interfaces SHOULD map them to entries in<br>
this dictionary when -- e.g.::<br>
<br>
pip --global-option=a --install-option=b --install-option=c<br>
<br>
could produce a dict like::<br>
<br>
{"--global-option": ["a"], "--install-option": ["b", "c"]}<br>
<br>
The hook's return value is a list of pathnames relative to the<br>
scratch directory. Each entry names a wheel file created by this<br>
build.<br>
<br>
Errors are signaled by raising an exception.<br>
<br>
When performing an in-place build (e.g. for ``pip install -e .``),<br>
then the same steps are followed, except that instead of the<br>
build.build-wheels hook, we call the build.build-in-place hook, and<br>
instead of returning a list of wheel files, it returns the name of a<br>
directory that should be placed onto ``sys.path`` (usually this will<br>
be the source tree itself, but may not be, e.g. if a build system<br>
wants to enforce a rule where the source is always kept pristine then<br>
it could symlink the .py files into a build directory, place the<br>
extension modules and dist-info there, and return that). This<br>
directory must contain importable versions of the code in the source<br>
tree, along with appropriate .dist-info directories.<br>
<br>
(FIXME: in-place builds are useful but intrinsically kinda broken --<br>
e.g. extensions / source / metadata can all easily get out of sync --<br>
so while I think this paragraph provides a reasonable hack that<br>
preserves current functionality, maybe we should defer specifying them<br>
to until after we've thought through the issues more?)<br>
<br>
When working with source trees, build tools like ``pip`` are<br>
encouraged to cache and re-use virtualenvs for performance.<br>
<br>
<br>
Other contents of _pypackage/<br>
-----------------------------<br>
<br>
_RECORD, _RECORD.jws, _RECORD.p7s: see below.<br>
<br>
_x/<pypi name>/: reserved for use by tools (e.g.<br>
_x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/)<br>
<br>
<br>
Source distributions<br>
====================<br>
<br>
A *source distribution* is a file in a well-known archive format such<br>
as zip or tar.gz, which contains a single directory, and this<br>
directory is a source tree (in the sense defined in the previous<br>
section).<br>
<br>
The ``_pypackage/`` directory in a source distribution SHOULD also<br>
contain a _RECORD file, as defined in PEP 427, and MAY also contain<br>
_RECORD.jws and/or _RECORD.p7s signature files.<br>
<br>
For official releases, source distributions SHOULD be named as<br>
``<package>-<version>.<ext>``, and the directory they contain SHOULD<br>
be named ``<package>-<version>``, and building this source tree SHOULD<br>
produce a wheel named ``<package>-<version>-<compatibility tag>.whl``<br>
(though it may produce other wheels as well).<br>
<br>
(FIXME: maybe we should add that if you want your sdist on PyPI then<br>
you MUST include a proper _RECORD file and use the proper naming<br>
convention?)<br>
<br>
Integration tools like ``pip`` SHOULD take advantage of this<br>
convention by applying the following heuristic: when seeking a package<br>
<package>, if no appropriate wheel can be found, but an sdist named<br>
<package>-<version>.<ext> is found, then:<br>
<br>
1) build the sdist<br>
2) add the resulting wheels to the package search space<br>
3) retry the original operation<br>
<br>
This handles a variety of simple and complex cases -- for example, if<br>
we need a package 'foo', and we find foo-1.0.zip which builds foo.whl<br>
and bar.whl, and foo.whl depends on bar.whl, then everything will work<br>
out. There remain other cases that are not handled, e.g. if we start<br>
out searching for bar.whl we will never discover foo-1.0.zip. We take<br>
the perspective that this is nonetheless sufficient for a transitional<br>
heuristic, and anyone who runs into this problem should just upload<br>
wheels already. If this turns out to be inadequate in practice, then<br>
it will be addressed by future extensions.<br>
<br>
<br>
Examples<br>
========<br>
<br>
**Example 1:** While we assume that installation tools will have to<br>
continue supporting version 0 sdists for the indefinite future, it's a<br>
useful check to make sure that our new format can continue to support<br>
packages using distutils / setuptools as their build system. We assume<br>
that a future version ``pip`` will take its existing knowledge of<br>
distutils internals and expose them as the appropriate hooks, and then<br>
existing distutils / setuptools packages can be ported forward by<br>
using the following ``_pypackage/_pypackage.cfg``::<br>
<br>
[build]<br>
requirements = """<br>
pip >= whatever<br>
wheel<br>
"""<br>
# Applies monkeypatches, then does 'setup.py dist_info' and<br>
# extracts the setup_requires<br>
requirements-dynamic = "pip.pypackage_hooks:setup_requirements"<br>
# Applies monkeypatches, then does 'setup.py wheel'<br>
build-wheels = "pip.pypackage_hooks:build_wheels"<br>
# Applies monkeypatches, then does:<br>
# setup.py dist_info && setup.py build_ext -i<br>
build-in-place = "pip.pypackage_hooks:build_in_place"<br>
<br>
This is also useful for any other installation tools that may want to<br>
support version 0 sdists without having to implement bug-for-bug<br>
compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is<br>
present, they can use this as a default.<br>
<br>
**Example 2:** For packages using numpy.distutils. This is identical<br>
to the distutils / setuptools example above, except that numpy is<br>
moved into the list of static build requirements. Right now, most<br>
projects using numpy.distutils don't bother trying to declare this<br>
dependency, and instead simply error out if numpy is not already<br>
installed. This is because currently the only way to declare a build<br>
dependency is via the ``setup_requires`` argument to the ``setup``<br>
function, and in this case the ``setup`` function is<br>
``numpy.distutils.setup``, which... obviously doesn't work very well.<br>
Drop this ``_pypackage.cfg`` into an existing project like this and it<br>
will become robustly pip-installable with no further changes::<br>
<br>
[build]<br>
requirements = """<br>
numpy<br>
pip >= whatever<br>
wheel<br>
"""<br>
requirements-dynamic = "pip.pypackage_hooks:setup_requirements"<br>
build-wheels = "pip.pypackage_hooks:build_wheels"<br>
build-in-place = "pip.pypackage_hooks:build_in_place"<br>
<br>
**Example 3:** `flit <<a href="https://github.com/takluyver/flit" rel="noreferrer" target="_blank">https://github.com/takluyver/flit</a>>`_ is a tool<br>
designed to make distributing simple packages simple, but it currently<br>
has no support for sdists, and for convenience includes its own<br>
installation code that's redundant with that in pip. These 4 lines of<br>
boilerplate make any flit-using source tree pip-installable, and lets<br>
flit get out of the package installation business::<br>
<br>
[build]<br>
requirements = "flit"<br>
build-wheels = "flit.pypackage_hooks:build_wheels"<br>
build-in-place = "flit.pypackage_hooks:build_in_place"<br>
<br>
<br>
FAQ<br>
===<br>
<br>
**Why is it version 1 instead of version 2?** Because the legacy sdist<br>
format is barely a format at all, and to `remind us to keep things<br>
simple <<a href="https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect" rel="noreferrer" target="_blank">https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect</a>>`_.<br>
<br>
**What about cross-compilation?** Standardizing an interface for<br>
cross-compilation seems premature given how complicated the<br>
configuration required can be, the lack of an existing de facto<br>
standard, and the authors of this PEP's inexperience with<br>
cross-compilation. This would be a great target for future extensions,<br>
though. In the mean time, there's no requirement that<br>
``_pypackage/_pypackage.cfg`` contain the *only* entry points to a<br>
project's build system -- packages that want to support<br>
cross-compilation can still do so, they'll just need to include a<br>
README explaining how to do it.<br>
<br>
**PEP 426 says that the new sdist format will support automatically<br>
creating policy-compliant .deb/.rpm packages. What happened to that?**<br>
Step 1: enhance the wheel format as necessary so that a wheel can be<br>
automatically converted into a policy-compliant .deb/.rpm package (see<br>
PEP 491). Step 2: make it possible to automatically turn sdists into<br>
wheels (this PEP). Step 3: we're done.<br>
<br>
**What about automatically running tests?** Arguably this is another<br>
thing that should be pushed off to wheel metadata instead of sdist<br>
metadata: it's good practice to include tests inside your built<br>
distribution so that end-users can test their install (and see above<br>
re: our focus here being on stuff that end-users want to do, not<br>
dedicated package developers), there are lots of packages that have to<br>
be built before they can be tested anyway (e.g. because of binary<br>
extensions), and in any case it's good practice to test against an<br>
installed version in order to make sure your install code works<br>
properly. But even if we do want this in sdist, then it's hardly<br>
urgent (e.g. there is no ``pip test`` that people will miss), so we<br>
defer that for a future extension to avoid blocking the core<br>
functionality.<br>
<span><font color="#888888"><br>
--<br>
Nathaniel J. Smith -- <a href="http://vorpus.org" rel="noreferrer" target="_blank">http://vorpus.org</a><br>
_______________________________________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" target="_blank">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
</font></span></blockquote></div><br></div>
_______________________________________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" target="_blank">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
</blockquote></div></blockquote></div>