[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Robert Collins]

On 12 October 2015 at 21:15, Thomas Kluyver <takowl at gmail.com> wrote:
> On 12 October 2015 at 05:37, Robert Collins <robertc at robertcollins.net>
> wrote:
>>
>> And a bugfix :) - I didn't link to the docs for the build system
>> interface we have today -
>>
>> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface
>
>
> I'm happy to see that the command line interface pip requires from setup.py
> is now documented. But the first thing it describes is the egg_info command,
> and the description of what that involves is basically 'whatever setuptools
> does'. Egg info is not any kind of standard, AFAIK - unlike dist info. One

https://www.python.org/dev/peps/pep-0314/ describes PKG_INFO, which
can in principle include dependency data. I don't know why they're not
being fully populated - I suspect its another case of
forward-looking-spec-not-quite-right-and-thus-never-used. Someone with
more history on Python packaging will have to fill that in. One option
would be to push on that.

> of my main goals in writing flit is to build wheels without involving
> setuptools at all, so I'm probably never going to implement that.

Well thats fair enough. but: pip uses pkg_resources to find out whats
installed and what it depends on, and that pulls out of egg-info on
the file system for non-wheel-installs (dist-info for wheel installs).
So, if that isn't well enough documented, it would be entirely
reasonable to make a PEP to capture it so that you can emit it with
confidence. There's already buy-in from everyone in the ecosystem to
use PEPs to document interoperability impacting changes across these
tools, so I can't imagine it being a contentious proposal.

> I also don't want to have a setup.py in the VCS at all, because it's an
> invitation for people to run setup.py whatever, and then file bug reports
> that it doesn't do exactly what setuptools/distutils would do.

Ok, so I understand that frustration. You're going to get those
through pip too though - right now there's a giant escape clause in
pip where arbitrary options are passed through to setup.py, because we
don't have semantic modelling of all the things that people want to
do. And until thats achieved - and the options aged out - folk are
going to use those options and then be surprised when flit using
packages don't behave the same way.

> This is what I like about Nathaniel's proposal. By looking for a new file,
> it clearly indicates that this is a new style Python source tree, without
> potential confusion over what setup.py does, and it lets us specify a simple
> build system interface based on standardised pieces like wheels rather than
> messing around with the details of setuptools.

The problem with 'simple' is that we've got a rich (not just old:
fully featured) interface. Building something that isn't complected
out from behind that is both valuable and hard. An 85% solution or 95%
solution will retain the trap-doors that let folk tickle the system
from the outside, and thats going to lead back to the same bug reports
you don't want IMO :/.

I'm not against a spec that avoids the need for each tree to have a
setup.py - I just don't think its going to be the end of the story in
any deep fashion :).

Here's a draft one I'd be happy with (with no boilerplate or
anything). For commands we can define a small number environment
variables like e.g. OUTPUT_PATH and PYTHON have those supplied when
invoking the command.

Python packaging config in $root/pypa.yaml

Defined keys:
----
version:
  # not needed yet, since its a new file, defaults to 1.
setup-requires:
  - requirement
  - requirement[extra]
  - requirement:markers
  - requirement[extra]:markers
build-tool:
  # basic command that will spit a hunk of json back to the caller defining the
  # commands to use with the build tool.
----

Build tool output json (in yaml format for ease of reading):
version:
  # not needed yet, defaults to 1
egg-info:
   # comand to run to generate an egg-info directory
   # only used if dist-info is not defined
dist-info:
   # command to run to generate a dist-info directory
wheel:
   # command to build a wheel
develop:
   # command to do an in-place installation
install:
  # command to do a direct installation
  # only used if wheeling failed
provided-by:
  # the requirement that provides the build system.
  # this is used to facilitate caching the build tool output
  # so that we don't need to probe everytime. (e.g. if the
  # same dependency version is present in two cases
  # we can presume the output - I threw this in as an
  # obvious tweak for large dependency chains but it is
  # probably not worth it for typical 10-20 package things

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud