[Distutils] Meeting info re: sdists

[Robert Collins]

On 14 October 2015 at 05:09, Nathaniel Smith <njs at pobox.com> wrote:
> Hi all,
>
> I was going to send around a summary from this meeting, but I seem to have
> come down with the flu last night. So if someone else wants to do it while
> it's fresh in our minds, that'd be great, or else I'll try to get to it next
> week or so.
>
> Cheers,
> -n
>
> On Oct 12, 2015 11:01 AM, "Nathaniel Smith" <njs at pobox.com> wrote:
>>
>> Hangouts URL:
>>   https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a
>>
>> Shared document for notes:
>>
>> https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing

So that ^ is worth a read for any interested party.

The summary:

We agreed on the following design constraints(MUSTS):

* Graceful forward compat (old pip either can install, or fails
cleanly, on new VCS and sdists, pypi can be changed )
* Backwards compat (new pip can install both old and new VCS/sdists,
new pypi can handle old and new sdists)
* Setup-requires static to the point the build system can be installed by pip
* Setup-requirements need to be isolated: must be able to use
different incompatible versions of the same setup-requirement during
one install run.
* Describing how to use the build system has to be user friendly.
[e.g. json configs are out]

And the following design goals(SHOULDs):

* Be able to use things like flit cleanly within pip without needing a
setup.py in-tree.
* Long term - reduce variation in code paths
* Long term - reduce amount of dynamism? in install pipeline?
  * We may never be able to remove it entirely, but being able to not
run egg-info in the common case should be achievable
  * Things that have reason to change (deps) are more reasonable to be
dynamic (even with PEP-426 markers there are exceptions)
  * project description, version etc in sdist have no reason to be dynamic
* Files in VCS should not need to be different in sdist - ever


We had broad agreement on the basic design:
* There will be a file that describes how to install the build tool,
and how pip or other systems should invoke it
* It will be minimal: this file is strictly additive to the system, no
overlap with existing things (even setup-requires - because
setup-requires presumes the buildtool (setuptools) is already
installed)

I *think* we had broad agreement that its not desirable, overall, to
try and skip steps like metadata generation - because with a resolver
we'd end up potentially doing things like building dozens of versions
of scipy just to extract metadata about dependencies.

I like the term 'bootstrap-requires' -because we're bootstrapping the
ability to use the build tool.

The build tool interface needs to be sufficient that one could write a
shim to existing setuptools in it. Previous posts have described what
that might look like, but to reprise my sketch it had the following
entries
schema: #was version
egg-info:
dist-info:
wheel:
develop:
install:
provided-by:

We have a design decision to make.

We could make bootstrap-requires be a superset of setup-requires. That
is, projects should put anything that is functionally a setup-requires
thing into bootstrap-requires. This had the ad-hoc effect of dictating
to setuptools how static setup-requires would be expressed. I don't
tlike that.

Alternatively we can make the interface permit querying for things to
install before running any of the other things in the build system
interface: e.g. one way would be

build-requires:
  # command to query for package specific build dependencies
  # returns a PEP-426 JSON file, only the keys
https://www.python.org/dev/peps/pep-0426/#metadata-version and
"build_requires" are consulted.

Alternatives here include:
 - just generating {dist.egg}-info twice - once to get setup_requires,
once to get everything once thats resolved
 - drop dist-info and egg-info and use PEP-426 for those calls as well
 - don't use pep-426 at all, just use requires.txt format which is
already well established
 - Use a pep-345 file

I think pep-426 is ok since we're already using it within wheels
(metadata.json). I'm a little nervous about dropping
egg-info/dist-info - but hell, lets try it (since its common to have a
proof of concept before accepting peps, we'll know it works for
setuptools since that should be in the proof of concept). That would
give a build tool interface of:

schema: #was version
build-requires: # command to get a pep-426 json blob on stdout with
build_requires populated
dist-info: # command to get a pep-426 json blob on stdout with
everything populated
wheel: # build a wheel
develop: # do a develop install
install: # do a direct install
provided-by: # allows caching the build tool interface

We need to define the exact environment that the build tool will be
run in: will it inherit already installed things from the install
target? Will it be isolated or hermetic? Being able to fix things at a
distance is important - and remembering we have a lot of legacy paths,
that means a greenfield approach isn't very viable.

Separately we need to define how-and-when dist-info will be called in
the case of sdists which have valid static metadata. Thats
sufficiently different we think it should be a separate PEP.

Go forward:

Nathaniel to write two PEPs:
 * bootstrap-requirements and build system interface - enables
multiple build systems
 * usable static metadata in sdists - allow the common case to do less
work by ensuring valid metadata in sdists

-Rob
-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud