[Distutils] Meeting info re: sdists

[Wes Turner]

On Oct 13, 2015 6:01 PM, "Robert Collins" <robertc at robertcollins.net> wrote:
>
> On 14 October 2015 at 05:09, Nathaniel Smith <njs at pobox.com> wrote:
> > Hi all,
> >
> > I was going to send around a summary from this meeting, but I seem to
have
> > come down with the flu last night. So if someone else wants to do it
while
> > it's fresh in our minds, that'd be great, or else I'll try to get to it
next
> > week or so.
> >
> > Cheers,
> > -n
> >
> > On Oct 12, 2015 11:01 AM, "Nathaniel Smith" <njs at pobox.com> wrote:
> >>
> >> Hangouts URL:
> >>   https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a
> >>
> >> Shared document for notes:
> >>
> >>
https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing
>
> So that ^ is worth a read for any interested party.
>
> The summary:
>
> We agreed on the following design constraints(MUSTS):
>
> * Graceful forward compat (old pip either can install, or fails
> cleanly, on new VCS and sdists, pypi can be changed )
> * Backwards compat (new pip can install both old and new VCS/sdists,
> new pypi can handle old and new sdists)
> * Setup-requires static to the point the build system can be installed by
pip
> * Setup-requirements need to be isolated: must be able to use
> different incompatible versions of the same setup-requirement during
> one install run.
> * Describing how to use the build system has to be user friendly.
> [e.g. json configs are out]

(.cfg | .yml) > JSONLD

Should these build settings all be included in e.g. PEP 426 + JSONLD
metadata.jsonld?
- for backward compat, there would then be both metadata.json and
metadata.jsonld (because of normative JSONLD form and PEP 426+)

.cfg (ConfigParse): (section, key, value)

.yml (YAML lib xyz): easily maps to JSONLD (which always maps to an RDF
triple graph (of linked typed packages with attributes, generated from
[...]))

>
> And the following design goals(SHOULDs):
>
> * Be able to use things like flit cleanly within pip without needing a
> setup.py in-tree.
> * Long term - reduce variation in code paths
> * Long term - reduce amount of dynamism? in install pipeline?

* OS Packaging compat (--prefix)
* Conda solves for this. (static files with package-level checksums)

>   * We may never be able to remove it entirely, but being able to not
> run egg-info in the common case should be achievable
>   * Things that have reason to change (deps) are more reasonable to be
> dynamic (even with PEP-426 markers there are exceptions)
>   * project description, version etc in sdist have no reason to be dynamic
> * Files in VCS should not need to be different in sdist - ever
>
>
> We had broad agreement on the basic design:
> * There will be a file that describes how to install the build tool,
> and how pip or other systems should invoke it
> * It will be minimal: this file is strictly additive to the system, no
> overlap with existing things (even setup-requires - because
> setup-requires presumes the buildtool (setuptools) is already
> installed)
>
> I *think* we had broad agreement that its not desirable, overall, to
> try and skip steps like metadata generation - because with a resolver
> we'd end up potentially doing things like building dozens of versions
> of scipy just to extract metadata about dependencies.
>
> I like the term 'bootstrap-requires' -because we're bootstrapping the
> ability to use the build tool.
>
> The build tool interface needs to be sufficient that one could write a
> shim to existing setuptools in it. Previous posts have described what
> that might look like, but to reprise my sketch it had the following
> entries
> schema: #was version
> egg-info:
> dist-info:
> wheel:
> develop:
> install:
> provided-by:
>
> We have a design decision to make.
>
> We could make bootstrap-requires be a superset of setup-requires. That
> is, projects should put anything that is functionally a setup-requires
> thing into bootstrap-requires. This had the ad-hoc effect of dictating
> to setuptools how static setup-requires would be expressed. I don't
> tlike that.
>
> Alternatively we can make the interface permit querying for things to
> install before running any of the other things in the build system
> interface: e.g. one way would be
>
> build-requires:
>   # command to query for package specific build dependencies
>   # returns a PEP-426 JSON file, only the keys
> https://www.python.org/dev/peps/pep-0426/#metadata-version and
> "build_requires" are consulted.
>
> Alternatives here include:
>  - just generating {dist.egg}-info twice - once to get setup_requires,
> once to get everything once thats resolved
>  - drop dist-info and egg-info and use PEP-426 for those calls as well
>  - don't use pep-426 at all, just use requires.txt format which is
> already well established
>  - Use a pep-345 file
>
> I think pep-426 is ok since we're already using it within wheels
> (metadata.json). I'm a little nervous about dropping
> egg-info/dist-info - but hell, lets try it (since its common to have a
> proof of concept before accepting peps, we'll know it works for
> setuptools since that should be in the proof of concept). That would
> give a build tool interface of:
>
> schema: #was version
> build-requires: # command to get a pep-426 json blob on stdout with
> build_requires populated
> dist-info: # command to get a pep-426 json blob on stdout with
> everything populated
> wheel: # build a wheel
> develop: # do a develop install
> install: # do a direct install
> provided-by: # allows caching the build tool interface
>
> We need to define the exact environment that the build tool will be
> run in: will it inherit already installed things from the install
> target? Will it be isolated or hermetic? Being able to fix things at a
> distance is important - and remembering we have a lot of legacy paths,
> that means a greenfield approach isn't very viable.
>
> Separately we need to define how-and-when dist-info will be called in
> the case of sdists which have valid static metadata. Thats
> sufficiently different we think it should be a separate PEP.
>
> Go forward:
>
> Nathaniel to write two PEPs:
>  * bootstrap-requirements and build system interface - enables
> multiple build systems
>  * usable static metadata in sdists - allow the common case to do less
> work by ensuring valid metadata in sdists
>
> -Rob
> --
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151013/aa46879d/attachment-0001.html>