[Distutils] Meeting info re: sdists

[Robert Collins]

On 15 October 2015 at 06:06, Wes Turner <wes.turner at gmail.com> wrote:
>
> On Oct 14, 2015 3:33 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:
>>
>> On 14 October 2015 at 01:46, Robert Collins <robertc at robertcollins.net>
>> wrote:
>> > On 14 October 2015 at 13:25, Marcus Smith <qwcode at gmail.com> wrote:
>> >>
>> >>
>> >> thanks for the summary!
>>
>> Agreed.
>>
>> >>
>> >>>   * Things that have reason to change (deps) are more reasonable to be
>> >>> dynamic (even with PEP-426 markers there are exceptions)
>> >>
>> >>
>> >> as we know, for *many* cases, run-time deps aren't dynamic.
>> >> is there a consensus for those cases?  exist in the sdist metadata? or
>> >> no?
>> >
>> > The plan we hashed out (this would be the new PEP on static metadata in
>> > sdists)
>> >  - pip etc to lint and error if they encounter a malformed dist-info in
>> > an sdist
>> >  - then start putting dist-info, not egg-info into sdists
>> >  - for any field if the field is entirely absent, we'll get it by
>> > running the dist-info build-tool command I described in the other
>> > mails
>
> so, validate and transform to OrderedDicts which can then be json.dump'd to
> metadata.jsonld?

That would be one strategy going forward for utilising jsonld.
However, *this current effort* is not altering the existing
definitions of metadata that we have today - and doing so would
effectively put the initiative to make things like flit have an
interopability standard back 5-10 **years** [because the half life of
enterprise software environments is ridiculously long).

>> >
>> > Concretely:
>> > {'build_requires': []} -> no build requirements
>> > {} -> get build requirements by running the build system
>>
>> One use case that I don't think is covered here is publishing
>> dependency metadata via PyPI. I believe distlib needs this for some
>> functions (at the moment, I think it uses an externally hosted set of
>> package dependency data that's maintained by Vinay), and I know there
>> have been a number of utility scripts I've needed to write that I
>> simply haven't been able to because doing so would involve a
>> significant amount of extra work downloading and running package code.
>>
>
>> If there are dependencies that are only detectable at wheel build
>> time, then so be it (I'm still looking for an example, but it's clear
>> that the potential is there) but I'd like some way of getting at
>> whatever dependency information a wheel (source or binary) provides
>> via the PyPI JSON API - and I'd like an assurance that if dependency
>> information *can* be determined statically, it will be.----------*
>
> * the ship has already sailed on a static declarative dependency model
> (because 'if sys.platform:' in setup.py

Our strategy here is to provide newer features and encourage folk to
adopt them. Where the feature will work correctly in pre-its-existence
environments, we can expect better uptake than where a flag day is
required (and we get back into that decade long wait...). PEP-426
conditionals address most (but not all) of the dynamic dependencies
cases. We can't remove dynamic dependencies until a) -all- the cases
for things we want on PyPI are solved and b) every single package that
needs it has migrated to use the new feature. We haven't delivered (a)
yet, so we have to design any initiatives that we want to be available
now, to work with dynamic dependencies: but its not a statement that
we've stopped pushing for static expression of them all.

> * I agree that we should preserve all (dynamically determine at setup.py
> runtime) requires edges as static JSON-LD

There's an existing metadata format: PEP-426. Changing that would be a
different discussion, even though its technically draft, because
changing it imposes adoption and interopability challenges.

Finally I don't see any new data schemas in the proposed bootstrap
mechanisms - PEP-426 already covers requirement specifiers, and the
command line string templates we need are trivial.

so - I don't see anything that JSON-LD will help us with at this point
- other than introducing churn, when minimising the impact (to permit
adoption) is an explicit design point.

I don't know about the other contributors, but I don't have enough
bandwidth to tackle many proposals at once (and do them justice) - so
at least from my part, I want to defer any and all JSON-LD discussion
until we're making non-trivial changes to PEP-426. E.g. not for
metadata 2.1 (which is what I expect the fixed handling of python
2.7.10 and markers to end up in), but for metadata 3.x.

-Rob


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud