[Distutils] setup_requires for dev environments

[Daniel Holth]

You ought to be able to get away with not supporting it in warehouse,
we will surely be able to write legacy .egg-info to sdists
indefinitely.

On Mon, Mar 16, 2015 at 3:09 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> On Mar 16, 2015, at 2:33 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> On 16 March 2015 at 17:14, Donald Stufft <donald at stufft.io> wrote:
>>> The bulk of the effort of pushing the standards, pip, and PyPI through is done
>>> by a handful of people, and of those handful I believe that the largest share
>>> is done by myself. That's not to toot my own horn or any such nonsense but to
>>> simply state the fact that the available bandwidth of people able and willing
>>> to work on problems is low. However the things we bless here as official are
>>> things which need to be able to last for a decade or more, which means that
>>> they do need careful consideration before we bless them.
>>
>> As a serious question - is there anything I (or indeed anyone else)
>> can do to make progress on PEP 426? If I'm honest, right now I don't
>> exactly see what tool changes are needed to change it from draft to
>> accepted to actually implemented.
>>
>> As far as I can see, acceptance consists largely of someone, somehow,
>> confirming that there are no major loopholes in the spec. I think that
>> mostly comes down to the fact that no-one has raised objections since
>> the PEP was published, plus someone with experience of some of the
>> more difficult distribution scenarios sanity-checking things. And
>> then, getting it implemented in tools.
>>
>> I guess the tool side consists of:
>>
>> 1. Making pip write a pydist.json file when installing from wheel.
>> 2. Making setuptools write pydist.json when installing from sdist, and
>> when creating a sdist.
>> 3. Making wheel write pydist.json when writing a wheel.
>>
>> (Also, when distlib writes a wheel, it should write pydist.json, but
>> I'm considering distlib as "non-core" for the sake of this
>> discussion).
>>
>> There's also presumably work to add support for specifying some of the
>> new metadata in setup.py, which I guess is setuptools work again.
>>
>> Have I missed anything crucial?
>> Paul
>>
>> PS I'm ignoring the "standard metadata extensions" PEP where console
>> wrappers, and post-install scripts figure. Those are probably bigger
>> design issues.
>
> Probably similar to what I did for PEP 440.
>
> Start branches implementing it and try to run as much stuff through it as
> possible to see how it fails. When implementing PEP 440 I was running every
> version number that existed on PyPI through it to test to make sure things
> would work. That’s not going to be possible with PEP 426, but ideally we should
> be able to get branches in the various tools that work on it, grab the latest
> versions of the top N packages (or the latest versions of everything) and
> compare the results.
>
> Another thing is determining if there's anything else we can/should split out
> from PEP 426 to narrow the scope. A quick skim to refresh myself doesn't show
> me anything that stands out, but some thought to this wouldn't hurt.
>
> It’d also probably not hurt to go through the setuptools and pip bug trackers
> to see if there are any relevant issues and see how they would be effected by
> the new standard.
>
> PEP 426 itself isn't much in the way of groundbreaking. It's basically taking
> the dynamic metadata and "compiling" it down into a static form which is JSON
> based. The "easy" metadata (name etc) is a no brainer and probably doesn't
> require much poking, trying out an example of an extension probably wouldn't
> be the worst thing either. Even if stanardizing script wrappers, for instance,
> is held off, getting a demo extension using it would validate the standard.
>
> Potentially more thought/guidance should be mentioned in how projects should
> straddle the line between legacy systems. If a new style metadata exists in a
> format in addition to an old syle should the new style be preferred? Should
> there be any sanity checks to make sure the two aren't completely different?
>
> As an aside, one thing I've had in the back of my mind, is looking at the
> possibility of defining the environment markers which specify versions as
> PEP 440 versions and enabling all the same comparisons with them as we get
> in the specifiers. I think the string based comparison we currently do is
> kinda janky and any installer already has the tooling to handle specifiers.
>
> Generally what I did with PEP 440, which I think worked well, is I had
> everything pretty much implemented prior to it get accepting, and we were then
> able to use the things I found out while implementing it to adjust the PEP
> before it was formally accepted. I just didn’t merge anything until that point.
> This was pretty valuable in findings things where the PEP was too vague for
> someone to make an indepdent implementation going by what was in the PEP, or
> if it specified something where implementing it turned out to be
> hard/problematic, etc.
>
> A major reason why I’m personally focusing on Warehouse first is that
> integrating with Warehouse will be easier than integrating with PyPI legacy.
> However that doesn’t have to block anyone else, that’s just myself not wanting
> to spend the time integrating a new major metadata version into the old legacy
> code base.
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>