[Distutils] setup_requires for dev environments

[Donald Stufft]

> On Mar 16, 2015, at 2:33 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 16 March 2015 at 17:14, Donald Stufft <donald at stufft.io> wrote:
>> The bulk of the effort of pushing the standards, pip, and PyPI through is done
>> by a handful of people, and of those handful I believe that the largest share
>> is done by myself. That's not to toot my own horn or any such nonsense but to
>> simply state the fact that the available bandwidth of people able and willing
>> to work on problems is low. However the things we bless here as official are
>> things which need to be able to last for a decade or more, which means that
>> they do need careful consideration before we bless them.
> 
> As a serious question - is there anything I (or indeed anyone else)
> can do to make progress on PEP 426? If I'm honest, right now I don't
> exactly see what tool changes are needed to change it from draft to
> accepted to actually implemented.
> 
> As far as I can see, acceptance consists largely of someone, somehow,
> confirming that there are no major loopholes in the spec. I think that
> mostly comes down to the fact that no-one has raised objections since
> the PEP was published, plus someone with experience of some of the
> more difficult distribution scenarios sanity-checking things. And
> then, getting it implemented in tools.
> 
> I guess the tool side consists of:
> 
> 1. Making pip write a pydist.json file when installing from wheel.
> 2. Making setuptools write pydist.json when installing from sdist, and
> when creating a sdist.
> 3. Making wheel write pydist.json when writing a wheel.
> 
> (Also, when distlib writes a wheel, it should write pydist.json, but
> I'm considering distlib as "non-core" for the sake of this
> discussion).
> 
> There's also presumably work to add support for specifying some of the
> new metadata in setup.py, which I guess is setuptools work again.
> 
> Have I missed anything crucial?
> Paul
> 
> PS I'm ignoring the "standard metadata extensions" PEP where console
> wrappers, and post-install scripts figure. Those are probably bigger
> design issues.

Probably similar to what I did for PEP 440.

Start branches implementing it and try to run as much stuff through it as
possible to see how it fails. When implementing PEP 440 I was running every
version number that existed on PyPI through it to test to make sure things
would work. That’s not going to be possible with PEP 426, but ideally we should
be able to get branches in the various tools that work on it, grab the latest
versions of the top N packages (or the latest versions of everything) and
compare the results.

Another thing is determining if there's anything else we can/should split out
from PEP 426 to narrow the scope. A quick skim to refresh myself doesn't show
me anything that stands out, but some thought to this wouldn't hurt.

It’d also probably not hurt to go through the setuptools and pip bug trackers
to see if there are any relevant issues and see how they would be effected by
the new standard.

PEP 426 itself isn't much in the way of groundbreaking. It's basically taking
the dynamic metadata and "compiling" it down into a static form which is JSON
based. The "easy" metadata (name etc) is a no brainer and probably doesn't
require much poking, trying out an example of an extension probably wouldn't
be the worst thing either. Even if stanardizing script wrappers, for instance,
is held off, getting a demo extension using it would validate the standard.

Potentially more thought/guidance should be mentioned in how projects should
straddle the line between legacy systems. If a new style metadata exists in a
format in addition to an old syle should the new style be preferred? Should
there be any sanity checks to make sure the two aren't completely different?

As an aside, one thing I've had in the back of my mind, is looking at the
possibility of defining the environment markers which specify versions as
PEP 440 versions and enabling all the same comparisons with them as we get
in the specifiers. I think the string based comparison we currently do is
kinda janky and any installer already has the tooling to handle specifiers.

Generally what I did with PEP 440, which I think worked well, is I had
everything pretty much implemented prior to it get accepting, and we were then
able to use the things I found out while implementing it to adjust the PEP
before it was formally accepted. I just didn’t merge anything until that point.
This was pretty valuable in findings things where the PEP was too vague for
someone to make an indepdent implementation going by what was in the PEP, or
if it specified something where implementing it turned out to be
hard/problematic, etc.

A major reason why I’m personally focusing on Warehouse first is that
integrating with Warehouse will be easier than integrating with PyPI legacy.
However that doesn’t have to block anyone else, that’s just myself not wanting
to spend the time integrating a new major metadata version into the old legacy
code base.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150316/c84ac35f/attachment.sig>