On Wed, May 4, 2016 at 10:42 PM, Robert Collins <robertc@robertcollins.net> wrote:
On 5 May 2016 at 16:22, Nathaniel Smith <njs@pobox.com> wrote: ...
I'm sympathetic to the general approach, but on net I think I prefer a slightly different proposal.
Downsides to just standardizing [setup_requires]:
If I write a PEP, it won't be standardising setup_requires, it will be standardising bootstrap requirements. The distinction is nuance but critical:
- we don't expect setuptools to ever need to honour it (of course, it could, should it choose to, and ease of adoption may suggest that it should) - as a new feature, making it opt in allows folk to adopt it when they are ready; if it was literally just a new home for setup_requires, we may see build systems auto-populating it, and the potential swamp you describe below would then apply.
The main argument I was making there was that it needs to be a new opt-in thing, so if we're agreed on that then great :-).
- if projects have existing ones, that's actually kinda good / kinda bad -- pip has never been respecting these before, so if we suddenly start doing that then existing stuff could break. I don't know how likely this is, but experience suggests that *something* will break and make someone angry. (I'm still blinking at getting angry complaints arguing that uploading a wheel to pypi, where before there was only an sdist, should be treated as a compatibility-breaking change that requires a new version etc.)
Yes, things will break: anyone using this will need a new pip, by definition. Not everyone will be willing to wait 10 years before using it :).
Just to clarify (since we seem to agree): I meant that if pip starts interpreting an existing setup.cfg thing, then the new-pip/old-package situation could break, which would be bad.
- IMO an extremely valuable aspect of this new declarative setup-requirements thing is that it gives us an opportunity to switch to enforcing the accuracy of this metadata. Right now we have a swamp we need to drain, where there's really no way to know what environment any given setup.py needs to run. Sometimes there are setup_requires, sometimes not; if there are setup_requires then sometimes they're
Huh? I've not encountered any of this, ever. I'd love some examples to go look at. The only issue I've ever had with setup_requires is the easy_install stuff it ties into.
I don't think I've ever seen a package that had accurate setup_requires (outside the trivial case of packages where setup_requires=[] is accurate). Scientific packages in particular universally have undeclared setup requirements.
...
instead of punishing them -- is if we make the rule be "projects that use the declarative setup-requirements feature also get isolated build environments". (Then the messaging is: "see, this helps you check that you actually set it up right! if your new metadata works for you in testing, it'll also work for your users!) But this again would mean we can't reuse the existing [setup_requires] section.
I'm very much against forcing isolated build environments as part of this effort. I get where you are coming from, but really it conflates two entirely separate things, and you'll *utterly* break building anything with dependencies that are e.g. SWIG based unless you increase the size of the PEP by about 10-fold. (Thats not hyperbole, I think).
Okay, now's my turn to be baffled :-). I literally have no idea what you're talking about here. What would this 10x longer PEP be talking about? Why would this be needed?
Working around that will require a bunch of UX work, and its transitive: you have to expose how-to-workaround-the-fact-that-all-our-deps-are-not-installable all the way up the chain. That's probably a good thing to do (e.g. see bindep, or the aborted callout-to-system-packages we proposed after PyCon AU last year), but tying these two things together is not necessary, though I can certainly see the appeal; the main impact I see is that it will just impede adoption.
What are these things that aren't pip-installable and why isn't the solution to fix that? I definitely don't want to break things that work now, but providing new features that incentivize folks to clean up their stuff is a good thing, surely? Yeah, it means that the bootstrap-requirements stuff will take some time and cleanup to spread, but that's life. We've spent a huge amount of effort on reaching the point where pretty much everything *can* be made pip installable. Heck, *PyQt5*, which is my personal benchmark for a probably-totally-unpackageable package, announced last week that they now have binary wheels on pypi for all of Win/Mac/Linux: https://pypi.python.org/pypi/PyQt5/5.6 I want to work towards a world where this stuff just works, not keep holding the entire ecosystem back with compromise hacks to work around a minority of broken packages.
The reality, AFAICT, is that most projects with undeclared build deps today get corrected fairly quickly: a bug is filed, folk fix it, and we move on. A robotic system that isolates everything such that folk *cannot* fix it is much less usable, and I'm very much in favour of pragmatism here.
Again, in my world ~100% of packages have undeclared build deps...
- setup.cfg is kind of a terrible format for standardizing things because the only definition of the format is "read the ConfigParser source". You cannot parse setup.cfg from a non-Python language. And the *only* benefit is that it already exists; teaching pip to read pybuild.json or pybuild.toml instead would be completely trivial. So if we aren't going to try and grandfather in existing [setup_requires] sections, then we might as well switch to a better file format at the same time -- this is not the hard part.
The patch to read a single list-valued key out of setup.cfg is trivial and shallow. We've not managed to settle on consensus on a file format choice in a year of debate. I hold zero confidence we will going forward either. If the BDFL delegate makes a call - fine. I read Nick's earlier email in the thread as such a thing TBH :).
Oh sure, I think everyone agrees that the file format choice is not a make-or-break decision and that ultimately this is a good case for of BDFL-style pronouncement rather than endless consensus seeking. But I'm still allowed to make technical arguments for why I think the BDFL-delegate should pronounce one way or another :-).
So I like the idea of splitting out the declarative setup-requirements PEP from the new build system hook interface PEP, but I think that the way we should do this is by defining a new pybuild.whatever file like the ones in PEP 516 / PEP 517 (they're identical in this regard) that *only* has schema-version and bootstrap-requirements, and then we can add the build backend key as a second step.
I think the risk there is that that presumes the answer, and without the abstract build effort actually moving forward we may be years before we actually see the file being used for anything else. If ever.
I see the risk, but I think it's minimal. We're not presuming anything about the answer except that somehow it will involve adding some kind of static metadata, which seems very safe. -n -- Nathaniel J. Smith -- https://vorpus.org