On 5 May 2016 at 18:32, Nathaniel Smith <njs@pobox.com> wrote:
On Wed, May 4, 2016 at 10:42 PM, Robert Collins ...
Yes, things will break: anyone using this will need a new pip, by definition. Not everyone will be willing to wait 10 years before using it :).
Just to clarify (since we seem to agree): I meant that if pip starts interpreting an existing setup.cfg thing, then the new-pip/old-package situation could break, which would be bad.
No. Old pip new package will break, new pip old package is entirely safe AFAICT.
- IMO an extremely valuable aspect of this new declarative setup-requirements thing is that it gives us an opportunity to switch to enforcing the accuracy of this metadata. Right now we have a swamp we need to drain, where there's really no way to know what environment any given setup.py needs to run. Sometimes there are setup_requires, sometimes not; if there are setup_requires then sometimes they're
Huh? I've not encountered any of this, ever. I'd love some examples to go look at. The only issue I've ever had with setup_requires is the easy_install stuff it ties into.
I don't think I've ever seen a package that had accurate setup_requires (outside the trivial case of packages where setup_requires=[] is accurate). Scientific packages in particular universally have undeclared setup requirements.
Are those requirements pip installable? ..
I'm very much against forcing isolated build environments as part of this effort. I get where you are coming from, but really it conflates two entirely separate things, and you'll *utterly* break building anything with dependencies that are e.g. SWIG based unless you increase the size of the PEP by about 10-fold. (Thats not hyperbole, I think).
Okay, now's my turn to be baffled :-). I literally have no idea what you're talking about here. What would this 10x longer PEP be talking about? Why would this be needed?
Take an i386 linux machine, and build something needing pyqt5 on it :). Currently, you apt-get/yum/dnf/etc install python-pyqt5, then run pip install. If using a virtualenv you enable system site packages. When you introduce isolation, the build will only have the standard library + whatever is declared as a dep: and pyqt5 has no source on PyPI. So the 10x thing is defining how the thing doing the isolation (e.g. pip) should handle things that can't be installed but are already available on the system. And that has to tunnel all the way out to the user, because its context specific, its not an attribute of the dependencies per se (since new releases can add or remove this situation), nor of the consuming thing (same reason). Ultimately, its not even an interopability question: pip could do isolated builds now, if it chose, and it has no ramifications as far as PEPs etc are concerned. ...
What are these things that aren't pip-installable and why isn't the solution to fix that? I definitely don't want to break things that work now, but providing new features that incentivize folks to clean up their stuff is a good thing, surely? Yeah, it means that the bootstrap-requirements stuff will take some time and cleanup to spread, but that's life.
We've a history in this group of biting off too much and things not getting executed. We're *still* in the final phases of deploying PEP-508, and it was conceptually trivial. I'm not arguing that we shouldn't make things better, I'm arguing that tying two separate things together because we *can* seems, based on the historical record, to be unwise.
We've spent a huge amount of effort on reaching the point where pretty much everything *can* be made pip installable. Heck, *PyQt5*, which is my personal benchmark for a probably-totally-unpackageable package, announced last week that they now have binary wheels on pypi for all of Win/Mac/Linux:
https://pypi.python.org/pypi/PyQt5/5.6
I want to work towards a world where this stuff just works, not keep holding the entire ecosystem back with compromise hacks to work around a minority of broken packages.
Sure, but the underlying problem here is that manylinux is a 90% solve: its great for the common cases but it doesn't actually solve the actual baseline problem: we can't represent the actual system dependencies needed to rebuild many Python packages. pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64 is going to be super common, Power8 another one, let alone less common but still extant and used architectures like PPC, itanium, or new ones like x86_32 [If I remember the abbreviation right - no, its not i386]. Solve that underlying problem - great, then isolation becomes an optimisation question for things without manylinux wheels. But if we don't solve it then isolation becomes a 'Can build X at all' question, which is qualitatively different. I'm all for solving the underlying problem, but not at the cost of *not solving* the 'any easy-install is triggered' problem for another X months while that work takes place.
The reality, AFAICT, is that most projects with undeclared build deps today get corrected fairly quickly: a bug is filed, folk fix it, and we move on. A robotic system that isolates everything such that folk *cannot* fix it is much less usable, and I'm very much in favour of pragmatism here.
Again, in my world ~100% of packages have undeclared build deps...
So - put a patch forward to pip to do isolated builds. If/when bug reports come in, we can solve them there. There's no standards body work involved in that as far as I can see.... -Rob -- Robert Collins <rbtcollins@hpe.com> Distinguished Technologist HP Converged Cloud