On 28 June 2018 at 02:37, Nathaniel Smith
On Wed, Jun 27, 2018 at 2:25 PM, Paul Moore
wrote: On 27 June 2018 at 22:09, Pradyun Gedam
wrote: On Wed, Jun 27, 2018 at 9:15 PM Paul Moore
wrote:
But Brett clearly stated that he views PEP 518 as stating that the build-system.requires key in pyproject.toml is *not* optional. And I think that's the correct reading of the PEP.
FWIW, I'm a co-author on the PEP, and I've apparently been shipping software without this key for most of a year: https://github.com/python-trio/trio/blob/master/pyproject.toml Clearly I should have read my PEP more closely...
Oh! I'm really sorry - I thought you'd seen the discussion and the fact that you hadn't responded to Brett's assertion directly meant you were OK with it. I think I must have been skimming at the time and not fully understood the point you were making in your post. My error, and thanks for making me aware of it.
OK. But I'm going to take the view that having explicitly requested clarification on distutils-sig, if we want to do anything other than reject a pyproject.toml with a missing build-system.requires as invalid, we need to first of all get the PEP changed. And for that we'll have to have a good use case to justify the change. Personally, I don't see the value.
If pip starts erroring out when pyproject.toml is missing build-system.requires, then 'pip install twisted' will stop working and everyone will start screaming on twitter: https://github.com/twisted/twisted/blob/trunk/pyproject.toml
(Trio won't actually be affected b/c it has a universal wheel, so pip always uses that instead of the sdist. But twisted only has wheels for python 2 on Windows; all other configurations build from source, and thus will break.)
I can see the practicality argument here. I don't think (correct me if I'm wrong!) that anyone particularly anticipated when we were drafting PEP 518 that the tools section would be used by anything other than build tools - I know I hadn't anticipated that usage. In hindsight it's an obvious thing to do, but it does mean that pyproject.toml is more of a "general project configuration" file than purely a build configuration one.
We probably need to update the PEPs some here anyway, if only for clarity.
Agreed
Also, right now PEP 517 says "If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should fall back to running setup.py." But thinking about it again, we probably don't want to do this, because it adds Yet Another build configuration that pip has to handle:
1. no pyproject.toml -> legacy non-isolated build via legacy setup.py support 2. pyproject.toml without build-backend specified -> new isolated build via legacy setup.py support 3. pyproject.toml with build-backend specified -> new isolated build via new build-backend support
Once pip and setuptools both have PEP 517 build backend support (which should hopefully happen soon?), then option (2) will become weird and unnecessary. It would be nice to get rid of it. So I think we'll be having a similar discussion in a few months about how to handle pyproject.toml without build-backend keys. At that point we'll have a few options:
- We could say that build-system.build-backend is mandatory, and error out if it's missing. But that will break the world (probably even more so than making build-system.requires mandatory now will break the world, since by the time this happens there will be more pyproject.toml files in the wild).
- We could shrug and say that updating PEPs is a lot of work so let's just go with what we wrote way back when. But that forces pip to forever carry around code to implement both option (2) and (3), which do essentially the same thing except with extra code and probably there will be weird bugs that show up in one configuration or the other, etc.
- We could say that when build-system.build-backend is missing, it defaults to "setuptools". But it's *really weird* to have a default value for build-system.build-backend while not having one for build-system.requires, because, effectively, the interpretation of build-system.build-backend depends on the value of build-system.requires. The build backend is an object that gets looked up inside those required packages. If we're going to be like "we have no idea what setuptools is, you have to tell us to install it every time", then it's very weird to turn around in the next line like "oh yeah we *use* setuptools by default, we just don't *install* it by default".
So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
That's certainly the other option that we'd want to follow if the keys aren't mandatory. Focusing back on the PEP 518 side of things, the biggest problem is that the behaviour under build-system.requires = ["setuptools", "wheel"] is somewhat different from the legacy code path (although I'd have to do some research to say exactly how - at the moment things we proposed to do, things we plan on doing, and things we actually do right now, are hopelessly muddled in my mind) and I think that switching mode triggered by the existence of pyproject.toml has caused problems for some people (although that may only be because of build isolation, and the --no-build-isolation flag covers that case, albeit by making it the end user's problem rather than the project's). But I don't have a strong opinion either way here - that's why I originally suggested bringing the discussion to distutils-sig. I'm happy for pip to go with whatever the consensus ends up being. And whatever way it goes, a PEP update to make the intent explicit would probably be worthwhile.
This is already a feature request for pip for a different use case and I think it's a reasonable request.
What's the issue number? I don't recall seeing anything like this.
I found that discussion confusing at the time, and on rereading I don't think my view has changed :-(
That discussion is indeed pretty confusing, but the basic idea of letting the user override build requires is easy to motivate.
Say you're trying to install 'foomatic v1.2', which has a build requirement of "setuptools". But oops, setuptools v45.0 was just released, and it turns out that foomatic v1.2 depends on some internal implementation details of setuptools that changed in this release. That's annoying. Probably you should file a bug, and the foomatic authors will fix it and eventually make a new release. But that doesn't help you *right now*, when you're trying to get out an emergency deploy and your boss is breathing down your neck and you just need the stupid foomatic v1.2 to install so you can get on with things. Wouldn't it be nice (tm), if you could do:
pip install foomatic==1.2 --force-build-requirement='setuptools<45'
Hmm, OK. For that scenario, I'd argue the user can just say --no-build-isolation and set up the build environment manually. There's a risk here that we're creating ever more complicated mechanisms for relatively niche corner cases[1], and I think we need to be very careful to examine the trade-offs between code (and design) complexity and overall utility. The use of pip in development workflows is already a huge source of complexity (people have incredibly diverse workflows, which they typically cannot describe very well, because they often have no idea how unusual what they are doing can be). We have to cater for at least some of that, but (with my pip maintainer hat on) I think we need to draw the line somewhere, if only because of lack of manpower. Paul [1] From my POV, pip building from source is already a relatively niche case - and even if that's not the reality today for some users, I remain pretty firmly convinced that it's what we should be aiming for.