Thanks for continuing to push us forward on this, Thomas :) A small PEP readability request: given how the number of hooks has grown, could we get a section that just lists the required hooks and the optional hooks? Alternatively, give each hook its own subsection under "Build backend interface", so the table of contents at the top of the PEP serves as a quick summary. On 16 June 2017 at 06:43, Nathaniel Smith <njs@pobox.com> wrote:

On Thu, Jun 15, 2017 at 6:12 AM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:

...

Yes, it's PEP 517 again! Here's the current text:

https://www.python.org/dev/peps/pep-0517/

We currently say:

...

All other hooks [except get_build_requires] are executed in an environment which contains both the bootstrap requirements specified in the pyproject.toml hook and those specified by the get_build_requires hook.

It's not clear to me whether this should be required for the build_sdist and prepare_build_files hooks, nor whether there are any adverse consequences of specifying it like this anyway. Thoughts?

I think we should rename get_build_requires to get_build_wheel_requires, and add a get_build_sdist_requires. And the rule would be:

get_build_sdist_requires: can assume build-system.requires are available get_build_wheel_requires: can assume build-system.requires are available

build_sdist: can assume build-system.requires and get_build_sdist_requires are available prepare_wheel_metadata, build_wheel: can assume build-system.requires and get_build_wheel_requires are available

+1 from me

Rationale: (a) conceptually the sdist and wheel phases are totally separate, so we shouldn't couple them by requiring a single hook to provide the union of requirements for both. (b) there are known cases where building an sdist has different requirements than building a wheel. Examples: packages that run cython at sdist generation time.

Another example: Thomas expects flit to require VCS interaction support for sdist generation, but not for wheel building (or wheel build file preparation).

It's not immediately obvious to me how prepare_build_files would fit into this; if it's only supposed to be used for building from an sdist, then it's like an extra half-phase in between the sdist and wheel phases -- maybe it's more part of the wheel phase and should get the wheel requirements? Is that the only time it's used? I guess I'll wait to worry about it until after I see how people respond to my argument in the other thread that prepare_build_files shouldn't exist at all :-).

As with get_build_requires, this was less ambiguous when "build_wheel" was the only build hook. Now that we have "build_sdist" as well, then it may make more sense to rename it to "prepare_wheel_input_files". Something else that I believe the PEP currently leaves implicit is the assumptions that backend hook implementations are allowed to make regarding the current working directory when they're invoked. As I understand it, there are three defined possibilities: - original source tree (potentially VCS metadata, no PKG-INFO file) - unpacked sdist (no VCS metadata, PKG-INFO file) - prepared wheel input files (no VCS metadata, maybe PKG-INFO file) Given those options, the hooks that can be called given a particular kind of input directory are: * original source tree: * all hooks * unpacked sdist: * all hooks * prepared wheel input files (if `prepare_wheel_input_files` is defined): * prepare_wheel_metadata * build_wheel * NOT get_build_wheel_requires (see below) My rationale for requiring get_build_wheel_requires to be called in the source directory is that it means that `prepare_wheel_input_files` can rely on those dynamic dependencies, appropriately reflecting it's status as an optional substep of the `build wheel` process. The available dependencies for each hook are then: * build-system.requires only: * get_build_sdist_requires * get_build_wheel_requires * build-system.requires + get_build_sdist_requires: * build_sdist * build-system.requires + get_build_wheel_requires: * prepare_wheel_input_files * prepare_wheel_metadata * build_wheel Make sense? If folks agree with that, we could make the above explicit in the PEP using a pair of dedicated paragraphs in each hook description: * One starting "Current working directory: ..." * One starting "Available dependencies: ..." Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Jun 16, 2017 at 2:08 AM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:

On Fri, Jun 16, 2017, at 08:41 AM, Nick Coghlan wrote:

...

...

I think we should rename get_build_requires to get_build_wheel_requires, and add a get_build_sdist_requires. And the rule would be:

get_build_sdist_requires: can assume build-system.requires are available get_build_wheel_requires: can assume build-system.requires are available

build_sdist: can assume build-system.requires and get_build_sdist_requires are available prepare_wheel_metadata, build_wheel: can assume build-system.requires and get_build_wheel_requires are available

+1 from me

*Sigh*, another hook. It makes sense in context, but I can't shake the feeling that what was a relatively simple spec is steadily turning into a complex monster. I still resent that we're trying to standardise an interface to build sdists at the same time as one to build wheels.

Yeah. Well, except I'm not *too* bothered by the sdist part in particular, since the spec already had to define what an sdist was. And if you zoom out then we have get_build_{wheel,sdist}_requires which have identical interfaces, and build_{wheel,sdist} which also have identical interfaces, so the total complexity for this part isn't too high at all. The messy complications come from prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't surprising, since those are the two hooks where we're squinting into our crystal ball to try and guess what will be useful for software that doesn't exist yet, but will later, maybe, we hope. Hmm, here's another plea for simplicity, but from a slightly different direction that I just thought of: what if we said that any hooks starting with "ext_pip_..." are reserved for pip's use, and pip can make up whatever semantics it likes for them. And then as the parts of pip that actually want to use prepare_wheel_metadata and/or get_prepared_wheel_input_files come online, we use the ext_pip_* versions of those hooks to prototype them and work out any issues. And then once there's an actual implementation and proven value, we write a new PEP to drop the ext_pip_ prefix and make them standard. What do you think? -n -- Nathaniel J. Smith -- https://vorpus.org

On Jun 16, 2017, at 6:05 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:

On Fri, Jun 16, 2017, at 10:48 PM, Nathaniel Smith wrote:

...

The messy complications come from prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't surprising, since those are the two hooks where we're squinting into our crystal ball to try and guess what will be useful for software that doesn't exist yet, but will later, maybe, we hope.

I'm not exactly clear on what use cases the prepare_wheel_metadata hook satisfies - that was in the spec before I was particularly involved in it.

Basically it exists because when we’re resolving dependencies to decide what set of projects/versions to install, we need a way to take a sdist (because a wheel might not exist!) and decide what dependencies it needs. We ultimately might end up throwing this away because maybe it ends up conflicting and we can’t actually utilize this particular version at all. So we might end up cycling through 10 different versions of the same project looking for a version whose dependencies don’t conflict with the rest of what we’re trying to install. Without prepare_wheel_metadata our only real option is to just build the wheel and inspect that. For a tool like flit that is probably fine, because since it’s pure Python only building a wheel is going to be exceedingly quick. However something like Scikit or something where building the wheel might take tens of minutes or longer that can degrade things very quickly. If you imagine something that takes even 5 minutes to go from sdist to wheel, if we need to do that 10 times while looking for a version that we can use, that means resolution ends up taking over an hour just because of that package alone. This cost is tempered somewhat by the fact that in the ideal case we can cache those wheels so in the future resolution can be extremely quick, however not everyone can or will run with caching enabled and even then, ``pip install …`` taking an hour even just the first time is still a pretty horrible user experience. The goal of prepare_wheel_metadata then is basically allowing us to ask a sdist “what dependencies would you have, if we were to build you as a wheel” without having to go through the entire build process. This will ideally be much much faster. It’s not *just* dependencies either, in many cases we don’t know the name or version of something because we’re just given a fairly generic tarball (for instance `master.zip` from Github). We need a way to get name/version from that tarball, which would exist in the wheel but we end up falling into the same “but what if things take forever to build” problem. Another possible solution is to go down the path of trying to make a sdist 2.0 that has this metadata in a static format so we can promise that we can get it without needing to build anything. However not only is that a significant chunk of extra work, but some folks (Nathaniel I think?) has indicated that some projects simply can’t determine their dependencies statically at sdist build time, because it’s going to change at build time (I think the example was something can build against Numpy >=X, but once built against X+5, it has to have Numpy >=X.5 at runtime).

I do think we've hashed out a concrete need for prepare_build_files (or whatever it ends up called): to copy the files to a build directory without either copying large amounts of unnecessary data or figuring out what belongs in an sdist. The only alternative would be to require backends to avoid leaving clutter in the working directory, so they would take care of any necessary copy step rather than exposing a separate hook. People insist that trusting backends is a non-starter, though.

The other solution of course is to just say that all backends needs to be able to no-op copy a sdist from an existing sdist. So we have three options really: 1) Require backends to be able to no-op copy a sdist from an existing unpacked sdist. However Thomas is against this, as it would make flit’s job harder. 2) Require frontends to trust that backends are going to DTRT with regards to in-place builds and isolation. However myself and the other pip devs were against this, as we feel it is important for pip to do it’s job. 3) Add a hook that let’s the backends copy the files it needs into a staging area, without the need to prepare a full sdist. This is basically (1) except with some compromises to make the use cases that Thomas says makes flit’s job harder easier to deal with. Nathaniel is against (3) for simplicity sake, and personally I would prefer (1) because I think it is simpler and because I think that it shouldn’t be *too* much additional effort for a backend to make a no-op build_sdist in the case they’re being made out of something that is already a sdist. That being said, I am OK with (3) since Thomas believes that is better for flit than (1) and I don’t have any evidence to refute or back up that claim personally. — Donald Stufft

On Jun 16, 2017, at 5:48 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Fri, Jun 16, 2017 at 2:08 AM, Thomas Kluyver <thomas@kluyver.me.uk <mailto:thomas@kluyver.me.uk>> wrote:

...

On Fri, Jun 16, 2017, at 08:41 AM, Nick Coghlan wrote:

...

...

I think we should rename get_build_requires to get_build_wheel_requires, and add a get_build_sdist_requires. And the rule would be:

get_build_sdist_requires: can assume build-system.requires are available get_build_wheel_requires: can assume build-system.requires are available

build_sdist: can assume build-system.requires and get_build_sdist_requires are available prepare_wheel_metadata, build_wheel: can assume build-system.requires and get_build_wheel_requires are available

+1 from me

*Sigh*, another hook. It makes sense in context, but I can't shake the feeling that what was a relatively simple spec is steadily turning into a complex monster. I still resent that we're trying to standardise an interface to build sdists at the same time as one to build wheels.

Hmm, here's another plea for simplicity, but from a slightly different direction that I just thought of: what if we said that any hooks starting with "ext_pip_..." are reserved for pip's use, and pip can make up whatever semantics it likes for them. And then as the parts of pip that actually want to use prepare_wheel_metadata and/or get_prepared_wheel_input_files come online, we use the ext_pip_* versions of those hooks to prototype them and work out any issues. And then once there's an actual implementation and proven value, we write a new PEP to drop the ext_pip_ prefix and make them standard.

I’d probably want to spec this out as ext_{name-on-pypi}_* to remove a special case on pip in the PEP, to let others do experimentation as well. However this solution is fine with me on both of the Non build_{sdist,wheel} hooks. Although I feel like it kind of defeats the purpose of the prepare_build_files hooks, because I think Thomas’ goal is to not have to support build_sdist from within another sdist, and if he depends on a pip extension (rather than it being an optional speed up like prepare_wheel_metadata) then we either break flit users if we decide it wasn’t worth it and we remove it, or we force other frontends to understand ext_pip_prepare_build_files and we get into the same kind of “everyone is just trying to emulate setuptools” bog that we’re trying to escape. — Donald Stufft

On Jun 16, 2017, at 11:08 PM, Nathaniel Smith <njs@pobox.com> wrote:

(because the legacy setup.py command errors out, because the build_sdist hook is missing, because the build_sdist hook errors out...); probably falling back on shutil.copytree.

Ah okay. This is where the disconnect was. I assumed if a project was using a PEP 517 build tool the result of a build_sdist failure (or a build_wheel failure) would be surfacing an error to the end user, not some sort of fallback.

On 17 June 2017 at 08:07, Nathaniel Smith <njs@pobox.com> wrote:

Proposal: "NotImplemented" is a legal return value from build_sdist (similar to dunder methods), and should trigger whatever fallback behavior the frontend would do if the hook was simply undefined.

The PEP currently allows build_sdist to fail, but frontends don't have much option do do anything other than present that failure direct to the user. Not because there isn't a special "fall back to something else" return value, but because the PEP provides no guarantee that any fallback is possible. We've been very careful to *not* require anything of source trees than what the hooks offer, as I understand it, so it seems like we're doing a bit of a U-turn now if we say "if the hook fails, front ends will fall back" and don't provide any information as to what front ends are allowed to assume as part of that fallback. You mention falling back to copying, but the PEP doesn't even specify that source trees must be copyable. I'd probably have to work quite hard to come up with a scenario where they aren't, but if we want to make that guarantee, let's just be explicit about it. A single sentence "backend operations on source trees must behave the same if the source tree is copied to a different filesystem location, and front ends are free to copy source trees if needed" is easy to add, if that's what we want to guarantee. Personally, I prefer not adding constraints on source trees, and leaving the PEP as it is, with no fallback required or expected if a hook fails. Paul

I have prepared a PR against the PEP adding get_build_sdist_requires , and renaming a couple of the other hooks for clarity (get_build_wheel_requires, prepare_build_wheel_files): https://github.com/python/peps/pull/297

On Sat, Jun 24, 2017 at 7:57 AM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:

Nick has merged that PR, and the updated PEP is visible here:

https://www.python.org/dev/peps/pep-0517/

Hopefully we're nearing a consensus on this now. If you're interested, please do have a read through the latest version.

Reading through it just now, there were some bits that gave me the impression that prepare_build_wheel_files must *always* be called before calling build_wheel (e.g. "Because the wheel will be built from a temporary build directory, build_wheel may create intermediate files in the working directory, and does not need to take care to clean them up."). If we end up keeping prepare_build_wheel_files, then I think we should add some text making clear exactly which paths are legal. ...and in the current spec this is complex enough that something like a state machine might be the clearest way? Here's an attempt at an NFA representation. Notation: [foo] is a state, --bar--> is a labeled transition. Required transitions: [VCS checkout] --build_wheel--> [wheel] Required transitions if you want to have a useful sdist at all: [VCS checkout] --build_sdist+unpack--> [non-VCS tree] [non-VCS tree] --build_wheel--> [wheel] Required if you have a prepare_wheel_metadata hook: [VCS checkout] --prepare_wheel_metadata--> [checkout+metadata] [checkout+metadata] --build_wheel(..., metadata_directory=...)--> [wheel] [non-VCS tree] --prepare_wheel_metadata--> [non-VCS tree+metadata] [non-VCS tree+metadata] --build_wheel(..., metadata_directory=...)--> [wheel] Plus you should support EITHER: [non-VCS tree] --build_sdist+unpack--> [non-VCS tree] OR: [VCS checkout] --prepare_wheel_build_files--> [non-VCS tree] [non-VCS tree] --prepare_wheel_build_files--> [non-VCS tree] ...I'm not sure this is really making things clearer. (Though a graphviz version would be less terrible than the text version, maybe?) -n -- Nathaniel J. Smith -- https://vorpus.org

On Sun, Jun 25, 2017, at 08:41 AM, Nick Coghlan wrote:

Aside from not producing an archive, the bit that makes `prepare_wheel_build_files` notably different from `build_sdist` is that it relies on `get_build_wheel_requires` *not* on `get_build_sdist_requires`: https://www.python.org/dev/peps/pep-0517/#build-environment

This reminds me: while implementing a wrapper to call PEP 517 hooks, a problem occurred to me. I have implemented prepare_wheel_build_files with a fallback to making an sdist if the hook is not defined: https://github.com/takluyver/pep517/blob/ee43a9334c377d7c37badcc8527cb7a8500... But in this case, the build_sdist hook will be invoked in an environment with the build-wheel-deps installed, not the build-sdist-deps. Working around this would require moving the fallback a couple of levels up the stack to a component that knows about installing packages for the build process. That's not impossible, but it's inelegant and less efficient. Thomas

On 25 June 2017 at 16:58, Nathaniel Smith <njs@pobox.com> wrote:

The current spec's solution:

- Let's define an exhaustive list of all the reasons you might want to make an sdist: 1) you're a hypothetical future version of pip that has decided to implement some specific build-from-unpacked-source-tree semantics 2) the user has specifically requested an sdist, so if one can't be made then the only thing to do is to fail and pass on some unstructured error log 3) that's it, we've made a list of all possible front-end semantics. - Then define a hook that does (1) and a hook that does (2).

pip doesn't care about making sdists, it only cares about doing out of tree wheel builds. It's other tools (e.g. tox) that specifically care about making sdists. So we only have two front-end scanarios that we need to handle: 1. preparing for an out-of-tree call to build_wheel 2. actually building an sdist (e.g. for testing or publication) The possibility of only defining one hook arises from the fact that providing just the build_sdist hook would be sufficient for both tasks *if* we were willing to impose the constraint that everything a backend depends on to build an sdist will always also be a pre-requisite for building a wheel file. Thomas has indicated that that *isn't* the case for flit - he expects the `build_sdist` hook to need runtime dependencies (either Python level or external) that `build_wheel` doesn't. Thus `prepare_build_wheel_files` to cover the case where the frontend doesn't really want an sdist at all, it just wants to do an out-of-tree wheel build, but the backend doesn't want to ensure that the preconditions for `build_sdist` and `build_wheel` are identical. However, we've chosen to make the second hook optional, so backends that have a very simple `build_sdist` implementation don't have to worry about the optional wheel preparation hook. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 June 2017 at 17:41, Nathaniel Smith <njs@pobox.com> wrote:

Maybe you're right and there are exactly 2 front-end use cases and it will turn out that the current PEP addresses them perfectly. I don't have a crystal ball; I'm making an argument from ignorance.

I'm not - we have two concrete potential consumers of the interface (pip and tox, aka "build to use" and "build to test"), and I'm designing the interface to cover their needs (i.e. out-of-tree wheel builds and actual sdists). If we discover other use cases later, we'll worry about them then (and the easy of doing so is the nicest benefit of defining this as a Python API), but the temptation to design in hyper-flexibility now falls under YAGNI (You Ain' Gonna Need It). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

My impressions on reading what is hopefully the current version of the PEP Are there different paths to get to sdist/wheel for example tree -> prepare sdist/wheel files -> build sdist and tree -> build sdist, tree -> build wheel depending on what [pip] decides to do? Should the frontend do both and compare the result to make sure it is the same? I'm not entirely clear on what the prepare hooks should do. The rationalizations interleaved between the descriptions of each hook distract from what the hooks should actually do. Suggested replacement rationalization: "Some people like to build [format]. This hook builds [format]". I will make an effort to implement this for enscons, probably with the most direct tree -> wheel, tree -> sdist hooks and without any optional hooks. Thanks, Daniel On Mon, Jun 26, 2017 at 8:36 AM Nathaniel Smith <njs@pobox.com> wrote:

On Jun 25, 2017 12:46 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:

On 25 June 2017 at 17:41, Nathaniel Smith <njs@pobox.com> wrote:

...

Maybe you're right and there are exactly 2 front-end use cases and it will turn out that the current PEP addresses them perfectly. I don't have a crystal ball; I'm making an argument from ignorance.

I'm not - we have two concrete potential consumers of the interface (pip and tox, aka "build to use" and "build to test"), and I'm designing the interface to cover their needs (i.e. out-of-tree wheel builds and actual sdists).

If we discover other use cases later, we'll worry about them then (and the easy of doing so is the nicest benefit of defining this as a Python API), but the temptation to design in hyper-flexibility now falls under YAGNI (You Ain' Gonna Need It).

My proposal also covers their needs AFAICT? At least I thought Donald said he thought the would work for pip. And you can't use YAGNI to argue for a more complicated proposal, that's cheating :-).

-n

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 26 June 2017 at 22:36, Nathaniel Smith <njs@pobox.com> wrote:

On Jun 25, 2017 12:46 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:

On 25 June 2017 at 17:41, Nathaniel Smith <njs@pobox.com> wrote:

...

Maybe you're right and there are exactly 2 front-end use cases and it will turn out that the current PEP addresses them perfectly. I don't have a crystal ball; I'm making an argument from ignorance.

I'm not - we have two concrete potential consumers of the interface (pip and tox, aka "build to use" and "build to test"), and I'm designing the interface to cover their needs (i.e. out-of-tree wheel builds and actual sdists).

If we discover other use cases later, we'll worry about them then (and the easy of doing so is the nicest benefit of defining this as a Python API), but the temptation to design in hyper-flexibility now falls under YAGNI (You Ain' Gonna Need It).

My proposal also covers their needs AFAICT?

No, as you don't know in your proposal whether or not build_sdist can fail until after you've already called it, and you always need to call both `get_build_sdist_requires()` and `build_sdist()`, even if you only care about doing an out-of-tree wheel build. That categorically rules out two-pass introspection based implementations that first interrogate the backend to find out which hooks it supports, and then use that to decide their execution strategy. Tangent: it turns out the word order difference between `get_build_wheel_requires` and `prepare_wheel_build_files` hurts my brain, so I've changed the latter to `prepare_wheel_input_files` below. That keeps the common prefix with `prepare_wheel_metadata`, while avoiding the confusing word order reversal relative to `build_wheel`. That is, the current PEP allows a frontend to do the following when given an arbitrary source tree rather than an sdist archive: 1. Look for `prepare_wheel_input_files()` on the backend. If you find it: - call `get_build_wheel_requires()` - install the additional dependencies - call `prepare_wheel_input_files()` - call `build_wheel()` 2. If you don't find the input preparation hook: - call `get_build_sdist_requires()` - call `get_build_wheel_requires()` - install both sets of additional dependencies - call `build_sdist()` - unpack the sdist into the build directory - call `build_wheel()` Failure of any hook in any way (whether that's raising an exception or returning something that isn't in line with the requirements of PEP 517) indicates a failed build. To put it another way, the key reason we can be confident that PEP 517 is comprehensive as currently drafted is that we only have two target artifact types (sdist & wheel), one source archiving strategy (in-tree), and two binary build strategies (in-tree or out-of-tree). That means source archiving is straightforward: - call `get_build_sdist_requires()` - install the additional dependencies - call `build_sdist()` In-tree binary builds are also straightforward: - call `get_build_wheel_requires()` - install the additional dependencies - call `build_wheel()` Out-of-tree binary builds via sdist combine the two: - call `get_build_sdist_requires()` - call `get_build_wheel_requires()` - install both sets of additional dependencies - call `build_sdist()` - unpack the sdist into the build directory - call `build_wheel()` (Frontends would also be free to use two distinct environments, one to create the sdist, and then a separate one to build the wheel) However, that last strategy has an undesirable property: it means that installing the sdist archiving dependencies becomes a constraint on out-of-tree builds, which is annoying for both frontends and backends. Adding direct support for out-of-tree builds resolves that problem. Out-of-tree binary builds via a dedicated hook: - call `get_build_wheel_requires()` - install the additional dependencies - call `prepare_wheel_input_files()` - call `build_wheel()` By contrast, if we only define a "try it and see if it works" approach for build_sdist, and provide no other way to request an out-of-tree build, then frontends *have* to do the following: - call `get_build_sdist_requires()` - call `get_build_wheel_requires()` - install both sets of additional dependencies - call `build_sdist()` - if it returns `NotImplemented`... umm, give up, same as if it failed? Fall back to the bad "copy everything" default that pip is trying to get away from? - unpack the sdist into the build directory - call `build_wheel()` Without a dedicated out-of-tree build preparation hook, there's no way for frontends to know in advance that build_sdist might fail, there's no way for them to avoid installing the sdist archiving dependencies when doing an out-of-tree build, and there's no way for backends to provide a fallback build directory preparation strategy that relies solely on the wheel building dependencies. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 27 June 2017 at 18:54, Nathaniel Smith <njs@pobox.com> wrote:

On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

2. If you don't find the input preparation hook:

- call `get_build_sdist_requires()` - call `get_build_wheel_requires()` - install both sets of additional dependencies - call `build_sdist()` - unpack the sdist into the build directory - call `build_wheel()`

Side note: I think this is wrong -- I think you can't call get_build_wheel_requires until after you've unpacked the sdist, because that's a whole new tree that might be arbitrarily different. (In practice I guess this should never happen, but we should be clear conceptually I think.)

No, both `get_build_wheel_requires()` and `build_wheel()` also have to work in-place - backends aren't allowed to assume that *all* builds will be out-of-tree, even though pip specifically is expected to work that way. Neither of these hooks is allowed to give different results depending on whether you're doing an in-place or out-of-tree build. To put it another way, there are two build paths that backends are required to support: - in-place, direct from the original source tree - out-of-tree, from an unpacked sdist There's also a third, optional, build path for backends that define `prepare_wheel_input_files`: - out-of-tree, from a prepared wheel input directory The result wheel should be "the same" regardless of how the build is executed. In practice, we expect there will be some variation, just because reproducible builds are hard. However, the output of the different paths should be at least as similar as any two builds of the same source tree run at different times with the same system configuration. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 June 2017 at 13:43, Nathaniel Smith <njs@pobox.com> wrote:

On Tue, Jun 27, 2017 at 4:20 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

On 27 June 2017 at 18:54, Nathaniel Smith <njs@pobox.com> wrote:

...

On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

2. If you don't find the input preparation hook:

- call `get_build_sdist_requires()` - call `get_build_wheel_requires()` - install both sets of additional dependencies - call `build_sdist()` - unpack the sdist into the build directory - call `build_wheel()`

Side note: I think this is wrong -- I think you can't call get_build_wheel_requires until after you've unpacked the sdist, because that's a whole new tree that might be arbitrarily different. (In practice I guess this should never happen, but we should be clear conceptually I think.)

No, both `get_build_wheel_requires()` and `build_wheel()` also have to work in-place - backends aren't allowed to assume that *all* builds will be out-of-tree, even though pip specifically is expected to work that way. Neither of these hooks is allowed to give different results depending on whether you're doing an in-place or out-of-tree build.

Right. But if you're building and sdist and unpacking it and then calling build_wheel there, then that's not an out-of-tree build, it's an in-tree build in a new tree. The wheel hooks are certainly allowed to give different results in a VCS-tree and an unpacked-sdist-tree, they have different pyproject.toml files!

Umm, no. The pyproject.toml file MUST NOT change when building the sdist. It's a human-managed input file - if any step of the build process mutates it, that step is *broken*. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 June 2017 at 14:25, Nathaniel Smith <njs@pobox.com> wrote:

pyproject.toml is just an example anyway; get_build_wheel_requires runs arbitrary code, and that code could potentially return different results in different trees.

No, I don't want to give backends permission to work that way. If a frontend runs get_build_wheel_requires() against the VCS tree, then that should also be sufficient to set up an environment to build an unpacked sdist - the backend isn't permitted to require that both hooks be run against the exact same tree. (It's fine if it returns some extra dependencies that may not technically be needed for a from-sdist build, though) To put it another way, if the "VCS-tree-for-build-requirements, unpacked-sdist-tree for build" combination doesn't work, that's a bug in the backend, not a bug in the frontend. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Jun 27, 2017 at 9:29 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

Sorry, I also meant to reply to this part before hitting send.

On 28 June 2017 at 13:43, Nathaniel Smith <njs@pobox.com> wrote:

...

I would say that there's only one required path, which is in-place. (But the tree that they do the in-place build in might have a more or less complicated history). It worries me that you seem to think that build_sdist and build_wheel should be coupled like this... this proliferation of cases and the reification of "out of tree builds" as a special thing that's different from an in-tree build in a temporary tree is already what worried me about prepare_wheel_input_files, and now it seems to be spreading :-).

I don't know what additional coupling you're seeing: the only coupling is that building a wheel directly from a VCS source tree and first exporting an sdist and then building a wheel from that *must give the same result* (modulo any problems related to reproducible builds, or the lack thereof). If a project can't meet that expectation, then their sdist generation is broken, since it clearly isn't exporting the necessary content to actually build the project properly.

If you weren't aware of that inherent coupling, and are surprised by the fact that it's a constraint on publishing useful sdists, then I'd consider it a good thing that PEP 517 is now making it more explicit.

What I'm concerned about is that in your formulation, you're running different code to build a wheel from an sdist that we just generated, versus an sdist that we downloaded from PyPI. By making it possible to distinguish these cases, you're facilitating projects that break that inherent coupling (either accidentally or on purpose). I think that in these two cases, all the hooks used in the wheel-building phase should be run in exactly the same way, i.e., inside the unpacked sdist, with no reference to the original working tree that that sdist was generated from. Since the PyPI case can't access the original working tree, the local out-of-place-build case generate the sdist and then after that pretend it can't access the original working tree either. -n -- Nathaniel J. Smith -- https://vorpus.org

On 28 June 2017 at 05:29, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 28 June 2017 at 13:43, Nathaniel Smith <njs@pobox.com> wrote:

...

I would say that there's only one required path, which is in-place. (But the tree that they do the in-place build in might have a more or less complicated history). It worries me that you seem to think that build_sdist and build_wheel should be coupled like this... this proliferation of cases and the reification of "out of tree builds" as a special thing that's different from an in-tree build in a temporary tree is already what worried me about prepare_wheel_input_files, and now it seems to be spreading :-).

I don't know what additional coupling you're seeing: the only coupling is that building a wheel directly from a VCS source tree and first exporting an sdist and then building a wheel from that *must give the same result* (modulo any problems related to reproducible builds, or the lack thereof). If a project can't meet that expectation, then their sdist generation is broken, since it clearly isn't exporting the necessary content to actually build the project properly.

If you weren't aware of that inherent coupling, and are surprised by the fact that it's a constraint on publishing useful sdists, then I'd consider it a good thing that PEP 517 is now making it more explicit.

Note that the whole concept of "out of tree builds" is not something theoretical - it comes directly from pip's requirement to be able to do clean builds. So saying that tree->sdist->unpacked sdist is "just another in-place build of a different tree" is missing the point, which is that we need a mechanism to produce an effectively identical, in the sense that it must result in the same wheel, copy of a tree - and because we chose the sdist as the route for that, there are constraints on the contents of a sdist. I view those constraints as pretty much self evident, as I would be extremely surprised if I had a source tree of *any* form for a project, and I build a sdist from it and gave it to a friend, and when we both built wheels and installed them, we had different installed copies of the project. The alternative would be to have a separate "produce a copy of this tree that builds the same wheel" hook. And I thought we already went round that loop, which is how we ended up deciding that the sdist was our approach... Paul

Is there a prototype implementation of pep 517 yet? On Wed, Jun 28, 2017, 06:58 Robert Collins <robertc@robertcollins.net> wrote:

Re: returning magic strings to indicate exceptions. Please no. Just no. Have any pep build host add a small library to Python path with any symbols we want to define. Hardly an implementation hurdle.

Rob _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

I was able to implement PEP 517 build_wheel and build_sdist for enscons (on bitbucket), and a click cli calling any backend mentioned in pyproject.toml. Pretty simple. Not sure what to do with the config dictionary. SCons is not designed to be called twice in the same process with different arguments. It would be easier for me to know that pip would only invoke one PEP 517 function per subprocess, or to know all target directories in advance. Otherwise the enscons.api subprocess has to invoke SCons in another subprocess. SCons also builds all targets (wheel and sdist in same invocation) by default. "Alakazam!" On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver <thomas@kluyver.me.uk> wrote:

On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote:

Is there a prototype implementation of pep 517 yet?

- Flit has a PR with a prototype backend implementation, though it's not up to date with all the changes the PEP has undergone. I'll update it when we've agreed on a spec - it's still a fast moving target right now. - I have a prototype module frontends could use to call hooks here: https://github.com/takluyver/pep517 . It's mostly up to date, except for the issue with using sdist as a fallback for copying files for a wheel.

Re: magic strings - like Nathaniel said, I haven't noticed them as part of any proposal so far.

Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

That would fit with setup.py, which also likes to sys.exit() after one command. On Wed, Jun 28, 2017, 22:25 Nick Coghlan <ncoghlan@gmail.com> wrote:

...

I was able to implement PEP 517 build_wheel and build_sdist for enscons (on bitbucket), and a click cli calling any backend mentioned in

On 29 June 2017 at 11:42, Daniel Holth <dholth@gmail.com> wrote: pyproject.toml.

...

Pretty simple. Not sure what to do with the config dictionary.

That's designed to allow frontends to tunnel custom settings from the user through to the backend, so if you don't support any config settings yet, it probably makes sense to just error out with "Unknown config setting" if the dictionary is non-empty. Alternatively, you could silently ignore it.

...

SCons is not designed to be called twice in the same process with different arguments. It would be easier for me to know that pip would only invoke one PEP 517 function per subprocess, or to know all target directories in advance. Otherwise the enscons.api subprocess has to invoke SCons in another subprocess. SCons also builds all targets (wheel and sdist in same invocation) by default.

That's an interesting point of tension between supporting imperative frontends like pip (which have a strong opinion about the order in which steps should be executed) and declarative build systems like Scons (which are more "produce the defined artifact set").

However, I think the way to go for now would be to say:

- each hook call should be made in a fresh subprocess (so backends don't need to use a second subprocess "just to be on the safe side") - in a *future* API extension, we may add an optional "build_all" hook (whereby the backend produced both an sdist and all wheels it knew how to create for the current platform), but anything like that will be out of scope for the initial PEP 517 API

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Daniel, Perhaps the better way to go about it than to call scons N times is to have N variant dirs? where there's a variant dir for each build type you want, and then youd call scons build_wheel which is aliased Alias('build_wheel','variant_dir for building wheel path') Maybe bring this up on scons users mailing list and we'll see if it can be resolved? On Wed, Jun 28, 2017 at 6:42 PM, Daniel Holth <dholth@gmail.com> wrote:

I was able to implement PEP 517 build_wheel and build_sdist for enscons (on bitbucket), and a click cli calling any backend mentioned in pyproject.toml. Pretty simple. Not sure what to do with the config dictionary.

SCons is not designed to be called twice in the same process with different arguments. It would be easier for me to know that pip would only invoke one PEP 517 function per subprocess, or to know all target directories in advance. Otherwise the enscons.api subprocess has to invoke SCons in another subprocess. SCons also builds all targets (wheel and sdist in same invocation) by default.

"Alakazam!"

On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver <thomas@kluyver.me.uk> wrote:

...

On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote:

Is there a prototype implementation of pep 517 yet?

- Flit has a PR with a prototype backend implementation, though it's not up to date with all the changes the PEP has undergone. I'll update it when we've agreed on a spec - it's still a fast moving target right now. - I have a prototype module frontends could use to call hooks here: https://github.com/takluyver/pep517 . It's mostly up to date, except for the issue with using sdist as a fallback for copying files for a wheel.

Re: magic strings - like Nathaniel said, I haven't noticed them as part of any proposal so far.

Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

I don't think anyone has proposed anything involving magic strings? On Jun 28, 2017 3:58 AM, "Robert Collins" <robertc@robertcollins.net> wrote:

Re: returning magic strings to indicate exceptions. Please no. Just no. Have any pep build host add a small library to Python path with any symbols we want to define. Hardly an implementation hurdle.

Rob

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Fri, Jun 16, 2017 at 7:48 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 17 June 2017 at 07:48, Nathaniel Smith <njs@pobox.com> wrote:

...

Hmm, here's another plea for simplicity, but from a slightly different direction that I just thought of: what if we said that any hooks starting with "ext_pip_..." are reserved for pip's use, and pip can make up whatever semantics it likes for them. And then as the parts of pip that actually want to use prepare_wheel_metadata and/or get_prepared_wheel_input_files come online, we use the ext_pip_* versions of those hooks to prototype them and work out any issues. And then once there's an actual implementation and proven value, we write a new PEP to drop the ext_pip_ prefix and make them standard.

What do you think?

pip's needs here aren't unique to pip - they're just "system integrator" needs, rather than "software developer" needs.

It's fairly common for system integrator needs to seem like pointless complexity to folks more focused on addressing the "single component publisher (aka software developer)" case, but that's part of why we have the PEP process: to encourage us to figure out the best place for complexity to live in order to advance the overall Python ecosystem, rather than trying to convince ourselves that the underlying complexity doesn't exist.

I'm not making an argument that the complexity doesn't exist or that pip is the only project that matters; I'm arguing that it's risky to standardize things before we've really hit the problem, because we might not fully understand it yet. Which is a truism :-). But sometimes you don't have much choice. The real suggestion is hey, maybe in this case there's a way to get these important-but-not-fully-understood issues off the critical path for PEP 517 without losing our chance to handle them later. -n -- Nathaniel J. Smith -- https://vorpus.org