I'm still not convinced, but this PEP has been WIP for too long. I think it should be marked Accepted. On Aug 17, 2017 9:51 PM, "Nick Coghlan" wrote:

On 18 August 2017 at 02:05, 12345 67890 wrote:

...

After reviewing the PEP, I personally feel that it relies too much on the backend Doing The Right Thing. As it currently stands, it is my understanding that the build backend is called in the source directory and is then responsible for handling the entire process to build a compliant wheel. In a PEP 517 world, the build backends may be poorly written initially and may not fully comply with the PEP’s goal to produce the same wheel that would be produced from an sdist.

Current process:

Frontend calls backend in source directory build-wheel

The PEP *allows* this, but it doesn't *require* it.

...

Proposed process:

Frontend copies source tree to temporary directory Frontend invokes build-sdist to build an sdist Frontend extracts sdist to new temporary directory Frontend reloads backend from sdist directory and invokes build-wheel

A particularly paranoid front-end remains free to work this way if it wants to.

From the PEP's point of view, it's just four distinct operations:

Step 0: make a copy of the original source tree Step 1: build an sdist from the copied source tree Step 2: unpack the sdist Step 3: build a wheel from the unpacked sdist

...

The proposed process is more computationally intensive, but moves compliance logic out of the build backend. In addition, we can make some modifications based on this proposed process:

* Remove parameter for build_dir in build_wheel since it is always the current directory

We've had that discussion - build_dir models something different from what you propose (specifically, it's designed to invoke the backend's native out-of-tree build support, when that's what the frontend actually wants to invoke. If a frontend doesn't want that, don't use the setting)

...

Optimization: add new parameter to build_sdist called copy_extra, that when set to false, only includes files in sdist needed to build a wheel.

No, this is the operation we decided makes no sense (and hence removed from the latest iteration of the PEP), because no build system we could find actually provides it natively.

That means it would necessarily be a Python-specific capability anyway, and that further means that any backend which could be trusted to implement this feature correctly can instead be trusted to implement building an sdist or wheel directly from a source tree correctly.

It's also not a major concern if early backend iterations *don't* do things correctly - those errors will be reported as bugs, fixed versions of the backends will be released, and future builds will start doing the right thing.

...

Specify that build_wheel should fail if PKG-INFO is not present so that it is always invoked in an sdist

No, we don't want to make that a mandatory requirement, since frontends are already free to enforce that rule if they want to (by checking for PKG-INFO before they call build_wheel), and particular backends are also free to enforce it for the packages they manage (by raising NotImplemented when they don't find PKG-INFO).

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:

The other aspect I'm not clear on is whether or not both PEP authors are otherwise happy for the current version to be *considered* for acceptance

Nathaniel has mentioned to me a list of issues he sees that we still need to work through. So I think we're going to be working on this for a while yet. Thomas

I'll let Nathaniel relay them. On Fri, Aug 18, 2017, at 06:23 PM, xoviat wrote:

Thomas:

What are the specific issues that need to be worked out?

Regards,

xoviat

2017-08-18 3:09 GMT-05:00 Thomas Kluyver :

...

On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:

...

The other aspect I'm not clear on is whether or not both PEP authors are otherwise happy for the current version to be *considered* for acceptance

Nathaniel has mentioned to me a list of issues he sees that we still>> need to work through. So I think we're going to be working on this for a>> while yet.

Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

There's prototype machinery to call the hooks here, though it's outdated because the spec keeps changing:https://github.com/takluyver/pep517 There's a prototype backend in a branch of flit, but it's even more outdated:https://github.com/takluyver/flit/tree/toml-config On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote:

Apart from the issues, can we get some prototype implementations?

On Fri, Aug 18, 2017, 13:24 xoviat wrote:

...

Thomas:

What are the specific issues that need to be worked out?

Regards,

xoviat

2017-08-18 3:09 GMT-05:00 Thomas Kluyver :

...

On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:

...

The other aspect I'm not clear on is whether or not both PEP authors are otherwise happy for the current version to be *considered* for acceptance

Nathaniel has mentioned to me a list of issues he sees that we still>>> need to work through. So I think we're going to be working on this for a>>> while yet.

Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

At this point, I can only offer implementations as I have been attempting to do (some of which are outdated because the PEP has changed), but I cannot make anyone pay attention to or accept my pull requests and it seems I cannot accelerate the discussion here. 2017-08-18 14:08 GMT-05:00 xoviat :

I have attempted to implement prototype machinery here https://github.com/pypa/pip/pull/4589 and here https://github.com/pypa/setuptools/pull/1039, but it's been widely ignored IHMO because as Thomas said, the spec keeps changing.

2017-08-18 12:37 GMT-05:00 Thomas Kluyver :

...

There's prototype machinery to call the hooks here, though it's outdated because the spec keeps changing: https://github.com/takluyver/pep517

There's a prototype backend in a branch of flit, but it's even more outdated: https://github.com/takluyver/flit/tree/toml-config

On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote:

Apart from the issues, can we get some prototype implementations?

On Fri, Aug 18, 2017, 13:24 xoviat wrote:

Thomas:

What are the specific issues that need to be worked out?

Regards,

xoviat

2017-08-18 3:09 GMT-05:00 Thomas Kluyver :

On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:

...

The other aspect I'm not clear on is whether or not both PEP authors are otherwise happy for the current version to be *considered* for acceptance

Nathaniel has mentioned to me a list of issues he sees that we still need to work through. So I think we're going to be working on this for a while yet.

Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 19 August 2017 at 05:28, Thomas Kluyver wrote:

We've probably all wished that the discussion could be brought to a swift conclusion. But there are real questions to work out, and people have many other things to pay attention to. I'm frustrated by how long it's taking as well, but there's no magic button anyone can press to make it go quickly.

Technically, commercial redistributors do have a magic button we could press called "ongoing funding for sustaining engineering in the upstream Python ecosystem" (since that kind of funding can also cover needs-driven UX improvements), but alas, whether or not we ever actually press that is conditional on potential customers pressing the "customer demand" button hard enough and often enough to light up the "viable business opportunity" sign :P I'm actually genuine curious to see how those commercial dynamics start changing as the end date for community support of the Python 2.x series gets closer :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Ah the joy of Python 2.7; it seems I've forgotten its perils: unloading is not possible which means that we would need to come up with another solution to this problem. Perhaps setting aside a namespace for the build frontend in the subprocess? 2017-08-19 14:23 GMT-04:00 xoviat :

I assume that the outstanding issues mentioned are related to sys.path in the build tree. My view on that is the following: the build frontend should be responsible using any mechanism that it chooses for invoking the build backend, which must be imported with '' at the front of sys.path. This obviously means that if the build frontend experiences faults before it manages to import and invoke the backend, then it's the build frontend's fault.

The only potential issue remaining that I can think of then is: what about modules that are already imported when the build backend is called? WRT standard library modules, we could simply say that build backends must be prepared to function in any environment where standard library modules are already imported, which I think is a reasonable requirement. But what about imports that aren't standard library modules but are fairly common, like pip imports? The simplest way to simplify the interface is probably to say that all non-standard library modules must be removed from sys.modules (unloaded) before the build backend is called.

What do others think?

2017-08-18 23:41 GMT-04:00 Nick Coghlan :

...

On 19 August 2017 at 05:28, Thomas Kluyver wrote:

...

We've probably all wished that the discussion could be brought to a swift conclusion. But there are real questions to work out, and people have many other things to pay attention to. I'm frustrated by how long it's taking as well, but there's no magic button anyone can press to make it go quickly.

Technically, commercial redistributors do have a magic button we could press called "ongoing funding for sustaining engineering in the upstream Python ecosystem" (since that kind of funding can also cover needs-driven UX improvements), but alas, whether or not we ever actually press that is conditional on potential customers pressing the "customer demand" button hard enough and often enough to light up the "viable business opportunity" sign :P

I'm actually genuine curious to see how those commercial dynamics start changing as the end date for community support of the Python 2.x series gets closer :)

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Yes, it probably is. But then again, I assumed that it was obvious that PYTHONPATH would not be modified before invoking the hook (at least obvious to me) until the 'PEP 517 Status' discussion implied that it was not so obvious and needed to be specified in the PEP. 2017-08-19 16:10 GMT-04:00 Daniel Holth :

It's probably a tiny wrapper running the hook in its own subprocess. Probably few modules loaded.

On Sat, Aug 19, 2017, 14:31 xoviat wrote:

...

Ah the joy of Python 2.7; it seems I've forgotten its perils: unloading is not possible which means that we would need to come up with another solution to this problem. Perhaps setting aside a namespace for the build frontend in the subprocess?

2017-08-19 14:23 GMT-04:00 xoviat :

...

I assume that the outstanding issues mentioned are related to sys.path in the build tree. My view on that is the following: the build frontend should be responsible using any mechanism that it chooses for invoking the build backend, which must be imported with '' at the front of sys.path. This obviously means that if the build frontend experiences faults before it manages to import and invoke the backend, then it's the build frontend's fault.

The only potential issue remaining that I can think of then is: what about modules that are already imported when the build backend is called? WRT standard library modules, we could simply say that build backends must be prepared to function in any environment where standard library modules are already imported, which I think is a reasonable requirement. But what about imports that aren't standard library modules but are fairly common, like pip imports? The simplest way to simplify the interface is probably to say that all non-standard library modules must be removed from sys.modules (unloaded) before the build backend is called.

What do others think?

2017-08-18 23:41 GMT-04:00 Nick Coghlan :

...

On 19 August 2017 at 05:28, Thomas Kluyver wrote:

...

We've probably all wished that the discussion could be brought to a swift conclusion. But there are real questions to work out, and people have many other things to pay attention to. I'm frustrated by how long it's taking as well, but there's no magic button anyone can press to make it go quickly.

Technically, commercial redistributors do have a magic button we could press called "ongoing funding for sustaining engineering in the upstream Python ecosystem" (since that kind of funding can also cover needs-driven UX improvements), but alas, whether or not we ever actually press that is conditional on potential customers pressing the "customer demand" button hard enough and often enough to light up the "viable business opportunity" sign :P

I'm actually genuine curious to see how those commercial dynamics start changing as the end date for community support of the Python 2.x series gets closer :)

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Also, I disagree with this point. I think it's still possible (and in fact preferable) to have setuptools and pip 'test this out' with appropriate fallbacks before opening this up to the wider community. Most people wouldn't even notice because they wouldn't be using HEAD. 2017-08-18 15:28 GMT-04:00 Thomas Kluyver :

I wouldn't expect established tools like pip & setuptools to accept pull requests implementing a PEP which is still under discussion and subject to change. They will naturally want a stable, accepted spec before they consider merging code for it.

We've probably all wished that the discussion could be brought to a swift conclusion. But there are real questions to work out, and people have many other things to pay attention to. I'm frustrated by how long it's taking as well, but there's no magic button anyone can press to make it go quickly.

Thomas

On Fri, Aug 18, 2017, at 08:17 PM, xoviat wrote:

At this point, I can only offer implementations as I have been attempting to do (some of which are outdated because the PEP has changed), but I cannot make anyone pay attention to or accept my pull requests and it seems I cannot accelerate the discussion here.

2017-08-18 14:08 GMT-05:00 xoviat :

I have attempted to implement prototype machinery here https://github.com/pypa/pip/pull/4589 and here https://github.com/pypa/setuptools/pull/1039, but it's been widely ignored IHMO because as Thomas said, the spec keeps changing.

2017-08-18 12:37 GMT-05:00 Thomas Kluyver :

There's prototype machinery to call the hooks here, though it's outdated because the spec keeps changing: https://github.com/takluyver/pep517

There's a prototype backend in a branch of flit, but it's even more outdated: https://github.com/takluyver/flit/tree/toml-config

On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote:

Apart from the issues, can we get some prototype implementations?

On Fri, Aug 18, 2017, 13:24 xoviat wrote:

Thomas:

What are the specific issues that need to be worked out?

Regards,

xoviat

2017-08-18 3:09 GMT-05:00 Thomas Kluyver :

On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote:

...

The other aspect I'm not clear on is whether or not both PEP authors are otherwise happy for the current version to be *considered* for acceptance

Nathaniel has mentioned to me a list of issues he sees that we still need to work through. So I think we're going to be working on this for a while yet.

Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

The setuptools and pip master branches are for things which *are* expected to go into the next version, not for experimental changes

I agree with that statement (the PRs as they currently are need work because they have become stale) in the sense that code in the master branch should work at all times. But that's different than the PEP being fully fleshed out for all build systems rather than just pip and setuptools. And also, a big reason for this is that to be frank, the first implementation will probably not work. That's not just some special rule that I have applied to myself, but it's also the reality with PEP 518. The reality is that PEP 518 does not currently work and it is in the master branch, but people did not discover that fact until it was in the master branch. You can't know what you can't know. If an implementation that does not comply with the PEP yet but works with appropriate fallbacks to verify setuptools support and compliance, then it's not the end of the world if the packages are still built and installed correctly. 2017-08-19 16:48 GMT-04:00 Paul Moore :

On 19 August 2017 at 21:30, xoviat wrote:

...

Also, I disagree with this point. I think it's still possible (and in fact preferable) to have setuptools and pip 'test this out' with appropriate fallbacks before opening this up to the wider community. Most people wouldn't even notice because they wouldn't be using HEAD.

The same people who would notice it are probably just as able to apply a patch/PR or grab a fork. So your PRs are available for people to test if they want to, they don't need to be merged for that to happen.

The setuptools and pip master branches are for things which *are* expected to go into the next version, not for experimental changes (for pip, we should be able to release from master whenever we want to, I suspect setuptools has a similar policy).

Paul