I'm also not going to have much time to keep discussing this over the next few days. On Thu, Jul 20, 2017, at 11:22 AM, Paul Moore wrote:

2) I'd much rather if we're going for a "stop and take stock" style of reset, that we base that off what's currently in the official version of the PEP. Trying to read a new document and identify the differences from the working version is both time consuming and error prone. I don't think there's any reason that any of your questions wouldn't just as reasonably apply to the official version of the PEP.

Agreed. I skimmed through Nathaniel's version. The changes I notice relative to the working version in the peps repo are: 1. Removes the build_directory= parameter 2. Removes the optional hook prepare_metadata_for_build_wheel 3. Adds an extension namespace keyed by the name of the frontend specifying the hooks (e.g. backend.extensions['pip'].some_extra_hook ) Thomas

On 20 July 2017 at 20:55, Nathaniel Smith wrote:

On Thu, Jul 20, 2017 at 3:22 AM, Paul Moore wrote: My impression is that people are going to have to pretty much read from scratch anyway, because the official version is kind of in a messy state right now.

Aside from the need to better specify the semantics of the build_directory parameter, there's nothing messy about the actual specification section, which is what Paul is referring to (since he's one of the folks that reviewed the PRs): https://www.python.org/dev/peps/pep-0517/#build-backend-interface

It's had a number of changes merged recently, it has more changes pending in a PR,

As Donald picked up, what got missed were some of the later examples in the non-normative section, which is the PR that Thomas hadn't had a change to review yet. He's done so now, and I've merged it, so the web version should be self-consistent again shortly.

it's missing features that everyone agrees are needed (NotImplementedError), and it's in a confusing shape because of how it's grown organically over the last ~1.5 years, while my draft is cleaned up into some rational structure.

Thomas also cleaned up the PEP itself, and that's the version he, Paul and I were iteratively updating as the discussion was in progress.

And even if we eventually want to stick with the version in the PEPs repository, then IMO it's probably more useful to look at a minimal thing and ask "what is this missing?" than to look at something with extra stuff added and ask "what can we take away?".

That's what we did that when you first posted your proposed changes, and several of us found it to be both lacking (missing the metadata preparation hook and out-of-tree build support) *and* overly complicated (adding support for arbitrary tool-specific extensions for no clear reason). However, the related discussion did lead to one already captured change (switching explicitly out-of-tree builds from an optional hook to a simple build_directory parameter, which makes them much easier to delegate to third party build systems), as well as one pending change that isn't yet present in either variant: specifically raising NotImplementedError to indicate that an operation isn't supported in the current environment (either because of missing metadata or missing tools). So while your set of questions is good, you're proposing to ask them about the wrong API design at this point, as it's the one currently in PEP 517 that's close to acceptance & implementation, not anything else. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

It looks like we've run out of things to say about PEP 517, except, how soon can we get it into pip? These function signatures will serve us well, significantly lowering the barrier to entry for new pip-compatible build systems. On Thu, Jul 20, 2017 at 3:36 PM Brett Cannon wrote:

On Thu, 20 Jul 2017 at 03:56 Nathaniel Smith wrote:

...

...

On 20 July 2017 at 10:46, Nathaniel Smith wrote:

...

To make this concrete: I'm *pretty* sure (?) that at this point all the basic elements in my "simplified" rewrite are things that we now have consensus are needed in some form, so maybe we can use that as a kind of "minimal core" reference point:

https://github.com/njsmith/peps/commits/517-refactor-streamline/pep-0517.txt

I don't have time this week to look at this, sorry, but a couple of

On Thu, Jul 20, 2017 at 3:22 AM, Paul Moore wrote: points:

...

1) This links to a commit in your repo, not to the actual document.

Ugh, sorry! Pasted the wrong link. This is the correct one:

https://github.com/njsmith/peps/blob/517-refactor-streamline/pep-0517.txt

...

Also, I'd *really* prefer a rendered version if I'm going to review this.

Annoyingly, the PEPs repo uses a .txt extension for .rst files, thus cleverly preventing github from doing anything useful by default...

We will happily accept help in converting plaintext PEPs into reST ones so we can universally change the file extensions to .rst (see https://github.com/python/peps/issues/1) :)

-Brett _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

I'm not worried about it. If you can generate sdist the file format is simple, if you cannot then the build tool asks you for a wheel. Build tools can have dependencies too and could depend on a nifty sdist generating library. It seems to me that the whole spec lets you start by figuring out how to build wheels, and then proceed to implement the rest in order for your project to be lauded by PyPA. Thus build systems of differing complexity and distribution are accommodated. On Fri, Jul 28, 2017 at 3:53 PM Thomas Kluyver wrote:

On Fri, Jul 28, 2017, at 04:16 PM, Daniel Holth wrote:

It looks like we've run out of things to say about PEP 517, except, how soon can we get it into pip?

I admire your optimism! ;-)

While I partly hope that I get a unanimous disagreement, as it would be simpler, I have a nagging concern about something that someone mentioned ages ago: does it make sense for building sdists and building wheels to be part of the same backend?

Flit now makes both sdists and wheels, but for a long time it only made wheels, and the two parts are largely separate: it wouldn't take much work to use flit's sdist machinery but build the wheel with a different tool (e.g. if it had compiled parts).

Requiring one backend to build both formats may result in a significantly higher barrier to entry for backend developers: I don't know if I would have started writing flit if PEP 517 had already been finalised and I had to make both wheels and sdists to comply with it. They're also, at least to my mind, quite different kinds of thing: an sdist is almost like an archive of a VCS tag, whereas a wheel is the end result of any build steps the project needs.

So I'd like us to circle back round and reconsider allowing projects to specify 'use tool X to make wheels, and tool Y to make sdists'. If everyone else thinks that's unnecessary, I think we'd all be glad to finish this discussion up, but this concern has been growing in my mind for a while, and I want to get it out there before we finalise the PEP.

Thomas

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

See how trivial it would be to put the delegated sdist generator into the build-backend within the confines of the current PEP? The build-backend could point to a .py in the current directory that implements itself with different tools, or a delegating build backend could parse a separate source-backend out of pyproject.toml for kicks. So why worry? On Fri, Jul 28, 2017 at 4:30 PM Leonardo Rochael Almeida < leorochael@gmail.com> wrote:

Hi Thomas,

On 28 July 2017 at 16:53, Thomas Kluyver wrote:

...

[...] I have a nagging concern about something that someone mentioned ages ago: does it make sense for building sdists and building wheels to be part of the same backend?

[...]

...

So I'd like us to circle back round and reconsider allowing projects to specify 'use tool X to make wheels, and tool Y to make sdists'. If everyone else thinks that's unnecessary, I think we'd all be glad to finish this discussion up, but this concern has been growing in my mind for a while, and I want to get it out there before we finalise the PEP.

Great, now you've planted your nagging concern on my mind as well ;-)

What would using different backends for sdist and wheel look like? Something like this?

# pyproject.toml [build-system] requires = ["sdister", "wheeler"] source-backend = "sdister.api:main" build-backend = "wheeler.api:main"

If it's something like the above (or even, if it's a different section (`source-system`? I'm trying hard to avoid the `sdist` word), then perhaps this could be developed in a future pep without much issue. considering what Daniel just reminded us about tools being able to claim ignorance on how to develop sdists.

But if it's also that simple, maybe it could be added to this pep (/me ducks). _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Yes, the cwd strongly should be on sys.path, so many possibilities, and we are used to that behavior in setup.py. On Fri, Jul 28, 2017 at 4:59 PM Thomas Kluyver wrote:

On Fri, Jul 28, 2017, at 09:37 PM, Daniel Holth wrote:

See how trivial it would be to put the delegated sdist generator into the build-backend within the confines of the current PEP?

I agree that delegation within the backend is a reasonable answer to this.

The build-backend could point to a .py in the current directory that implements itself with different tools,

I think this was actually a question that someone (Nathaniel?) brought up: is the project source directory on sys.path when the backend is loaded? I would lean yes, so that it's possible to have a custom backend in the project tree, but I think whoever brought it up before was leaning no. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Fri, Jul 28, 2017 at 7:12 PM Nathaniel Smith wrote:

On Fri, Jul 28, 2017 at 1:58 PM, Thomas Kluyver wrote:

...

I think this was actually a question that someone (Nathaniel?) brought up: is the project source directory on sys.path when the backend is loaded? I would lean yes, so that it's possible to have a custom backend in the project tree, but I think whoever brought it up before was leaning no.

You're probably thinking of

https://github.com/njsmith/peps/blob/90f7adc130cbae741b1abd3a9ad407ba6731ea9...

As noted there, my suggestion would be that we don't put the project directory on sys.path by default, but that we do provide an explicit configuration key that lets projects name a directory in the project tree that should be prepended to sys.path. So example:

[build-system] python-path = ["."] build-backend = "my_custom_backend" # refers to my_custom_backend.py in the project root

# - or -

[build-system] python-path = ["tools"] build-backend = "my_custom_backend" # refers to tools/my_custom_backend.py

Rationale:

- Having the project source directory on sys.path when you don't expect it is confusing and error-prone - If you do want it on sys.path then it's easy to add a line to pyproject.toml, explicit > implicit - I think it's an anti-pattern how every piece of tooling wants to claim part of the top-level directory (.travis.yml, appveyor.yml, .readthedocs.yml, CONTRIBUTING.yml, ...) -- pyproject.toml has to go there because it's the root where all our tooling starts from, but we should support projects that want to tuck away the rest of their tooling into some subdirectory :-)

Given how much trouble we're having with PEP 517 already, it might make more sense to have PEP 517 just mandate that the directory not be on sys.path, and make a little follow-up PEP just for python-path.

But can you not also make bridges out of stone?

On Sat, Jul 29, 2017, at 09:47 AM, Paul Moore wrote:

+1 from me on just adding the project root to sys.path.

Ditto. I don't expect most projects to use it, but it seems like a handy bit of flexibility to have available. The potential clutter is a bit ugly, but I don't think it's worth the trouble of specifying an extra key to avoid that. Thomas

On 29 July 2017 at 17:46, Nathaniel Smith wrote:

So one consequence of this is that every time a new release of flit (for example) adds a new dependency on X, or one of flit's dependencies adds a new dependency on X, then this is technically a backwards incompatible change, because any existing packages that use flit and happen to have a top level directory named X will stop working. [...] Or am I worrying about a non-issue and it's fine if flit imports click from the source tree?

This sounds like a pretty rare issue, and one that I'd be inclined to assume isn't going to be a problem in practice - but if it were considered worth worrying about, why not just put the project root at the *end* of sys.path, so that packages installed normally will take precendence? Paul

On Jul 29, 2017, at 12:46 PM, Nathaniel Smith wrote:

I guess the most obvious example of when this would occur is: suppose click switches to using flit for builds, and then flit switches to using click for command line parsing. Now there's a bit of a chicken and egg problem where 'pip install click' will end up importing flit with the click source tree on the path, and this tree of course contains a directory named 'click', so unless special measures are taken flit will end up importing the code it's trying to build.

But of course this can happen for lots of reasons; most packages have names that you wouldn't expect to randomly encounter at the root of a source tree very often, but with 100,000+ packages on pypi I expect it will happen eventually.

This doesn't happen with setuptools because setuptools traditionally has few or no dependencies, but obviously we're changing that; the whole idea here is that now your build system has full access to pypi.

This is something to be discouraged anyways, because it creates a sort of broken situation (the same situation that setuptools itself had). The problem is that if you’re starting from only sdists, you have a circular dependency that cannot be broken. You can’t build click, because click requires flit, but you can’t install flit, because flit requires click. The only way to fix this is to either have an already built wheel that you can use (which obviously was either built with a flit that didn’t require click, or a click that didn’t require flit, or it’s provenance can be traced back to that) or do some hacks that will hopefully resolve the situation good enough to get your first wheel built. Setuptools tried to depend on things, and it broke shit for a lot of people because of this. You basically can’t depend on anything as a build system that uses you as a build system. You can only depend on things that use other, different build systems in the entire dependency tree. Likely the best thing for build systems to do is either have no dependencies, or to have minimal dependencies that promise to only use setuptools (or another build tool, which one doesn’t matter, just as long as it has no dependencies) forever (and have setuptools or this other build tool promise to never take a dependency). — Donald Stufft

I think the proposal is that flit depends on click depends on flit and neither one has a wheel and must be built from sdists. Then you have a circular build problem. So don't do that. I put this in the same category as accidentally conflicting with a stdlib module; it is confusing when it happens but it's also fairly avoidable. On Sat, Jul 29, 2017, 17:38 Alex Grönholm wrote:

Donald Stufft kirjoitti 29.07.2017 klo 23:47:

On Jul 29, 2017, at 12:46 PM, Nathaniel Smith wrote:

I guess the most obvious example of when this would occur is: suppose click switches to using flit for builds, and then flit switches to using click for command line parsing. Now there's a bit of a chicken and egg problem where 'pip install click' will end up importing flit with the click source tree on the path, and this tree of course contains a directory named 'click', so unless special measures are taken flit will end up importing the code it's trying to build.

But of course this can happen for lots of reasons; most packages have names that you wouldn't expect to randomly encounter at the root of a source tree very often, but with 100,000+ packages on pypi I expect it will happen eventually.

This doesn't happen with setuptools because setuptools traditionally has few or no dependencies, but obviously we're changing that; the whole idea here is that now your build system has full access to pypi.

This is something to be discouraged anyways, because it creates a sort of broken situation (the same situation that setuptools itself had). The problem is that if you’re starting from only sdists, you have a circular dependency that cannot be broken. You can’t build click, because click requires flit, but you can’t install flit, because flit requires click. The only way to fix this is to either have an already built wheel that you can use (which obviously was either built with a flit that didn’t require click, or a click that didn’t require flit, or it’s provenance can be traced back to that) or do some hacks that will hopefully resolve the situation good enough to get your first wheel built.

Setuptools tried to depend on things, and it broke shit for a lot of people because of this. You basically can’t depend on anything as a build system that uses you as a build system. You can only depend on things that use other, different build systems in the entire dependency tree. Likely the best thing for build systems to do is either have no dependencies, or to have minimal dependencies that promise to only use setuptools (or another build tool, which one doesn’t matter, just as long as it has no dependencies) forever (and have setuptools or this other build tool promise to never take a dependency).

Or vendorize their dependencies? To me it seems unrealistic for a build system to have no dependencies at all. Or perhaps this is exactly what you meant :)

— Donald Stufft

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Yes vendoring would be a solution but it's too soon to say whether the problem is significant. Setuptools and pip have a different problem which is that the package manager isn't available yet. In pep517 the package manager is available. On Sat, Jul 29, 2017, 17:50 Alex Grönholm wrote:

Daniel Holth kirjoitti 30.07.2017 klo 00:48:

I think the proposal is that flit depends on click depends on flit and neither one has a wheel and must be built from sdists. Then you have a circular build problem. So don't do that. I put this in the same category as accidentally conflicting with a stdlib module; it is confusing when it happens but it's also fairly avoidable.

Sure but vendorizing the dependencies would work around the problem, yes? Like how setuptools does?

On Sat, Jul 29, 2017, 17:38 Alex Grönholm wrote:

...

Donald Stufft kirjoitti 29.07.2017 klo 23:47:

On Jul 29, 2017, at 12:46 PM, Nathaniel Smith wrote:

I guess the most obvious example of when this would occur is: suppose click switches to using flit for builds, and then flit switches to using click for command line parsing. Now there's a bit of a chicken and egg problem where 'pip install click' will end up importing flit with the click source tree on the path, and this tree of course contains a directory named 'click', so unless special measures are taken flit will end up importing the code it's trying to build.

But of course this can happen for lots of reasons; most packages have names that you wouldn't expect to randomly encounter at the root of a source tree very often, but with 100,000+ packages on pypi I expect it will happen eventually.

This doesn't happen with setuptools because setuptools traditionally has few or no dependencies, but obviously we're changing that; the whole idea here is that now your build system has full access to pypi.

This is something to be discouraged anyways, because it creates a sort of broken situation (the same situation that setuptools itself had). The problem is that if you’re starting from only sdists, you have a circular dependency that cannot be broken. You can’t build click, because click requires flit, but you can’t install flit, because flit requires click. The only way to fix this is to either have an already built wheel that you can use (which obviously was either built with a flit that didn’t require click, or a click that didn’t require flit, or it’s provenance can be traced back to that) or do some hacks that will hopefully resolve the situation good enough to get your first wheel built.

Setuptools tried to depend on things, and it broke shit for a lot of people because of this. You basically can’t depend on anything as a build system that uses you as a build system. You can only depend on things that use other, different build systems in the entire dependency tree. Likely the best thing for build systems to do is either have no dependencies, or to have minimal dependencies that promise to only use setuptools (or another build tool, which one doesn’t matter, just as long as it has no dependencies) forever (and have setuptools or this other build tool promise to never take a dependency).

Or vendorize their dependencies? To me it seems unrealistic for a build system to have no dependencies at all. Or perhaps this is exactly what you meant :)

— Donald Stufft

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Pip has to vendor its dependencies primarily because it can't pip install them. Not the same problem as build systems. On Sat, Jul 29, 2017, 18:00 Alex Grönholm wrote:

Daniel Holth kirjoitti 30.07.2017 klo 00:58:

Yes vendoring would be a solution but it's too soon to say whether the problem is significant. Setuptools and pip have a different problem which is that the package manager isn't available yet. In pep517 the package manager is available.

What package manager are you referring to?

On Sat, Jul 29, 2017, 17:50 Alex Grönholm wrote:

...

Daniel Holth kirjoitti 30.07.2017 klo 00:48:

I think the proposal is that flit depends on click depends on flit and neither one has a wheel and must be built from sdists. Then you have a circular build problem. So don't do that. I put this in the same category as accidentally conflicting with a stdlib module; it is confusing when it happens but it's also fairly avoidable.

Sure but vendorizing the dependencies would work around the problem, yes? Like how setuptools does?

On Sat, Jul 29, 2017, 17:38 Alex Grönholm wrote:

...

Donald Stufft kirjoitti 29.07.2017 klo 23:47:

On Jul 29, 2017, at 12:46 PM, Nathaniel Smith wrote:

I guess the most obvious example of when this would occur is: suppose click switches to using flit for builds, and then flit switches to using click for command line parsing. Now there's a bit of a chicken and egg problem where 'pip install click' will end up importing flit with the click source tree on the path, and this tree of course contains a directory named 'click', so unless special measures are taken flit will end up importing the code it's trying to build.

But of course this can happen for lots of reasons; most packages have names that you wouldn't expect to randomly encounter at the root of a source tree very often, but with 100,000+ packages on pypi I expect it will happen eventually.

This doesn't happen with setuptools because setuptools traditionally has few or no dependencies, but obviously we're changing that; the whole idea here is that now your build system has full access to pypi.

This is something to be discouraged anyways, because it creates a sort of broken situation (the same situation that setuptools itself had). The problem is that if you’re starting from only sdists, you have a circular dependency that cannot be broken. You can’t build click, because click requires flit, but you can’t install flit, because flit requires click. The only way to fix this is to either have an already built wheel that you can use (which obviously was either built with a flit that didn’t require click, or a click that didn’t require flit, or it’s provenance can be traced back to that) or do some hacks that will hopefully resolve the situation good enough to get your first wheel built.

Setuptools tried to depend on things, and it broke shit for a lot of people because of this. You basically can’t depend on anything as a build system that uses you as a build system. You can only depend on things that use other, different build systems in the entire dependency tree. Likely the best thing for build systems to do is either have no dependencies, or to have minimal dependencies that promise to only use setuptools (or another build tool, which one doesn’t matter, just as long as it has no dependencies) forever (and have setuptools or this other build tool promise to never take a dependency).

Or vendorize their dependencies? To me it seems unrealistic for a build system to have no dependencies at all. Or perhaps this is exactly what you meant :)

— Donald Stufft

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

The only way to avoid it is to not have a circular dependency. Sent from my iPhone

On Jul 29, 2017, at 5:48 PM, Daniel Holth wrote:

I think the proposal is that flit depends on click depends on flit and neither one has a wheel and must be built from sdists. Then you have a circular build problem. So don't do that. I put this in the same category as accidentally conflicting with a stdlib module; it is confusing when it happens but it's also fairly avoidable.

There exist many systems in the real world that require building everything from source. Setuptools circular dependencies broke all of them (and any circular build time dependency would). The one I'm most familiar with has special cased a bootstrap C compiler and a very limited shell with a few other builtins and then the very first thing it does is build the "real" C compiler, shell, etc from source and then throws away the bootstrap scripts. Anything that has a circular dependency is going to force those systems to special case them as well. Most of them will just avoid those packages if they can at all help it because random Python packages do not rise to the level of special as gcc does. Sent from my iPhone

On Jul 29, 2017, at 7:19 PM, Nathaniel Smith wrote:

It's true that this is problematic for packages that don't have wheels, but flit and click are both pure python packages with wheels available, so I don't think there's any problem here in practice. (In fact flit currently uses flit as its build system, and depends on at least one other package that uses flit as its build system.)

I thought the big problem with setuptools unvendoring was that we don't currently have isolation for build requirements, so there was interference between what setuptools wanted and what other packages wanted? PEP 517 doesn't have that problem because build backends are installed into their own little quasi-virtualenv. Or am I wrong?

Are we content to say that sys.path includes the source directory where the hook is run? Shall I prepare a PR against the PEP for that? On Sun, Jul 30, 2017, at 02:12 PM, Nick Coghlan wrote:

On 30 July 2017 at 02:46, Nathaniel Smith wrote:

...

Or am I worrying about a non-issue and it's fine if flit imports click from the source tree?

Don't worry about it too much, as the problem here isn't really any worse than it is for normal runtime dependencies of any other project that relies on having the current directory first in sys.path. It just so happens that the project in question in this case is a Python project's build system.

Due to the preference for a flat module namespace as the default, there are plenty of ways to hit name shadowing problems in Python, and as Donald notes, build systems have other motives to vendor their dependencies rather than installing them normally.

Just switching the path order as has been suggested also doesn't solve the problem, as it merely inverts the issue: having "some_name" installed in site-packages would break source installations of packages that expected to be able to import "some_name" from their own root directory.

If the problem does come up in practice, then there are a number of ways for affected projects to work around it in their project directory structure:

1. Use a top-level "src" directory (we may want to reserve "src" on PyPI) 2. Use a top-level "tools" directory (we may want to reserve "tools" on PyPI) 3. Add a leading or trailing underscore to the local directory name (as while that's legal for Python imports, it's prohibited for PyPI project names, and hence will often sidestep naming conflicts with published packages)

Beyond that, the only approaches I'm aware of that systematically avoid this kind of problem at the language design level are to either use URL-based imports (ala Java or Go), or else to have separate syntax for "system-only" and "local resolution permitted" imports (ala C and C++), and Guido opted not to pursue either of those strategies for Python.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Aug 1, 2017 at 9:31 AM, Thomas Kluyver wrote:

Are we content to say that sys.path includes the source directory where the hook is run? Shall I prepare a PR against the PEP for that?

It doesn't matter whether sys.path includes the source directory when the hook is run. At that point the hook can manipulate sys.path however it likes, just like setup.py can. What matters is whether sys.path includes the source directory before the backend is imported. And I think that even if we keep the source directory in sys.path, then there's at least a further piece of refinement needed. The obvious mechanism for adding the source directory to sys.path is by setting PYTHONPATH before invoking the sub-python that runs the build backend. That's how we put other directories on sys.path for injecting requirements, and there's a bit of text in the PEP talking about how the execution environment needs to be inherited over calls like subprocess.run([sys.executable, ...]); if this requirement also applies to the source tree's presence in sys.path, then we *have* to put the source tree in PYTHONPATH. But... this creates some special problems. Putting the source directory in PYTHONPATH is different from the familiar 'python setup.py' behavior. When you run 'python setup.py', the sequence of events is: 1) The interpreter bootstraps itself 2) The interpreter adds the script directory to the front of sys.path 3) The code in setup.py starts running, and can choose to either leave sys.path alone or modify it before it imports anything with the new sys.path But if we use PYTHONPATH to add the source tree to sys.path, then what happens is: 1) The source tree gets added to the front of sys.path 2) The interpreter bootstraps itself 3) The frontend imports the backend 4) The frontend invokes the backend hook 5) The backend starts running, and can set up whatever environment it likes before doing anything else So the difference is that with setup.py, there aren't any imports between the sys.path manipulation and handing control the script, so if the script doesn't like their path they can easily change it. And if there is an import problem, then it's because of a clash between something that the package author's setup.py did, and something they put in their package, both of which are under their control. (For example, it's quite common to see setup.py's and other scripts do a 'sys.path.pop(0)' at the top to avoid any chance of mistakes.) The sys.path manipulation is a convenience, but it doesn't actually affect anything by itself. With PYTHONPATH, the sys.path manipulation happens earlier, so you can get collisions between the source tree and the Python bootstrap logic. For example: ~$ mkdir temp ~$ cd temp ~/temp$ touch io.py setup.py # No problem running a script in this dir ~/temp$ python3 setup.py # But with PYTHONPATH... ~/temp$ PYTHONPATH=. python3 setup.py Fatal Python error: Py_Initialize: can't initialize sys standard streams AttributeError: module 'io' has no attribute 'OpenWrapper' Current thread 0x00007fd862d83700 (most recent call first): zsh: abort PYTHONPATH=. python3 setup.py I bet if someone filed a bug saying "I ran 'pip install foo' and got that error mesage" then it would take some time to figure out what went wrong :-). Of course, it's unlikely that people will have io.py files sitting around. But there's a whole list of modules that get imported automatically during interpreter startup, and which are shadowed by PYTHONPATH. For example, I just checked Github's bigquery data and found dozens of repos that have a file called 'site.py' in their root -- probably because they're a web site or something. But of course if we use PYTHONPATH then this shadows Python's internal site.py that's imported for bootstrapping... We could avoid the interpreter bootstrap problems by adding language to the spec saying that yes, the source tree goes at the beginning of sys.path, BUT this MUST NOT be done using PYTHONPATH, and update the environment inheritance language to specify that this is a special case that is *not* inherited by child pythons. Kind of annoying when the only argument for automatically putting the source tree on sys.path is to simplify the spec, but ok... And then of course you can also get collisions between the source tree and the import of the build backend, as discussed earlier in the thread. Obviously this can't be fixed, since the whole point is to support cases where the build backend is in the source tree. If this does happen then... I dunno, maybe the backend shouldn't have imported that stuff, or maybe the source tree shouldn't have had a directory called that? But too late, it's out there on PyPI and pip is getting bug reports and there isn't even clear whose fault it is or who should fix it. If it were setup.py the obvious solution would be to add a 'sys.path.pop(0)', but here we've eliminated that option. I get the impression that a lot of people are like "well, that's a low probability event, and the rest of the time it's really handy, so whatever, it's worth it". And I agree it's a low probability event, but what I'm not sure people are fully realizing is... it's actually not very handy either :-). The *only* people who benefit from putting the source tree in sys.path are package authors who want to ship some custom one-off build backend inside their package. If you're writing a build backend that ships via PyPI, or you're writing a package that uses an existing build system pulled from PyPI, then having build frontends put the source directory on sys.path provides *zero* value to balance this small-but-non-zero risk. I think there's a real chance that if we ship this enabled by default in the name of "simplicity", then not only will we first have to add the special case alluded to above, but we'll eventually decide that we need to provide an opt-out flag, and then we'll have to communicate to projects that almost all of them actually want this opt-out flag, because the default is tuned for the minority of projects who are are doing something special. I don't know about you, but I am *so tired* of trying to explain to folks about "best practices" where the default is wrong. OTOH in the alternative approach where this opt-in, then the only folks who have to worry about this are the ones writing one-off build backends, and seriously once you've made the life choice to implement a one-off build backend then adding a line to pyproject.toml is really the least of your worries. People with special needs should be the ones paying the special tax. -n

On Sun, Jul 30, 2017, at 02:12 PM, Nick Coghlan wrote:

...

On 30 July 2017 at 02:46, Nathaniel Smith wrote:

...

Or am I worrying about a non-issue and it's fine if flit imports click from the source tree?

Don't worry about it too much, as the problem here isn't really any worse than it is for normal runtime dependencies of any other project that relies on having the current directory first in sys.path. It just so happens that the project in question in this case is a Python project's build system.

Due to the preference for a flat module namespace as the default, there are plenty of ways to hit name shadowing problems in Python, and as Donald notes, build systems have other motives to vendor their dependencies rather than installing them normally.

Just switching the path order as has been suggested also doesn't solve the problem, as it merely inverts the issue: having "some_name" installed in site-packages would break source installations of packages that expected to be able to import "some_name" from their own root directory.

If the problem does come up in practice, then there are a number of ways for affected projects to work around it in their project directory structure:

1. Use a top-level "src" directory (we may want to reserve "src" on PyPI) 2. Use a top-level "tools" directory (we may want to reserve "tools" on PyPI) 3. Add a leading or trailing underscore to the local directory name (as while that's legal for Python imports, it's prohibited for PyPI project names, and hence will often sidestep naming conflicts with published packages)

Beyond that, the only approaches I'm aware of that systematically avoid this kind of problem at the language design level are to either use URL-based imports (ala Java or Go), or else to have separate syntax for "system-only" and "local resolution permitted" imports (ala C and C++), and Guido opted not to pursue either of those strategies for Python.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-- Nathaniel J. Smith -- https://vorpus.org

On Wed, Aug 2, 2017, at 01:05 PM, Nick Coghlan wrote:

I wouldn't say that, as the two most clearly side effect free ways of putting the source directory on sys.path are:

1. make it the current working directory for a "python -m" or "python -c" invocation, so that the current directory gets injected as sys.path[0] by the interpreter. After all, it's already a requirement in the PEP that the current working directory be the directory containing pyproject.toml, so this approach handles that as well. 2. run a backend script directly (so it's the script directory that gets implicitly added to sys.path by the interpreter), and then inject the current working directory as sys.path[0] before importing the nominated backend

If a frontend decides to go messing about with PYTHONPATH instead of doing either of those, then that's on them (and after it breaks horribly, they'll hopefully realise the error of their ways and switch to one of the other more reliable approaches).

Nathaniel, can you live with the source directory being on sys.path to load the hooks? Or would you like to have another go at convincing people that it shouldn't be? Thomas

On 29 July 2017 at 06:58, Thomas Kluyver wrote:

On Fri, Jul 28, 2017, at 09:37 PM, Daniel Holth wrote:

See how trivial it would be to put the delegated sdist generator into the build-backend within the confines of the current PEP?

I agree that delegation within the backend is a reasonable answer to this.

Yeah, I think the way to lower this particular barrier to entry is through a traditional helper module, or in the absence of that, folks just copying ideas and code from existing permissively licensed backends like flit and enscons. As for when this idea came up originally, I'm pretty sure at least *I* suggested it at one point, and was subsequently persuaded to change my mind by the above argument :) So while I don't think we should change the proposal, it may be worth listing "Separate sdist and wheel building backends" explicitly in the "Rejected Options" section, with the rationale that different backends will still be able to share common sdist building code through regular open source development processes. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jul 28, 2017, at 3:53 PM, Thomas Kluyver wrote:

On Fri, Jul 28, 2017, at 04:16 PM, Daniel Holth wrote:

...

It looks like we've run out of things to say about PEP 517, except, how soon can we get it into pip?

I admire your optimism! ;-)

While I partly hope that I get a unanimous disagreement, as it would be simpler, I have a nagging concern about something that someone mentioned ages ago: does it make sense for building sdists and building wheels to be part of the same backend?

Flit now makes both sdists and wheels, but for a long time it only made wheels, and the two parts are largely separate: it wouldn't take much work to use flit's sdist machinery but build the wheel with a different tool (e.g. if it had compiled parts).

Requiring one backend to build both formats may result in a significantly higher barrier to entry for backend developers: I don't know if I would have started writing flit if PEP 517 had already been finalised and I had to make both wheels and sdists to comply with it. They're also, at least to my mind, quite different kinds of thing: an sdist is almost like an archive of a VCS tag, whereas a wheel is the end result of any build steps the project needs.

So I'd like us to circle back round and reconsider allowing projects to specify 'use tool X to make wheels, and tool Y to make sdists'. If everyone else thinks that's unnecessary, I think we'd all be glad to finish this discussion up, but this concern has been growing in my mind for a while, and I want to get it out there before we finalise the PEP.

I think if we do that, we *have* to mandate that the only path to take is VCS -> Sdist -> Wheel, because otherwise every tool is going to end up coupled. This is ultimately because you need to know how you’re supposed to get metadata. For instance, if we say “You can specify foo to build sdists, and bar to build wheels”, and if foo wants to pull version numbers from git tags, but bar wants to pull it from __init__.py AND you allow VCS -> sdist and VCS -> wheel, then you end up in a situation where the two tools disagree and do the complete wrong thing. However if you mandate that the only path is VCS -> sdist -> wheel, then the sdist tool is responsible for generating metadata like version etc, and the wheel tool only consumes the previous metadata created by the sdist tool (and adds it’s own, like wheel tag and such). I suspect that we don’t want to do that thought (even if I think that it’s a good idea!) because the flexibility afforded by allowing both means we can adapt to whatever the reality of the situation is in the end if one works better then the other. — Donald Stufft