I haven't read those discussions in full (sorry to be lazy, but there's quite a lot there), but my initial take is that the rule doesn't apply to running setup.py scripts, where there's a greater need for backward compatibility. I think the rule about the CWD not being on sys.path only applies to loading a proper PEP 517 build backend when that's specified in pyproject.toml. Furthermore, if the build backend then wants to run some code with the CWD on sys.path, I think it's free to do so, so this could be implemented in the setuptools backend. The rule is only for how the frontend loads the backend, not what the backend does afterwards. Where a real PEP 517 backend needs to be loaded from the CWD (e.g. for Flit to bootstrap itself), we've already got the intreehooks package to make this possible: https://pypi.org/project/intreehooks/ On Sun, Jan 27, 2019, at 2:26 PM, Chris Jerdonek wrote:

For those of you who participated in the PEP 517 discussion during the summer of 2017 (just prior to its provisional acceptance), I want to flag that one of the issues discussed back then has now resurfaced for discussion. This is because the feature was turned on by default in pip's latest release (19.0) less than a week ago.> The issue is the one around whether the source directory should be included in sys.path. The resurfacing is more or less as predicted. For example, in one email [1] from August 29, 2017, Nick summarized the state of things by saying (not too long before provisional acceptance)-->

...

So I think we can deem this one resolved in favour of "Frontends must ensure the current directory is *NOT* on sys.path before importing the designated backend", as starting there will mean we maximise our chances of learning something new as part of the initial rollout of the provisionally accepted API.> And also--

...

2. If omitting it is genuinely a problem, we'll likely find out soon enough as part of implementing a setup.py backend> Basically, that "soon enough" moment has arrived, and at least one discussion on how to resolve it has started on the tracker here:> https://github.com/pypa/pip/issues/6163

There is another discussion here, but it's probably better for the discussion to be in one spot:> https://github.com/pypa/setuptools/issues/1642 Maybe Discourse?

--Chris

[1]: https://mail.python.org/pipermail/distutils-sig/2017-August/031413.html> -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/LMGQS...

On Sun, 27 Jan 2019 at 14:39, Thomas Kluyver wrote:

I think the rule about the CWD not being on sys.path only applies to loading a proper PEP 517 build backend when that's specified in pyproject.toml.

I'm not entirely sure what you're intending when you refer to a "proper PEP 517 build backend". The setuptools backend is a perfectly acceptable one, the only difference here is that we default to it in pip when the project hasn't specified a backend. The reason we do that is simple - if we don't, we're unlikely to be able to remove the legacy setuptools code any time in the foreseeable future, as there will be essentially no incentive for existing projects to specify a setuptools backend. Also, we'd get basically no testing of the new PEP 517 code, for the same reason. (Yes, it would have been better to test before release, but we've never had any success getting beta testers for pip). We could have made PEP 517 entirely opt-in, but (a) that would have drastically slowed down adoption, and (b) implicit in everything we'd discussed was the assumption that there would be a setuptools backend that was semantically equivalent to running setup.py the way we currently did, so there was no need to. Clearly that assumption turned out to be wrong, and that's where the issues for our users arose. Having said this, I don't have any problem with changing the default backend used by pip. In fact, if the setuptools project don't view it as a goal of their existing backend to replicate setup.py behaviour, then I think it's pretty much essential that we change. Pip needs to default to a PEP 517 backend that behaves the same as pip's legacy code - that's essential for backward compatibility if we're to remove or deprecate the legacy setup.py code path. The only problem is that right now there doesn't appear to be a backend that suits our requirements. Paul Ganssle suggested that setuptools could provide an alternative ("legacy") setuptools backend that preserved full setup.py compatibility (including having the current directory on sys.path). That seems to me to be a reasonable solution, and I appreciate the offer. In practice, the legacy setuptools backend doesn't *have* to be implemented by setuptools itself - it could be a separate project if setuptools don't want to maintain it themselves. But IMO it would be better if the setuptools team are OK with maintaining it.(At a pinch, pip could even implement that backend internally, but adding pip to every build environment seems a bit heavyweight). So in terms of practical resolution of the problems people are hitting since pip 19.0 was released, I don't think there's a significant issue here. We need a new pip release, sure, and someone needs to write the new backend, but I don't think either of those tasks are huge. What is more difficult is the question of whether the PEP should change. As Chris pointed out, the previous discussion ended up saying that the build directory should not be on sys.path, but acknowledged that mandating that might cause issues. So the question now is, are the issues we've seen big enough that we want to change PEP 517 to say that the build directory *should* be on sys.path? We don't have to answer that question urgently (the urgent resolution is with a legacy backend, as noted above) but we do have to decide one way or another. And here the question is whether enforcing projects to adopt the "good practice" of not relying on being able to import the project at build time is worth the negative impact of requiring what appears to be a non-trivial number of projects to change their build code. Or whether there's an "intermediate" option (the discussions Chris linked to included talk of adding another pyproject.toml value which would be a list of directories the frontend should add to sys.path when calling hooks, for example). But that discussion shouldn't be made under a false impression of urgency - it's not on the critical path for fixing pip's immediate issue. Paul

Yes, of course people can. I think the problem is that common Python installations defaults to adding the cwd into sys.path, so people expect this to “just work”. A PEP 517 not doing it is not intuitive unless you follow Python packaging closely, and most would simply assume pip is broken. (Incidentally, the Windows embedded distribution also does not include the cwd in sys.path by default, and I do see confused people on discussion boards or issue trackers from time to time complaining things don’t work. The embedded distribution’s user base is quite small, however, so this is not really a problem.) -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone

On 28 Jan 2019, at 04:17, Bernat Gabor wrote:

It feels to me that importing from setup.py level is a code smell and should be avoided (I mean what happens if you move the source into src layout, do you now want tools to also append the src folder additionally if exists?). Can't people just inject at the start of setup.py into sys path the cwd if they really want to go with it? (feels superior over pinning pip to 18.0).

...

On Sun, Jan 27, 2019 at 7:48 PM Paul Moore wrote: On Sun, 27 Jan 2019 at 14:39, Thomas Kluyver wrote:

...

I think the rule about the CWD not being on sys.path only applies to loading a proper PEP 517 build backend when that's specified in pyproject.toml.

I'm not entirely sure what you're intending when you refer to a "proper PEP 517 build backend". The setuptools backend is a perfectly acceptable one, the only difference here is that we default to it in pip when the project hasn't specified a backend. The reason we do that is simple - if we don't, we're unlikely to be able to remove the legacy setuptools code any time in the foreseeable future, as there will be essentially no incentive for existing projects to specify a setuptools backend. Also, we'd get basically no testing of the new PEP 517 code, for the same reason. (Yes, it would have been better to test before release, but we've never had any success getting beta testers for pip). We could have made PEP 517 entirely opt-in, but (a) that would have drastically slowed down adoption, and (b) implicit in everything we'd discussed was the assumption that there would be a setuptools backend that was semantically equivalent to running setup.py the way we currently did, so there was no need to. Clearly that assumption turned out to be wrong, and that's where the issues for our users arose.

Having said this, I don't have any problem with changing the default backend used by pip. In fact, if the setuptools project don't view it as a goal of their existing backend to replicate setup.py behaviour, then I think it's pretty much essential that we change. Pip needs to default to a PEP 517 backend that behaves the same as pip's legacy code - that's essential for backward compatibility if we're to remove or deprecate the legacy setup.py code path. The only problem is that right now there doesn't appear to be a backend that suits our requirements.

Paul Ganssle suggested that setuptools could provide an alternative ("legacy") setuptools backend that preserved full setup.py compatibility (including having the current directory on sys.path). That seems to me to be a reasonable solution, and I appreciate the offer. In practice, the legacy setuptools backend doesn't *have* to be implemented by setuptools itself - it could be a separate project if setuptools don't want to maintain it themselves. But IMO it would be better if the setuptools team are OK with maintaining it.(At a pinch, pip could even implement that backend internally, but adding pip to every build environment seems a bit heavyweight).

So in terms of practical resolution of the problems people are hitting since pip 19.0 was released, I don't think there's a significant issue here. We need a new pip release, sure, and someone needs to write the new backend, but I don't think either of those tasks are huge.

What is more difficult is the question of whether the PEP should change. As Chris pointed out, the previous discussion ended up saying that the build directory should not be on sys.path, but acknowledged that mandating that might cause issues. So the question now is, are the issues we've seen big enough that we want to change PEP 517 to say that the build directory *should* be on sys.path? We don't have to answer that question urgently (the urgent resolution is with a legacy backend, as noted above) but we do have to decide one way or another. And here the question is whether enforcing projects to adopt the "good practice" of not relying on being able to import the project at build time is worth the negative impact of requiring what appears to be a non-trivial number of projects to change their build code. Or whether there's an "intermediate" option (the discussions Chris linked to included talk of adding another pyproject.toml value which would be a list of directories the frontend should add to sys.path when calling hooks, for example). But that discussion shouldn't be made under a false impression of urgency - it's not on the critical path for fixing pip's immediate issue.

Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/SG2VB... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/4HE67...

I guess this depends on how explicit you wan to be. PEP 517 is not enabled unconditionally, only when the project root contains pyproject.toml. The problem is that other projects (unrelated to packaging) take advantage of the file format’s existence, resulting in people “buy in” the situation unknowingly. Things are already explicit from pip’s point of view, but (unanticipated?) third-party usages of the same spec unrelated to PEP 517 muddles the situation. -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone

On 28 Jan 2019, at 04:37, Bernat Gabor wrote:

This is more an argument to not use pep-517 unless people explicitly specify the backend, at which point they acknowledge to buy in that the cwd is not on sys path (and they need to alter their packaging code accordingly).

...

On Sun, 27 Jan 2019, 20:31 Tzu-ping Chung

(Incidentally, the Windows embedded distribution also does not include the cwd in sys.path by default, and I do see confused people on discussion boards or issue trackers from time to time complaining things don’t work. The embedded distribution’s user base is quite small, however, so this is not really a problem.)

-- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone

...

On 28 Jan 2019, at 04:17, Bernat Gabor wrote:

It feels to me that importing from setup.py level is a code smell and should be avoided (I mean what happens if you move the source into src layout, do you now want tools to also append the src folder additionally if exists?). Can't people just inject at the start of setup.py into sys path the cwd if they really want to go with it? (feels superior over pinning pip to 18.0).

...

On Sun, Jan 27, 2019 at 7:48 PM Paul Moore wrote: On Sun, 27 Jan 2019 at 14:39, Thomas Kluyver wrote:

...

I think the rule about the CWD not being on sys.path only applies to loading a proper PEP 517 build backend when that's specified in pyproject.toml.

I'm not entirely sure what you're intending when you refer to a "proper PEP 517 build backend". The setuptools backend is a perfectly acceptable one, the only difference here is that we default to it in pip when the project hasn't specified a backend. The reason we do that is simple - if we don't, we're unlikely to be able to remove the legacy setuptools code any time in the foreseeable future, as there will be essentially no incentive for existing projects to specify a setuptools backend. Also, we'd get basically no testing of the new PEP 517 code, for the same reason. (Yes, it would have been better to test before release, but we've never had any success getting beta testers for pip). We could have made PEP 517 entirely opt-in, but (a) that would have drastically slowed down adoption, and (b) implicit in everything we'd discussed was the assumption that there would be a setuptools backend that was semantically equivalent to running setup.py the way we currently did, so there was no need to. Clearly that assumption turned out to be wrong, and that's where the issues for our users arose.

Having said this, I don't have any problem with changing the default backend used by pip. In fact, if the setuptools project don't view it as a goal of their existing backend to replicate setup.py behaviour, then I think it's pretty much essential that we change. Pip needs to default to a PEP 517 backend that behaves the same as pip's legacy code - that's essential for backward compatibility if we're to remove or deprecate the legacy setup.py code path. The only problem is that right now there doesn't appear to be a backend that suits our requirements.

Paul Ganssle suggested that setuptools could provide an alternative ("legacy") setuptools backend that preserved full setup.py compatibility (including having the current directory on sys.path). That seems to me to be a reasonable solution, and I appreciate the offer. In practice, the legacy setuptools backend doesn't *have* to be implemented by setuptools itself - it could be a separate project if setuptools don't want to maintain it themselves. But IMO it would be better if the setuptools team are OK with maintaining it.(At a pinch, pip could even implement that backend internally, but adding pip to every build environment seems a bit heavyweight).

So in terms of practical resolution of the problems people are hitting since pip 19.0 was released, I don't think there's a significant issue here. We need a new pip release, sure, and someone needs to write the new backend, but I don't think either of those tasks are huge.

What is more difficult is the question of whether the PEP should change. As Chris pointed out, the previous discussion ended up saying that the build directory should not be on sys.path, but acknowledged that mandating that might cause issues. So the question now is, are the issues we've seen big enough that we want to change PEP 517 to say that the build directory *should* be on sys.path? We don't have to answer that question urgently (the urgent resolution is with a legacy backend, as noted above) but we do have to decide one way or another. And here the question is whether enforcing projects to adopt the "good practice" of not relying on being able to import the project at build time is worth the negative impact of requiring what appears to be a non-trivial number of projects to change their build code. Or whether there's an "intermediate" option (the discussions Chris linked to included talk of adding another pyproject.toml value which would be a list of directories the frontend should add to sys.path when calling hooks, for example). But that discussion shouldn't be made under a false impression of urgency - it's not on the critical path for fixing pip's immediate issue.

Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/SG2VB... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/4HE67...

On Sun, 27 Jan 2019 at 21:02, Thomas Kluyver wrote:

On Sun, Jan 27, 2019, at 7:47 PM, Paul Moore wrote:

...

What is more difficult is the question of whether the PEP should change. As Chris pointed out, the previous discussion ended up saying that the build directory should not be on sys.path, but acknowledged that mandating that might cause issues. So the question now is, are the issues we've seen big enough that we want to change PEP 517 to say that the build directory *should* be on sys.path?

I don't see that this issue is an argument for changing the PEP. It seems like a problem with the backend that should be supporting legacy use cases. The rule about the frontend not using the CWD to load the backend doesn't constrain what the backend does once loaded.

From my perspective, the original arguments about the CWD still stand, but in addition PEP 517 is out now and there are tools implementing it, so it would be more disruptive to make changes. So at least for now, I'm -1 on changing the rule about the CWD.

Yes, that's my view too. I'm happy that we revisit the question, but personally, I remain in favour of the PEP as it stands. Paul

On Mon, 28 Jan 2019 at 09:04, Paul Moore wrote:

On Sun, 27 Jan 2019 at 21:02, Thomas Kluyver wrote:

...

On Sun, Jan 27, 2019, at 7:47 PM, Paul Moore wrote:

...

What is more difficult is the question of whether the PEP should change. As Chris pointed out, the previous discussion ended up saying that the build directory should not be on sys.path, but acknowledged that mandating that might cause issues. So the question now is, are the issues we've seen big enough that we want to change PEP 517 to say that the build directory *should* be on sys.path?

I don't see that this issue is an argument for changing the PEP. It seems like a problem with the backend that should be supporting legacy use cases. The rule about the frontend not using the CWD to load the backend doesn't constrain what the backend does once loaded.

From my perspective, the original arguments about the CWD still stand, but in addition PEP 517 is out now and there are tools implementing it, so it would be more disruptive to make changes. So at least for now, I'm -1 on changing the rule about the CWD.

Yes, that's my view too. I'm happy that we revisit the question, but personally, I remain in favour of the PEP as it stands.

Yep, I think letting the backend decide whether or not it wants to include the build directory on sys.path remains the right way to go. Thomas's https://github.com/takluyver/intreehooks then addresses opting in to that in the general case, while Paul Ganssle's proposed build_meta_legacy backend in https://github.com/pypa/setuptools/pull/1652 addresses it for the specific case of setup.py execution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 1/29/19 7:46 AM, David Shawley wrote:

I agree with pip's view that the presence of pyproject.toml means that you are *explicitly* announcing compliance with PEP-517.  The unexpected part was that the setuptools build-system shim was incomplete.  This will be addressed shortly if Paul Ganssle has anything to do with it (thanks Paul!).

Thanks for the shout out, and I really do appreciate the recognition, but I would like to clarify that if anything I have /delayed/ getting this fixed with my insistence that the semantics should be different between setuptools.build_meta and setup.py. I think there are already some useful features of this distinction, but it's certainly true that if we had simply modified the path in the existing build backend, this would be fixed now (with the release of the latest setuptools) instead of necessitating a new pip release. Anyway, even if I was right to insist on this distinction, I don't want to take the credit for fixing this when a lot of people put in a lot of effort discussing this and working out the right path forward, which is very tough in packaging, where people want many new and useful features, but they're also weary of all the churn in their build and deployment system. That said, I hope this doesn't sound like I'm complaining that you complimented my work. I would hate to make you think that someone could be bothered by the fact that you thanked an open source maintainer. Best, Paul