PEP 517 - source dir and sys.path

For those of you who participated in the PEP 517 discussion during the summer of 2017 (just prior to its provisional acceptance), I want to flag that one of the issues discussed back then has now resurfaced for discussion. This is because the feature was turned on by default in pip's latest release (19.0) less than a week ago. The issue is the one around whether the source directory should be included in sys.path. The resurfacing is more or less as predicted. For example, in one email [1] from August 29, 2017, Nick summarized the state of things by saying (not too long before provisional acceptance)-- So I think we can deem this one resolved in favour of "Frontends must
And also-- 2. If omitting it is genuinely a problem, we'll likely find out soon enough
as part of implementing a setup.py backend
Basically, that "soon enough" moment has arrived, and at least one discussion on how to resolve it has started on the tracker here: https://github.com/pypa/pip/issues/6163 There is another discussion here, but it's probably better for the discussion to be in one spot: https://github.com/pypa/setuptools/issues/1642 Maybe Discourse? --Chris [1]: https://mail.python.org/pipermail/distutils-sig/2017-August/031413.html

I haven't read those discussions in full (sorry to be lazy, but there's quite a lot there), but my initial take is that the rule doesn't apply to running setup.py scripts, where there's a greater need for backward compatibility. I think the rule about the CWD not being on sys.path only applies to loading a proper PEP 517 build backend when that's specified in pyproject.toml. Furthermore, if the build backend then wants to run some code with the CWD on sys.path, I think it's free to do so, so this could be implemented in the setuptools backend. The rule is only for how the frontend loads the backend, not what the backend does afterwards. Where a real PEP 517 backend needs to be loaded from the CWD (e.g. for Flit to bootstrap itself), we've already got the intreehooks package to make this possible: https://pypi.org/project/intreehooks/ On Sun, Jan 27, 2019, at 2:26 PM, Chris Jerdonek wrote:

The possibility of doing it in setuptools is discussed extensively in the threads, but a lot of us are going off of half-remembered discussions, so it's probably a good idea to get a central "agenda" of the things we want to resolve and create a new thread for discussion in the discourse. Items I see for the agenda: - The most urgent question for discussion that I see is what should be done /now/. This seems to me like a major issue, and I'm wondering if it would make sense to revert the major change in behavior until we iron out some of the details. It seems like many people in the thread (and people I've spoken to in person) are "solving" this by just pinning pip to < 19.0. - From the setuptools side, I think we don't want to be forced to rush things. setuptools is already a huge bundle of compromises and less-than-ideal defaults because we have so few opportunities to fix the crustier parts of it without introducing enormous code churn. I don't want us to have to put compatibility shims into our main build backend just to unbreak some people today if we can avoid it with a less permanent "band-aid" solution. On 1/27/19 9:39 AM, Thomas Kluyver wrote:

On Sun, 27 Jan 2019 at 14:39, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
I think the rule about the CWD not being on sys.path only applies to loading a proper PEP 517 build backend when that's specified in pyproject.toml.
I'm not entirely sure what you're intending when you refer to a "proper PEP 517 build backend". The setuptools backend is a perfectly acceptable one, the only difference here is that we default to it in pip when the project hasn't specified a backend. The reason we do that is simple - if we don't, we're unlikely to be able to remove the legacy setuptools code any time in the foreseeable future, as there will be essentially no incentive for existing projects to specify a setuptools backend. Also, we'd get basically no testing of the new PEP 517 code, for the same reason. (Yes, it would have been better to test before release, but we've never had any success getting beta testers for pip). We could have made PEP 517 entirely opt-in, but (a) that would have drastically slowed down adoption, and (b) implicit in everything we'd discussed was the assumption that there would be a setuptools backend that was semantically equivalent to running setup.py the way we currently did, so there was no need to. Clearly that assumption turned out to be wrong, and that's where the issues for our users arose. Having said this, I don't have any problem with changing the default backend used by pip. In fact, if the setuptools project don't view it as a goal of their existing backend to replicate setup.py behaviour, then I think it's pretty much essential that we change. Pip needs to default to a PEP 517 backend that behaves the same as pip's legacy code - that's essential for backward compatibility if we're to remove or deprecate the legacy setup.py code path. The only problem is that right now there doesn't appear to be a backend that suits our requirements. Paul Ganssle suggested that setuptools could provide an alternative ("legacy") setuptools backend that preserved full setup.py compatibility (including having the current directory on sys.path). That seems to me to be a reasonable solution, and I appreciate the offer. In practice, the legacy setuptools backend doesn't *have* to be implemented by setuptools itself - it could be a separate project if setuptools don't want to maintain it themselves. But IMO it would be better if the setuptools team are OK with maintaining it.(At a pinch, pip could even implement that backend internally, but adding pip to every build environment seems a bit heavyweight). So in terms of practical resolution of the problems people are hitting since pip 19.0 was released, I don't think there's a significant issue here. We need a new pip release, sure, and someone needs to write the new backend, but I don't think either of those tasks are huge. What is more difficult is the question of whether the PEP should change. As Chris pointed out, the previous discussion ended up saying that the build directory should not be on sys.path, but acknowledged that mandating that might cause issues. So the question now is, are the issues we've seen big enough that we want to change PEP 517 to say that the build directory *should* be on sys.path? We don't have to answer that question urgently (the urgent resolution is with a legacy backend, as noted above) but we do have to decide one way or another. And here the question is whether enforcing projects to adopt the "good practice" of not relying on being able to import the project at build time is worth the negative impact of requiring what appears to be a non-trivial number of projects to change their build code. Or whether there's an "intermediate" option (the discussions Chris linked to included talk of adding another pyproject.toml value which would be a list of directories the frontend should add to sys.path when calling hooks, for example). But that discussion shouldn't be made under a false impression of urgency - it's not on the critical path for fixing pip's immediate issue. Paul

It feels to me that importing from setup.py level is a code smell and should be avoided (I mean what happens if you move the source into src layout, do you now want tools to also append the src folder additionally if exists?). Can't people just inject at the start of setup.py into sys path the cwd if they really want to go with it? (feels superior over pinning pip to 18.0). On Sun, Jan 27, 2019 at 7:48 PM Paul Moore <p.f.moore@gmail.com> wrote:

Yes, of course people can. I think the problem is that common Python installations defaults to adding the cwd into sys.path, so people expect this to “just work”. A PEP 517 not doing it is not intuitive unless you follow Python packaging closely, and most would simply assume pip is broken. (Incidentally, the Windows embedded distribution also does not include the cwd in sys.path by default, and I do see confused people on discussion boards or issue trackers from time to time complaining things don’t work. The embedded distribution’s user base is quite small, however, so this is not really a problem.) -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone

This is more an argument to not use pep-517 unless people explicitly specify the backend, at which point they acknowledge to buy in that the cwd is not on sys path (and they need to alter their packaging code accordingly). On Sun, 27 Jan 2019, 20:31 Tzu-ping Chung <uranusjr@gmail.com wrote:

I guess this depends on how explicit you wan to be. PEP 517 is not enabled unconditionally, only when the project root contains pyproject.toml. The problem is that other projects (unrelated to packaging) take advantage of the file format’s existence, resulting in people “buy in” the situation unknowingly. Things are already explicit from pip’s point of view, but (unanticipated?) third-party usages of the same spec unrelated to PEP 517 muddles the situation. -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone

On Jan 27, 2019, at 3:58 PM, Tzu-ping Chung <uranusjr@gmail.com> wrote:
I guess this depends on how explicit you wan to be. PEP 517 is not enabled unconditionally, only when the project root contains pyproject.toml. The problem is that other projects (unrelated to packaging) take advantage of the file format’s existence, resulting in people “buy in” the situation unknowingly. Things are already explicit from pip’s point of view, but (unanticipated?) third-party usages of the same spec unrelated to PEP 517 muddles the situation
The group of engineers that I work with accidentally encountered PEP-517 by using pyproject.toml as an alternative to setup.cfg for tool configuration. In the specific instance it was the black utility[1] but it can happen with any other utility with similar recommendations. I agree with pip's view that the presence of pyproject.toml means that you are *explicitly* announcing compliance with PEP-517. The unexpected part was that the setuptools build-system shim was incomplete. This will be addressed shortly if Paul Ganssle has anything to do with it (thanks Paul!). I look at this failure as a chance to educate other developers on the impact of introducing something new without understanding the reason for doing so. For example, if you are already tied to setuptools and using setup.cfg to configure tools, then introducing a new configuration mechanism for a single tool is *not* something that should be done lightly. cheers, dave. [1]: https://github.com/ambv/black#configuration-format -- "There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton

On 1/29/19 7:46 AM, David Shawley wrote:
Thanks for the shout out, and I really do appreciate the recognition, but I would like to clarify that if anything I have /delayed/ getting this fixed with my insistence that the semantics should be different between setuptools.build_meta and setup.py. I think there are already some useful features of this distinction, but it's certainly true that if we had simply modified the path in the existing build backend, this would be fixed now (with the release of the latest setuptools) instead of necessitating a new pip release. Anyway, even if I was right to insist on this distinction, I don't want to take the credit for fixing this when a lot of people put in a lot of effort discussing this and working out the right path forward, which is very tough in packaging, where people want many new and useful features, but they're also weary of all the churn in their build and deployment system. That said, I hope this doesn't sound like I'm complaining that you complimented my work. I would hate to make you think that someone could be bothered by the fact that you thanked an open source maintainer. Best, Paul

On Sun, Jan 27, 2019, at 7:47 PM, Paul Moore wrote:
I don't see that this issue is an argument for changing the PEP. It seems like a problem with the backend that should be supporting legacy use cases. The rule about the frontend not using the CWD to load the backend doesn't constrain what the backend does once loaded. From my perspective, the original arguments about the CWD still stand, but in addition PEP 517 is out now and there are tools implementing it, so it would be more disruptive to make changes. So at least for now, I'm -1 on changing the rule about the CWD. Thomas

On Mon, 28 Jan 2019 at 09:04, Paul Moore <p.f.moore@gmail.com> wrote:
Yep, I think letting the backend decide whether or not it wants to include the build directory on sys.path remains the right way to go. Thomas's https://github.com/takluyver/intreehooks then addresses opting in to that in the general case, while Paul Ganssle's proposed build_meta_legacy backend in https://github.com/pypa/setuptools/pull/1652 addresses it for the specific case of setup.py execution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I haven't read those discussions in full (sorry to be lazy, but there's quite a lot there), but my initial take is that the rule doesn't apply to running setup.py scripts, where there's a greater need for backward compatibility. I think the rule about the CWD not being on sys.path only applies to loading a proper PEP 517 build backend when that's specified in pyproject.toml. Furthermore, if the build backend then wants to run some code with the CWD on sys.path, I think it's free to do so, so this could be implemented in the setuptools backend. The rule is only for how the frontend loads the backend, not what the backend does afterwards. Where a real PEP 517 backend needs to be loaded from the CWD (e.g. for Flit to bootstrap itself), we've already got the intreehooks package to make this possible: https://pypi.org/project/intreehooks/ On Sun, Jan 27, 2019, at 2:26 PM, Chris Jerdonek wrote:

The possibility of doing it in setuptools is discussed extensively in the threads, but a lot of us are going off of half-remembered discussions, so it's probably a good idea to get a central "agenda" of the things we want to resolve and create a new thread for discussion in the discourse. Items I see for the agenda: - The most urgent question for discussion that I see is what should be done /now/. This seems to me like a major issue, and I'm wondering if it would make sense to revert the major change in behavior until we iron out some of the details. It seems like many people in the thread (and people I've spoken to in person) are "solving" this by just pinning pip to < 19.0. - From the setuptools side, I think we don't want to be forced to rush things. setuptools is already a huge bundle of compromises and less-than-ideal defaults because we have so few opportunities to fix the crustier parts of it without introducing enormous code churn. I don't want us to have to put compatibility shims into our main build backend just to unbreak some people today if we can avoid it with a less permanent "band-aid" solution. On 1/27/19 9:39 AM, Thomas Kluyver wrote:

On Sun, 27 Jan 2019 at 14:39, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
I think the rule about the CWD not being on sys.path only applies to loading a proper PEP 517 build backend when that's specified in pyproject.toml.
I'm not entirely sure what you're intending when you refer to a "proper PEP 517 build backend". The setuptools backend is a perfectly acceptable one, the only difference here is that we default to it in pip when the project hasn't specified a backend. The reason we do that is simple - if we don't, we're unlikely to be able to remove the legacy setuptools code any time in the foreseeable future, as there will be essentially no incentive for existing projects to specify a setuptools backend. Also, we'd get basically no testing of the new PEP 517 code, for the same reason. (Yes, it would have been better to test before release, but we've never had any success getting beta testers for pip). We could have made PEP 517 entirely opt-in, but (a) that would have drastically slowed down adoption, and (b) implicit in everything we'd discussed was the assumption that there would be a setuptools backend that was semantically equivalent to running setup.py the way we currently did, so there was no need to. Clearly that assumption turned out to be wrong, and that's where the issues for our users arose. Having said this, I don't have any problem with changing the default backend used by pip. In fact, if the setuptools project don't view it as a goal of their existing backend to replicate setup.py behaviour, then I think it's pretty much essential that we change. Pip needs to default to a PEP 517 backend that behaves the same as pip's legacy code - that's essential for backward compatibility if we're to remove or deprecate the legacy setup.py code path. The only problem is that right now there doesn't appear to be a backend that suits our requirements. Paul Ganssle suggested that setuptools could provide an alternative ("legacy") setuptools backend that preserved full setup.py compatibility (including having the current directory on sys.path). That seems to me to be a reasonable solution, and I appreciate the offer. In practice, the legacy setuptools backend doesn't *have* to be implemented by setuptools itself - it could be a separate project if setuptools don't want to maintain it themselves. But IMO it would be better if the setuptools team are OK with maintaining it.(At a pinch, pip could even implement that backend internally, but adding pip to every build environment seems a bit heavyweight). So in terms of practical resolution of the problems people are hitting since pip 19.0 was released, I don't think there's a significant issue here. We need a new pip release, sure, and someone needs to write the new backend, but I don't think either of those tasks are huge. What is more difficult is the question of whether the PEP should change. As Chris pointed out, the previous discussion ended up saying that the build directory should not be on sys.path, but acknowledged that mandating that might cause issues. So the question now is, are the issues we've seen big enough that we want to change PEP 517 to say that the build directory *should* be on sys.path? We don't have to answer that question urgently (the urgent resolution is with a legacy backend, as noted above) but we do have to decide one way or another. And here the question is whether enforcing projects to adopt the "good practice" of not relying on being able to import the project at build time is worth the negative impact of requiring what appears to be a non-trivial number of projects to change their build code. Or whether there's an "intermediate" option (the discussions Chris linked to included talk of adding another pyproject.toml value which would be a list of directories the frontend should add to sys.path when calling hooks, for example). But that discussion shouldn't be made under a false impression of urgency - it's not on the critical path for fixing pip's immediate issue. Paul

It feels to me that importing from setup.py level is a code smell and should be avoided (I mean what happens if you move the source into src layout, do you now want tools to also append the src folder additionally if exists?). Can't people just inject at the start of setup.py into sys path the cwd if they really want to go with it? (feels superior over pinning pip to 18.0). On Sun, Jan 27, 2019 at 7:48 PM Paul Moore <p.f.moore@gmail.com> wrote:

Yes, of course people can. I think the problem is that common Python installations defaults to adding the cwd into sys.path, so people expect this to “just work”. A PEP 517 not doing it is not intuitive unless you follow Python packaging closely, and most would simply assume pip is broken. (Incidentally, the Windows embedded distribution also does not include the cwd in sys.path by default, and I do see confused people on discussion boards or issue trackers from time to time complaining things don’t work. The embedded distribution’s user base is quite small, however, so this is not really a problem.) -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone

This is more an argument to not use pep-517 unless people explicitly specify the backend, at which point they acknowledge to buy in that the cwd is not on sys path (and they need to alter their packaging code accordingly). On Sun, 27 Jan 2019, 20:31 Tzu-ping Chung <uranusjr@gmail.com wrote:

I guess this depends on how explicit you wan to be. PEP 517 is not enabled unconditionally, only when the project root contains pyproject.toml. The problem is that other projects (unrelated to packaging) take advantage of the file format’s existence, resulting in people “buy in” the situation unknowingly. Things are already explicit from pip’s point of view, but (unanticipated?) third-party usages of the same spec unrelated to PEP 517 muddles the situation. -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone

On Jan 27, 2019, at 3:58 PM, Tzu-ping Chung <uranusjr@gmail.com> wrote:
I guess this depends on how explicit you wan to be. PEP 517 is not enabled unconditionally, only when the project root contains pyproject.toml. The problem is that other projects (unrelated to packaging) take advantage of the file format’s existence, resulting in people “buy in” the situation unknowingly. Things are already explicit from pip’s point of view, but (unanticipated?) third-party usages of the same spec unrelated to PEP 517 muddles the situation
The group of engineers that I work with accidentally encountered PEP-517 by using pyproject.toml as an alternative to setup.cfg for tool configuration. In the specific instance it was the black utility[1] but it can happen with any other utility with similar recommendations. I agree with pip's view that the presence of pyproject.toml means that you are *explicitly* announcing compliance with PEP-517. The unexpected part was that the setuptools build-system shim was incomplete. This will be addressed shortly if Paul Ganssle has anything to do with it (thanks Paul!). I look at this failure as a chance to educate other developers on the impact of introducing something new without understanding the reason for doing so. For example, if you are already tied to setuptools and using setup.cfg to configure tools, then introducing a new configuration mechanism for a single tool is *not* something that should be done lightly. cheers, dave. [1]: https://github.com/ambv/black#configuration-format -- "There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton

On 1/29/19 7:46 AM, David Shawley wrote:
Thanks for the shout out, and I really do appreciate the recognition, but I would like to clarify that if anything I have /delayed/ getting this fixed with my insistence that the semantics should be different between setuptools.build_meta and setup.py. I think there are already some useful features of this distinction, but it's certainly true that if we had simply modified the path in the existing build backend, this would be fixed now (with the release of the latest setuptools) instead of necessitating a new pip release. Anyway, even if I was right to insist on this distinction, I don't want to take the credit for fixing this when a lot of people put in a lot of effort discussing this and working out the right path forward, which is very tough in packaging, where people want many new and useful features, but they're also weary of all the churn in their build and deployment system. That said, I hope this doesn't sound like I'm complaining that you complimented my work. I would hate to make you think that someone could be bothered by the fact that you thanked an open source maintainer. Best, Paul

On Sun, Jan 27, 2019, at 7:47 PM, Paul Moore wrote:
I don't see that this issue is an argument for changing the PEP. It seems like a problem with the backend that should be supporting legacy use cases. The rule about the frontend not using the CWD to load the backend doesn't constrain what the backend does once loaded. From my perspective, the original arguments about the CWD still stand, but in addition PEP 517 is out now and there are tools implementing it, so it would be more disruptive to make changes. So at least for now, I'm -1 on changing the rule about the CWD. Thomas

On Mon, 28 Jan 2019 at 09:04, Paul Moore <p.f.moore@gmail.com> wrote:
Yep, I think letting the backend decide whether or not it wants to include the build directory on sys.path remains the right way to go. Thomas's https://github.com/takluyver/intreehooks then addresses opting in to that in the general case, while Paul Ganssle's proposed build_meta_legacy backend in https://github.com/pypa/setuptools/pull/1652 addresses it for the specific case of setup.py execution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (9)
-
Bernat Gabor
-
Chris Jerdonek
-
David Shawley
-
Nick Coghlan
-
Paul Ganssle
-
Paul Moore
-
Pradyun Gedam
-
Thomas Kluyver
-
Tzu-ping Chung