Oh, BDFL, will you settle whether we should raise an exception or return Notimplemented? On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote: xoviat wrote:
I mean how is opening a file different than attempting to build an sdist?
1. Opening a file is a very common operation. 2. Most file opens are expected to succeed, and if one doesn't, the appropriate place to deal with that is almost never at the site of the open call, but somewhere further up. In contrast, there's probably only about one place in any given frontend where the backend's build_sdist method gets invoked, and it's unlikely the ability to let a not-implemented exception bubble up from that point would be of great use. -- Greg _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Looks like Nick is on the losing side here, but we shall see. On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
Oh, BDFL, will you settle whether we should raise an exception or return Notimplemented?
On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
xoviat wrote:
I mean how is opening a file different than attempting to build an sdist?
1. Opening a file is a very common operation.
2. Most file opens are expected to succeed, and if one doesn't, the appropriate place to deal with that is almost never at the site of the open call, but somewhere further up.
In contrast, there's probably only about one place in any given frontend where the backend's build_sdist method gets invoked, and it's unlikely the ability to let a not-implemented exception bubble up from that point would be of great use.
-- Greg
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I don't have any context here, but "return NotImplemented" is a very narrow idiom intended only for binary operators (e.g. __add__) when the interpreter should give the other operand a chance (e.g. __radd__) or use a default implementation. In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.) On Fri, Aug 25, 2017 at 6:53 PM, xoviat <xoviat@gmail.com> wrote:
Oh, BDFL, will you settle whether we should raise an exception or return Notimplemented?
On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
xoviat wrote:
I mean how is opening a file different than attempting to build an sdist?
1. Opening a file is a very common operation.
2. Most file opens are expected to succeed, and if one doesn't, the appropriate place to deal with that is almost never at the site of the open call, but somewhere further up.
In contrast, there's probably only about one place in any given frontend where the backend's build_sdist method gets invoked, and it's unlikely the ability to let a not-implemented exception bubble up from that point would be of great use.
-- Greg
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- --Guido van Rossum (python.org/~guido)
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None. Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do. Paul
Is everyone on board with that? On Aug 26, 2017 4:29 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do. Paul
I'm bored with that On Sat, Aug 26, 2017, 11:59 xoviat <xoviat@gmail.com> wrote:
Is everyone on board with that?
On Aug 26, 2017 4:29 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do. Paul
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[removed Guido from CC] On Aug 26, 2017 02:29, "Paul Moore" <p.f.moore@gmail.com> wrote: On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None. Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do. Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes. -n
The current PEP requires that build_sdist return the basename of the sdist. So forgetting a return statement is not an option unless people really don't read the PEP. On Aug 26, 2017 2:18 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
[removed Guido from CC]
On Aug 26, 2017 02:29, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do.
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
-n
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 26 August 2017 at 20:17, Nathaniel Smith <njs@pobox.com> wrote:
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Well, we've had an extensive discussion about how frontends need to trust backends to get things right. I don't really see it as reasonable to now argue that backends might "forget" to return the right value - they might just as well "forget" to properly isolate builds... As regards an explicit description of what went wrong, why can't we just use the same reporting methods that we will for any other build issue (backends simply report the problem on stdout/stderr)? I don't see why the backend has to package up its error information and send it to the frontend to report, when we already have a perfectly effective way for backends to report errors and/or warnings to the user. If you're worried that the frontend might suppress the information (maybe because it's planning on falling back to a direct wheel build) then isn't that just the converse - backends need to trust frontends to do the right thing? It's hard to come up with convincing counterarguments to hypothetical concerns. Do you have a specific scenario that worries you? I see "return the path, or None indicating you couldn't build the sdist" as a nice, simple solution. If there's a case we know of right now where it's *too* simple, then let's understand the specifics - otherwise, why not defer worrying about it for now? We can revise the PEP later if experience shows we missed something. Paul
On Sat, Aug 26, 2017 at 12:54 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 20:17, Nathaniel Smith <njs@pobox.com> wrote:
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Well, we've had an extensive discussion about how frontends need to trust backends to get things right. I don't really see it as reasonable to now argue that backends might "forget" to return the right value - they might just as well "forget" to properly isolate builds...
It's not about division of responsibilities, it's about handling errors gracefully when they happen. There are three bins: - creating an sdist succeeded - creating an sdist failed for expected reasons, and a clever frontend might be able to handle the problem automatically if it understands what the problem is (sdist creation isn't supported in this case) and understands its goals (just trying to build a wheel really, so the sdist isn't crucial) - creating an sdist failed for unexpected reasons, that need a human to sort out (due to a broken system, or bugs – hey, they happen – or ...) The whole discussion has been about how we can most reliably distinguish between the second and third categories, and give good error messages for the third category. The argument for NotImplemented is that it avoids cases where some internal call raises NotImplementedError and it "leaks out" accidentally, causing a unexpected error to be incorrectly treated as expected error -- we don't want pip to be hiding real bugs in backend code. The argument for NotImplementedError is that it produces better error messages on buggy frontends. 'return None' is kind of the worst of both worlds, in that it's an easy thing to return accidentally, and it gives confusing error messages if the frontend fails to handle it properly. (Even more confusing, actually, because 'NoneType object has no attribute ...' is even harder to track down than 'NotImplementedType object has no attribute ...'.)
As regards an explicit description of what went wrong, why can't we just use the same reporting methods that we will for any other build issue (backends simply report the problem on stdout/stderr)? I don't see why the backend has to package up its error information and send it to the frontend to report, when we already have a perfectly effective way for backends to report errors and/or warnings to the user. If you're worried that the frontend might suppress the information (maybe because it's planning on falling back to a direct wheel build) then isn't that just the converse - backends need to trust frontends to do the right thing?
What I mean is more, if you're some random user and you see this in a build backend, what do you guess it means? def get_requires_for_build_sdist(config_settings=None): return None Now how about these? def get_requires_for_build_sdist(config_settings=None): return NotImplemented def get_requires_for_build_sdist(config_settings=None): raise NotImplementedError def get_requires_for_build_sdist(config_settings=None): raise SdistBuildNotSupported I mean, obviously return None will work. Basically anything that's different from "return a list or string" will work :-). That's what makes this a bikeshed topic, and I still think we're mostly just spinning our wheels here until Nick and Donald have a chance to hash something out that they both can agree on. But I really don't see any advantages to 'return None' compared to the other options that have been discussed -n -- Nathaniel J. Smith -- https://vorpus.org
Nathaniel: We're not talking about signaling failure in get_requires* we're talking about signaling failure in build*. On Aug 26, 2017 3:42 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 12:54 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 20:17, Nathaniel Smith <njs@pobox.com> wrote:
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Well, we've had an extensive discussion about how frontends need to trust backends to get things right. I don't really see it as reasonable to now argue that backends might "forget" to return the right value - they might just as well "forget" to properly isolate builds...
It's not about division of responsibilities, it's about handling errors gracefully when they happen. There are three bins:
- creating an sdist succeeded - creating an sdist failed for expected reasons, and a clever frontend might be able to handle the problem automatically if it understands what the problem is (sdist creation isn't supported in this case) and understands its goals (just trying to build a wheel really, so the sdist isn't crucial) - creating an sdist failed for unexpected reasons, that need a human to sort out (due to a broken system, or bugs – hey, they happen – or ...)
The whole discussion has been about how we can most reliably distinguish between the second and third categories, and give good error messages for the third category. The argument for NotImplemented is that it avoids cases where some internal call raises NotImplementedError and it "leaks out" accidentally, causing a unexpected error to be incorrectly treated as expected error -- we don't want pip to be hiding real bugs in backend code. The argument for NotImplementedError is that it produces better error messages on buggy frontends. 'return None' is kind of the worst of both worlds, in that it's an easy thing to return accidentally, and it gives confusing error messages if the frontend fails to handle it properly. (Even more confusing, actually, because 'NoneType object has no attribute ...' is even harder to track down than 'NotImplementedType object has no attribute ...'.)
As regards an explicit description of what went wrong, why can't we just use the same reporting methods that we will for any other build issue (backends simply report the problem on stdout/stderr)? I don't see why the backend has to package up its error information and send it to the frontend to report, when we already have a perfectly effective way for backends to report errors and/or warnings to the user. If you're worried that the frontend might suppress the information (maybe because it's planning on falling back to a direct wheel build) then isn't that just the converse - backends need to trust frontends to do the right thing?
What I mean is more, if you're some random user and you see this in a build backend, what do you guess it means?
def get_requires_for_build_sdist(config_settings=None): return None
Now how about these?
def get_requires_for_build_sdist(config_settings=None): return NotImplemented
def get_requires_for_build_sdist(config_settings=None): raise NotImplementedError
def get_requires_for_build_sdist(config_settings=None): raise SdistBuildNotSupported
I mean, obviously return None will work. Basically anything that's different from "return a list or string" will work :-). That's what makes this a bikeshed topic, and I still think we're mostly just spinning our wheels here until Nick and Donald have a chance to hash something out that they both can agree on. But I really don't see any advantages to 'return None' compared to the other options that have been discussed
-n
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Nathaniel Smith wrote:
- creating an sdist failed for unexpected reasons, that need a human to sort out (due to a broken system, or bugs – hey, they happen – or ...)
I think that should still be reported via an exception. Returning None should only be for the specific case that the backend doesn't support the requested operation. -- Greg
On Sun, Aug 27, 2017 at 4:27 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nathaniel Smith wrote:
- creating an sdist failed for unexpected reasons, that need a human to sort out (due to a broken system, or bugs – hey, they happen – or ...)
I think that should still be reported via an exception. Returning None should only be for the specific case that the backend doesn't support the requested operation.
Well, you can't exactly say "if your code is buggy, then you should signal that by doing this well defined thing" :-). One of my objections to None is that it's very easy to return accidentally, i.e., buggy code *will* sometimes return None no matter what the spec says. -n -- Nathaniel J. Smith -- https://vorpus.org
If pip does uses build_wheel directly, as Paul now prefers, I think we can leave the NotImplemented/Error/None question for a later date. We only want some way to signal "I can't do that" because a frontend might try sdist->wheel with a fallback to making a wheel directly. If no frontend is actually planning to do that, we can leave specifying it until a frontend wants it. Donald, what do you think? IIRC, you were most keen on going sdist->wheel where possible, and I don't think you've commented on Paul's suggestion yet (apologies if I've overlooked a response). Thomas On Mon, Aug 28, 2017, at 12:47 AM, Nathaniel Smith wrote:
On Sun, Aug 27, 2017 at 4:27 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nathaniel Smith wrote:
- creating an sdist failed for unexpected reasons, that need a human to sort out (due to a broken system, or bugs – hey, they happen – or ...)
I think that should still be reported via an exception. Returning None should only be for the specific case that the backend doesn't support the requested operation.
Well, you can't exactly say "if your code is buggy, then you should signal that by doing this well defined thing" :-). One of my objections to None is that it's very easy to return accidentally, i.e., buggy code *will* sometimes return None no matter what the spec says.
-n
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Aug 28, 2017, at 4:43 AM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
If pip does uses build_wheel directly, as Paul now prefers, I think we can leave the NotImplemented/Error/None question for a later date. We only want some way to signal "I can't do that" because a frontend might try sdist->wheel with a fallback to making a wheel directly. If no frontend is actually planning to do that, we can leave specifying it until a frontend wants it.
Donald, what do you think? IIRC, you were most keen on going sdist->wheel where possible, and I don't think you've commented on Paul's suggestion yet (apologies if I've overlooked a response).
I still think it should, and prefer pip to attempt to build a sdist prior to building a wheel when we’re coming from a VCS directory. I think that is going to be the most robust mechanism with the least amount of surprising behavior for end users. I also think it is important to define this fallback now, because doing it now means we have flexibility to adjust the strategy that frontends use without going back through the PEP process and waiting on backends to update to the latest version of the PEP. I am okay with an exception if it’s a specific exception, the idea of having an exception class exported as part of the API is fine with me. I think it’s still a worse API to use an exception for a non-exceptional case, but if it’s a specific exception it at least prevents bubbling up unrelated errors and having them be treated as asking for the fallback behavior if possible. — Donald Stufft
On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft <donald@stufft.io> wrote:
Donald, what do you think? IIRC, you were most keen on going sdist->wheel where possible, and I don't think you've commented on Paul's suggestion yet (apologies if I've overlooked a response).
I still think it should, and prefer pip to attempt to build a sdist prior to building a wheel when we’re coming from a VCS directory. I think that is going to be the most robust mechanism with the least amount of surprising behavior for end users.
why? even as it stands, doesn't setuptools dump everything into the build dir anyway? The "Creating of an sdist" really seems like a build-system problem, not a package manager problem to me.
"when we’re coming from a VCS directory."
whether this is a VCS directory or some other source seems to me like something pip should not need to know... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Aug 28, 2017, at 12:29 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft <donald@stufft.io <mailto:donald@stufft.io>> wrote:
Donald, what do you think? IIRC, you were most keen on going sdist->wheel where possible, and I don't think you've commented on Paul's suggestion yet (apologies if I've overlooked a response).
I still think it should, and prefer pip to attempt to build a sdist prior to building a wheel when we’re coming from a VCS directory. I think that is going to be the most robust mechanism with the least amount of surprising behavior for end users.
why? even as it stands, doesn't setuptools dump everything into the build dir anyway? The "Creating of an sdist" really seems like a build-system problem, not a package manager problem to me.
"when we’re coming from a VCS directory."
whether this is a VCS directory or some other source seems to me like something pip should not need to know...
Differences between what files are in a sdist and what files are in a VCS directory will lead to different behaviors on install which makes ``pip install .`` and ``build-a-sdist && pip install the-sdist.tar.gz`` behave differently. Attempting to funnel everything through the same VCS -> sdist -> wheel path makes it less likely for these kinds of issues to occur. This is not a fully resolvable problem, and it is going to happen basically anytime you have two independent lists of what files get put into a sdist and what files get installed. It is not unique to setuptools nor is it a result of the way distutils/setuptools works— they only expose it more obviously because of their relevant APIs. Infact, both enscons and flit have this same problem (although flit has gone to some length to minimize the issue, so it’s somewhat hard, but not impossible, to actually trigger it). — Donald Stufft
Personally, my plan for the setuptools backend will be to build a source distribution (essentially using the command-line interface), extract into a tmpdir, and then build a wheel (essentially using the command line interface). So if pip calls build_sdist and then build_wheel, there will be two source distributions built (one by pip and one by setuptools) before building a wheel. There is not another way to do this that will comply with the specification because setuptools cannot currently be trusted to build a wheel directly. 2017-08-28 12:15 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Aug 28, 2017, at 12:29 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft <donald@stufft.io> wrote:
Donald, what do you think? IIRC, you were most keen on going sdist->wheel where possible, and I don't think you've commented on Paul's suggestion yet (apologies if I've overlooked a response).
I still think it should, and prefer pip to attempt to build a sdist prior to building a wheel when we’re coming from a VCS directory. I think that is going to be the most robust mechanism with the least amount of surprising behavior for end users.
why? even as it stands, doesn't setuptools dump everything into the build dir anyway? The "Creating of an sdist" really seems like a build-system problem, not a package manager problem to me.
"when we’re coming from a VCS directory."
whether this is a VCS directory or some other source seems to me like something pip should not need to know...
Differences between what files are in a sdist and what files are in a VCS directory will lead to different behaviors on install which makes ``pip install .`` and ``build-a-sdist && pip install the-sdist.tar.gz`` behave differently. Attempting to funnel everything through the same VCS -> sdist -> wheel path makes it less likely for these kinds of issues to occur.
This is not a fully resolvable problem, and it is going to happen basically anytime you have two independent lists of what files get put into a sdist and what files get installed. It is not unique to setuptools nor is it a result of the way distutils/setuptools works— they only expose it more obviously because of their relevant APIs. Infact, both enscons and flit have this same problem (although flit has gone to some length to minimize the issue, so it’s somewhat hard, but not impossible, to actually trigger it).
— Donald Stufft
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 28 August 2017 at 19:50, xoviat <xoviat@gmail.com> wrote:
Personally, my plan for the setuptools backend will be to build a source distribution (essentially using the command-line interface), extract into a tmpdir, and then build a wheel (essentially using the command line interface). So if pip calls build_sdist and then build_wheel, there will be two source distributions built (one by pip and one by setuptools) before building a wheel. There is not another way to do this that will comply with the specification because setuptools cannot currently be trusted to build a wheel directly.
This is precisely the question - whose responsibility is it to ensure that sdist->wheel vs build_wheel equivalence is maintained? The PEP says it's the backend's responsibility, so pip shouldn't need to do anything. But, we're currently having debates about "return None" which hinge on "if backends have bugs, return None might not work well". Sure - but if backends have bugs, they may not maintain wheel equivalence. I find this discrepancy immensely frustrating, as in one breath we're being told "trust the backend", and in the next we're being told "the backend may have bugs". To that extent, I'm with Donald - pip going sdist->wheel protects the user against the known-to-be-an-issue bug that the backend doesn't ensure wheel equivalence. But if we go with that, conscientious backend developers like xoviat suffer, because they take extra steps to do things correctly and as a result suffer bad performance. I've said I'm happy to trust the backend. But I'm starting to wonder if I should change that position: * Chris Barker has pointed out that backends have no reason to support sdists now. * Nathaniel is pushing a means of notifying "I can't build a sdist" that protects against backends accidentally not following the spec. * Donald has reservations (I don't fully understand them, except in the broad sense of "he wants pip to protect the user from problem cases" - which basically means "he doesn't trust the backend" - but in general I think they are valid) Should we trust the backend or not? Backends *will* have bugs - part of "trust the backend" is simply telling the user that the problem behaviour they found is not pip's issue and should be reported to the backend. Is that a sufficiently bad user experience that we should try to improve it (experience with setuptools says that it is - why are we assuming that developers of new backends will be so much more conscientious and careful than the setuptools developers? *I* certainly don't feel that the setuptools developers are unusually bad - quite the opposite!) Having a more robust means of saying "I can't build a sdist" than "return None" is protecting the user from issues with the backend. So is building via sdist. Donald's position is consistent - build via sdist and use a robust return. So was mine - build_wheel and return None. I don't want the PEP to end up defining an inconsistent position. And I'm starting to feel that conceding on build_wheel (and the "trust the backend" principle) was not seen in that context, but as conceding just the one item, leaving other aspects of trusting the backend open to debate. Maybe we go fully to Nick's proposal that we don't mandate any sort of consistency constraints in the PEP. That would mean pip *has* to go sdist->wheel (because pip does need consistent behaviour), and xoviat's setuptools backend can skip building a sdist on the way to building a wheel. It also means that the build_wheel hook is essentially unreliable in isolation, and that all frontends will likely have to do the build_sdist->build_wheel with fallback to inplace build_wheel dance that pip does. But we expect many less frontends than backends, so maybe that's the right trade-off? Paul
On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote:
Maybe we go fully to Nick's proposal that we don't mandate any sort of consistency constraints in the PEP. That would mean pip *has* to go sdist->wheel (because pip does need consistent behaviour), and xoviat's setuptools backend can skip building a sdist on the way to building a wheel. It also means that the build_wheel hook is essentially unreliable in isolation, and that all frontends will likely have to do the build_sdist->build_wheel with fallback to inplace build_wheel dance that pip does. But we expect many less frontends than backends, so maybe that's the right trade-off?
I've mentioned this before, but I have little faith in our ability to predict that one side of an interface will be far more numerous than the other. This is partly from my experience with Jupyter, where we got that prediction completely wrong. But I also think it's quite plausible that many frontend tools will want to use this interface (to argue against myself, most will be Python tools, so they could theoretically share a common wrapper module - but I have some doubts about whether they will). I'm not saying we shouldn't do this - as Donald is firmly in favour and you're wavering, it seems the easier option to wrap the discussion up. But I'm suspicious of the rationale that *there will be fewer frontends so they should have more responsibility*. Thomas
On Aug 28, 2017, at 3:32 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote:
Maybe we go fully to Nick's proposal that we don't mandate any sort of consistency constraints in the PEP. That would mean pip *has* to go sdist->wheel (because pip does need consistent behaviour), and xoviat's setuptools backend can skip building a sdist on the way to building a wheel. It also means that the build_wheel hook is essentially unreliable in isolation, and that all frontends will likely have to do the build_sdist->build_wheel with fallback to inplace build_wheel dance that pip does. But we expect many less frontends than backends, so maybe that's the right trade-off?
I've mentioned this before, but I have little faith in our ability to predict that one side of an interface will be far more numerous than the other. This is partly from my experience with Jupyter, where we got that prediction completely wrong. But I also think it's quite plausible that many frontend tools will want to use this interface (to argue against myself, most will be Python tools, so they could theoretically share a common wrapper module - but I have some doubts about whether they will).
I'm not saying we shouldn't do this - as Donald is firmly in favour and you're wavering, it seems the easier option to wrap the discussion up. But I'm suspicious of the rationale that *there will be fewer frontends so they should have more responsibility*.
I don’t see anything wrong with saying *both* sides should be doing this when they can. Backends should attempt to be consistent where possible— while recognizing that is not 100% possible in every situation (for instance, flit when you’re in an archive from github.com <http://github.com/> w/o .git). Likewise front ends should decide how important that is to them, which can range from “not important at all, YOLO” that just blindly calls build_wheel, to “We’d like it, but we aren’t going to mandate it” that calls build_sdist and falls back to build_wheel, and “We think this is a hard requirement” that calls build_sdist and fails if it can’t. My proposal isn’t exactly that I don’t trust the backends, it is that we don’t really know what the landscape is going to look at. Thus this proposal allows us to implement all three possible options without changing anything without mandating something that isn’t feasible (all backends must always be 100% consistent all the time). It would then be my desire that pip starts off with a pretty safe middle ground, try to build a sdist and fall back to directly building wheels that (A) helps to prevent issues where backends may have bugs or inadvertently are violating the ideal of a consistent build and (B) succeeds in most cases, even if (A) can’t be satisfied. Since we have flexibility we can then watch what happens, and if we find that a lot of problems are cropping up from inconsistent builds, then we can start talking about moving to mandating going via sdist OR if we find that the crop of build tools do a really good job and we think that we don’t really need the sanity check, migrate to just going straight to wheel. I think this is pretty important too, because how you want to handle not being able to create a sdist is going to be fairly task specific. Obviously if you can’t produce a sdist for a hypothetical ``twine sdist`` or ``pip sdist`` command, that is going to be a hard failure, but for ``pip install`` or ``pip wheel``, then there is a pretty reasonable argument that a fallback or straight to wheel is the correct answer. I also believe it is fundamentally impossible for the backends to guarantee consistency if they have a separate list for what gets installed vs what gets put into a sdist without literally building a sdist (or something similar)— and as I understand it one of the issues with that is that the tools don’t want to mandate being able to do that same logic in all situations. So I don’t think it works to say “You *must* ensure a consistent output, and I think the only thing we can do is say that you *SHOULD* try to be consistent, and leave it up to front ends to decide how seriously they take that as a requirement. — Donald Stufft
On 28 August 2017 at 20:47, Donald Stufft <donald@stufft.io> wrote:
I also believe it is fundamentally impossible for the backends to guarantee consistency if they have a separate list for what gets installed vs what gets put into a sdist without literally building a sdist (or something similar)— and as I understand it one of the issues with that is that the tools don’t want to mandate being able to do that same logic in all situations.
And yet that's precisely what xoviat is intending to do, based on what the PEP says is needed. Are you saying he shouldn't? And the setuptools should simply expose the same consistency problems we've been dealing with until now? And the solution is for pip to switch to sdist->wheel build as we've always planned in order to resolve this issue? And any other future frontends will have to go through the same cycle that pip did? In which case, isn't the issue here that we've failed to argue sufficiently persuasively that all backends must provide a non-optional build_sdist hook, so we can do that cleanly?
So I don’t think it works to say “You *must* ensure a consistent output, and I think the only thing we can do is say that you *SHOULD* try to be consistent, and leave it up to front ends to decide how seriously they take that as a requirement.
But how do frontends decide that? They have to base the decision on experience with backends. And it looks like the current crop of backends are split down the middle. Remember that if pip decides to go sdist->wheel, then that strongly motivates xoviat to abandon his current approach to ensure consistency, as it's duplicating the work pip will do. At which point the setuptools backend can't be assumed to be consistent, and the flit backend must be assumed to be (because we can't guarantee it can make a sdist). Paul
On Aug 28, 2017, at 3:58 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 28 August 2017 at 20:47, Donald Stufft <donald@stufft.io> wrote:
I also believe it is fundamentally impossible for the backends to guarantee consistency if they have a separate list for what gets installed vs what gets put into a sdist without literally building a sdist (or something similar)— and as I understand it one of the issues with that is that the tools don’t want to mandate being able to do that same logic in all situations.
And yet that's precisely what xoviat is intending to do, based on what the PEP says is needed. Are you saying he shouldn't? And the setuptools should simply expose the same consistency problems we've been dealing with until now? And the solution is for pip to switch to sdist->wheel build as we've always planned in order to resolve this issue? And any other future frontends will have to go through the same cycle that pip did?
I had some non-OSS stuff come up so I lost track of the discussion part way through so maybe I missed this recommendation, but I think that is going to be a bad path to go down because it is intermixing things in a way that I feel is going to result in a less optimal outcome. The problem is the backend doesn’t really know why we’re asking it to build a wheel to know whether we care or not about consistency guarantees. Future front ends can make their own decisions as it makes sense for their workflow. Something like Debian might choose to skip the sdist option (when they go straight from a VCS archive or w/e) because they have a human being sitting there ensuring that the install looks correct, and they are unlikely to want to go through sdist.
In which case, isn't the issue here that we've failed to argue sufficiently persuasively that all backends must provide a non-optional build_sdist hook, so we can do that cleanly?
I’m not sure what you mean by non-optional— Do you mean a build_Sdist hook that does not have a “not implemented” flag? That would probably be my preference, but I’m happy to compromise on the not implemented return value/exception/whatever as a middle ground. Honestly this PEP has been drawn out enough with enough hypotheticals and corner cases that one of the main things I’m trying to optimize for is flexibility so we can handle different possible scenarios without going through this again.
So I don’t think it works to say “You *must* ensure a consistent output, and I think the only thing we can do is say that you *SHOULD* try to be consistent, and leave it up to front ends to decide how seriously they take that as a requirement.
But how do frontends decide that? They have to base the decision on experience with backends. And it looks like the current crop of backends are split down the middle. Remember that if pip decides to go sdist->wheel, then that strongly motivates xoviat to abandon his current approach to ensure consistency, as it's duplicating the work pip will do. At which point the setuptools backend can't be assumed to be consistent, and the flit backend must be assumed to be (because we can't guarantee it can make a sdist).
Well I don’t think it’s solely based on their experience with backends, but also what the general goal of their tool is. Something that is designed to run in CI might take a very strict stance to be extra sure to catch errors, something that runs in cases where you care more about performance than correctness in edge cases or specialized tools where they’ve done extra work and aren’t processing arbitrary Packages might just skip straight to building wheels. — Donald Stufft
But I'm suspicious of the rationale that *there will be fewer frontends so they should have more responsibility*.
To be fair, pip is currently struggling to keep up with project requirements as it is, which has caused some frustration in the community (not that the frustration isn't wrong, but I don't think it's not going to resolve any problems). I don't see how having even more frontends will resolve these manpower issues. 2017-08-28 14:32 GMT-05:00 Thomas Kluyver <thomas@kluyver.me.uk>:
On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote:
Maybe we go fully to Nick's proposal that we don't mandate any sort of consistency constraints in the PEP. That would mean pip *has* to go sdist->wheel (because pip does need consistent behaviour), and xoviat's setuptools backend can skip building a sdist on the way to building a wheel. It also means that the build_wheel hook is essentially unreliable in isolation, and that all frontends will likely have to do the build_sdist->build_wheel with fallback to inplace build_wheel dance that pip does. But we expect many less frontends than backends, so maybe that's the right trade-off?
I've mentioned this before, but I have little faith in our ability to predict that one side of an interface will be far more numerous than the other. This is partly from my experience with Jupyter, where we got that prediction completely wrong. But I also think it's quite plausible that many frontend tools will want to use this interface (to argue against myself, most will be Python tools, so they could theoretically share a common wrapper module - but I have some doubts about whether they will).
I'm not saying we shouldn't do this - as Donald is firmly in favour and you're wavering, it seems the easier option to wrap the discussion up. But I'm suspicious of the rationale that *there will be fewer frontends so they should have more responsibility*.
Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Mon, Aug 28, 2017 at 3:48 PM xoviat <xoviat@gmail.com> wrote:
But I'm suspicious of the rationale that *there will be fewer frontends so they should have more responsibility*.
To be fair, pip is currently struggling to keep up with project requirements as it is, which has caused some frustration in the community (not that the frustration isn't wrong, but I don't think it's not going to resolve any problems). I don't see how having even more frontends will resolve these manpower issues.
Simple. It would be different people.
On 28 August 2017 at 20:32, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote:
Maybe we go fully to Nick's proposal that we don't mandate any sort of consistency constraints in the PEP. That would mean pip *has* to go sdist->wheel (because pip does need consistent behaviour), and xoviat's setuptools backend can skip building a sdist on the way to building a wheel. It also means that the build_wheel hook is essentially unreliable in isolation, and that all frontends will likely have to do the build_sdist->build_wheel with fallback to inplace build_wheel dance that pip does. But we expect many less frontends than backends, so maybe that's the right trade-off?
I've mentioned this before, but I have little faith in our ability to predict that one side of an interface will be far more numerous than the other. This is partly from my experience with Jupyter, where we got that prediction completely wrong. But I also think it's quite plausible that many frontend tools will want to use this interface (to argue against myself, most will be Python tools, so they could theoretically share a common wrapper module - but I have some doubts about whether they will).
I'm not saying we shouldn't do this - as Donald is firmly in favour and you're wavering, it seems the easier option to wrap the discussion up. But I'm suspicious of the rationale that *there will be fewer frontends so they should have more responsibility*.
Me too. At the moment, I only expect two backends of any substance - your flit backend and xoviat's setuptools one. But I only know of one frontend, namely pip - and talk of projects like tox or twine acting as frontends never seems to get any traction. The problem is that xoviat has to do a lot of work to provide guarantees that flit gets essentially for free, but flit can't provide the guaranteed sdist->wheel route that would let pip just go sdist->wheel cleanly. So there's a mismatch of requirements and *someone* needs to plaster over the cracks. At the moment, the debate is essentially whether that's xoviat or pip. Donald wants it to be pip because we can't guarantee that the next backend developer will be aware of all of the issues we debated here, and so we have to protect against that. I was persuaded[1] that we could document the requirements in the spec and expect that to be enough. I do worry that there's a risk that we're crippling the PEP with sufficient complexity that no-one will bother writing alternative frontends *or* additional backends (although I hope your support library might ease that problem somewhat). My main motivation for wavering is that I thought agreeing to trust the backend would simplify many of the decisions, and it's immensely frustrating to me that we're still debating the same question in the "return None" thread. Paul [1] I don't say "I thought" because I *never* thought backends would be sufficiently reliable. I just accepted that my concerns are only hypothetical. I wish others could do the same :-(
On Mon, Aug 28, 2017, at 08:50 PM, Paul Moore wrote:
My main motivation for wavering is that I thought agreeing to trust the backend would simplify many of the decisions, and it's immensely frustrating to me that we're still debating the same question in the "return None" thread.
The difference I see with the "return None" question is that there we have an alternative (return NotImplemented) which is just as simple for both sides, but avoids the identified issue with a buggy backend. The only argument there seems to be for using None is about semantics - and that's not a great argument, because 'practicality beats purity'. With the questions over sdist/wheel consistency, there's a complexity cost, for the spec and for frontends, in deciding that they can't. So we're weighing a trade-off: do we force ourselves to resolve the notimplemented question so that frontends can do sdist-wheel+fallback, or do we leave it up to frontends and risk some bugs which we might otherwise have prevented. Thomas
Then end the debate by letting the PEP authors decide the return type, and write a paragraph explaining why the other options were rejected. It is not going to make a big difference. On Mon, Aug 28, 2017 at 3:59 PM Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Mon, Aug 28, 2017, at 08:50 PM, Paul Moore wrote:
My main motivation for wavering is that I thought agreeing to trust the backend would simplify many of the decisions, and it's immensely frustrating to me that we're still debating the same question in the "return None" thread.
The difference I see with the "return None" question is that there we have an alternative (return NotImplemented) which is just as simple for both sides, but avoids the identified issue with a buggy backend. The only argument there seems to be for using None is about semantics - and that's not a great argument, because 'practicality beats purity'.
With the questions over sdist/wheel consistency, there's a complexity cost, for the spec and for frontends, in deciding that they can't. So we're weighing a trade-off: do we force ourselves to resolve the notimplemented question so that frontends can do sdist-wheel+fallback, or do we leave it up to frontends and risk some bugs which we might otherwise have prevented.
Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Then end the debate by letting the PEP authors decide the return type, and write a paragraph explaining why the other options were rejected. It is not going to make a big difference. Will that work now? Are we all so tired of this endless war that people will sign a peace treaty written by the people whose names are on the PEP (Nathaniel & me)? If so, let the trumpets sound, and the heralds declare that "return NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's
On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote: preference right ;-)
If so, let the trumpets sound, and the heralds declare that "return NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's preference right ;-)
For better or for worse, the trumpets seem to be sounding against this idea (Nathaniel seemed okay with whatever Donald and Nick thought was appropriate). 2017-08-28 15:27 GMT-05:00 Thomas Kluyver <thomas@kluyver.me.uk>:
On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote:
Then end the debate by letting the PEP authors decide the return type, and write a paragraph explaining why the other options were rejected. It is not going to make a big difference.
Will that work now? Are we all so tired of this endless war that people will sign a peace treaty written by the people whose names are on the PEP (Nathaniel & me)?
If so, let the trumpets sound, and the heralds declare that "return NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's preference right ;-)
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Mon, Aug 28, 2017 at 1:27 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote:
Then end the debate by letting the PEP authors decide the return type, and write a paragraph explaining why the other options were rejected. It is not going to make a big difference.
Will that work now? Are we all so tired of this endless war that people will sign a peace treaty written by the people whose names are on the PEP (Nathaniel & me)?
If so, let the trumpets sound, and the heralds declare that "return NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's preference right ;-)
Fine with me, though if it turns out Donald and Nick prefer the version where the backend has to export an exception class then I'm fine with that too. (I'm basing this on -- Donald has said he likes it, and Nick hasn't commented yet but AFAICT it does address his concerns with NotImplemented, so it seem like a plausible outcome.) I hope Nick had a good weekend at the beach or whatever, because this is going to be a heck of an email backlog to come back to... -n -- Nathaniel J. Smith -- https://vorpus.org
On Mon, 28 Aug 2017 at 16:29 Nathaniel Smith <njs@pobox.com> wrote:
On Mon, Aug 28, 2017 at 1:27 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote:
Then end the debate by letting the PEP authors decide the return type, and write a paragraph explaining why the other options were rejected. It is not going to make a big difference.
Will that work now? Are we all so tired of this endless war that people will sign a peace treaty written by the people whose names are on the PEP (Nathaniel & me)?
If so, let the trumpets sound, and the heralds declare that "return NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's preference right ;-)
Fine with me, though if it turns out Donald and Nick prefer the version where the backend has to export an exception class then I'm fine with that too. (I'm basing this on -- Donald has said he likes it, and Nick hasn't commented yet but AFAICT it does address his concerns with NotImplemented, so it seem like a plausible outcome.)
I loathe to weigh in on this and add yet another voice in this discussion, but the exported exception seems like the best solution for everyone involved from my lurking perspective. For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution). And I was fine with NotImplementedError but if people want something more specific and None is out due to worries of accidental bare returns, then the exported exception comes the closest to making everyone happy (it does tick the EIBTI box :) .
On Aug 28, 2017, at 3:59 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
The difference I see with the "return None" question is that there we have an alternative (return NotImplemented) which is just as simple for both sides, but avoids the identified issue with a buggy backend. The only argument there seems to be for using None is about semantics - and that's not a great argument, because 'practicality beats purity'.
With the questions over sdist/wheel consistency, there's a complexity cost, for the spec and for frontends, in deciding that they can't. So we're weighing a trade-off: do we force ourselves to resolve the notimplemented question so that frontends can do sdist-wheel+fallback, or do we leave it up to frontends and risk some bugs which we might otherwise have prevented.
I think the best option is to just export an exception that a frontend can catch. That gives like 95% of the benefit of either mechanism for an extra 5% of work. If a backend is unwilling to add a: class UnsupportedOperation(Exception): pass To their backend, then they’re likely not a backend that we should be spending a whole lot of time or effort trying to support. Two copy/pasteable lines of code is a tiny price to pay. I haven’t talked to Nick, but I’d be surprised if he was against this, since it’s basically what he wanted with a more specific exception (which is good practice anyways). — Donald Stufft
On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Me too. At the moment, I only expect two backends of any substance - your flit backend and xoviat's setuptools one. But I only know of one frontend, namely pip - and talk of projects like tox or twine acting as frontends never seems to get any traction.
If the build API is clean and supported enough, I can see conda using it. just sayin' NOTE: it wouldn't be conda per-se using it -- but individual conda recipes, which generally call setuptools directly now. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 29 August 2017 at 01:13, Chris Barker <chris.barker@noaa.gov> wrote:
On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Me too. At the moment, I only expect two backends of any substance - your flit backend and xoviat's setuptools one. But I only know of one frontend, namely pip - and talk of projects like tox or twine acting as frontends never seems to get any traction.
If the build API is clean and supported enough, I can see conda using it.
just sayin'
Thanks - that's really good to know. So *is* the build API clean enough for you? Specifically: 1. build_sdist can be missing, or can report back (somehow) that it failed to produce a sdist, and you should fall back to build_wheel. 2. build_wheel must always succeed and must always exist. 3. All hooks must be run in a separate subprocess There's also error handling, which I don't recall the details of, but I think boils down to the backend code can write what it likes to stdout/stderr and may raise an exception indicating "something blew up". Those are basically the key points at the moment. Add to that, does conda mind if build_wheel might result in a different wheel than build_sdist->build_wheel would produce? Paul
On 2017-08-29 01:20, Paul Moore wrote:
On 29 August 2017 at 01:13, Chris Barker <chris.barker@noaa.gov> wrote:
On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Me too. At the moment, I only expect two backends of any substance - your flit backend and xoviat's setuptools one. But I only know of one frontend, namely pip - and talk of projects like tox or twine acting as frontends never seems to get any traction.
scikit-build also expressed some interest: https://www.mail-archive.com/distutils-sig@python.org/msg26553.html although I don't see the pull request.
If the build API is clean and supported enough, I can see conda using it.
just sayin'
Thanks - that's really good to know. So *is* the build API clean enough for you? Specifically:
1. build_sdist can be missing, or can report back (somehow) that it failed to produce a sdist, and you should fall back to build_wheel. 2. build_wheel must always succeed and must always exist. 3. All hooks must be run in a separate subprocess
There's also error handling, which I don't recall the details of, but I think boils down to the backend code can write what it likes to stdout/stderr and may raise an exception indicating "something blew up".
Those are basically the key points at the moment. Add to that, does conda mind if build_wheel might result in a different wheel than build_sdist->build_wheel would produce?
conda-build doesn't produce or consume wheels. It works by creating a clean conda environment and running a shell script to install the python package into that environment (including its own versions of /usr/lib /usr/etc /usr/bin /urs/include /sbin -- which is why it can cope with non-python packages like g++, julia, R, git etc.). All files produced by the install script are then put into a tar file, which is conda's version of a wheel.
On 29 August 2017 at 17:22, Phil Austin <paustin@eoas.ubc.ca> wrote:
On 2017-08-29 01:20, Paul Moore wrote:
On 29 August 2017 at 01:13, Chris Barker <chris.barker@noaa.gov> wrote:
If the build API is clean and supported enough, I can see conda using it.
just sayin'
Thanks - that's really good to know. So *is* the build API clean enough for you? Specifically:
1. build_sdist can be missing, or can report back (somehow) that it failed to produce a sdist, and you should fall back to build_wheel. 2. build_wheel must always succeed and must always exist. 3. All hooks must be run in a separate subprocess
There's also error handling, which I don't recall the details of, but I think boils down to the backend code can write what it likes to stdout/stderr and may raise an exception indicating "something blew up".
Those are basically the key points at the moment. Add to that, does conda mind if build_wheel might result in a different wheel than build_sdist->build_wheel would produce?
conda-build doesn't produce or consume wheels. It works by creating a clean conda environment and running a shell script to install the python package into that environment (including its own versions of /usr/lib /usr/etc /usr/bin /urs/include /sbin -- which is why it can cope with non-python packages like g++, julia, R, git etc.). All files produced by the install script are then put into a tar file, which is conda's version of a wheel.
OK, so I don't see how conda would use PEP 517 the way Chris suggested. But never mind - if it doesn't help conda that's fine. Paul
On Mon, Aug 28, 2017 at 12:20 PM, Paul Moore <p.f.moore@gmail.com> wrote:
interface). So if pip calls build_sdist and then build_wheel, there will be two source distributions built (one by pip and one by setuptools) before building a wheel.
exactly why pip should NOT concern itself with the sdist -- either the back end needs to do it anyway, or it doesn't need to do it at all, and would only be doing so to satisfy pip.
This is precisely the question - whose responsibility is it to ensure that sdist->wheel vs build_wheel equivalence is maintained? The PEP says it's the backend's responsibility, so pip shouldn't need to do anything.
makes sense to me. But, we're currently having debates about "return None" which hinge on
"if backends have bugs, return None might not work well". Sure - but if backends have bugs, they may not maintain wheel equivalence.
If backends have bugs (and they will), any number of things can go wrong. Making the spec so that front ends can have a better idea what went wrong is a fine idea, but it can only go so far. The only thing we really need to distinguish is " that functionality is not implemented" from "something went wrong" And not even that if we don't have pip trying to go through some machinations about what do if the back-end does not produce a sdist.
To that extent, I'm with Donald - pip going sdist->wheel protects the user against the known-to-be-an-issue bug that the backend doesn't ensure wheel equivalence.
pip can not protect the user from a poorly written back-end. And it shouldn't try. * Chris Barker has pointed out that backends have no reason to support
sdists now.
not quite -- the reason to support sdist is because you want an sdist, not because it's a necessary part of the path to a wheel. I don't quite understand your (Paul's) point about the importance of sdists to open source (as opposed to a plain old tarball of the source, like what gitHub produces when you do a release. * Nathaniel is pushing a means of notifying "I can't build a sdist"
that protects against backends accidentally not following the spec.
"I can't do that" respond to any possible request makes sense. Unless "I can't build a sdist" violates the protocol -- which I don't think it should.
Should we trust the backend or not? Backends *will* have bugs - part
of "trust the backend" is simply telling the user that the problem behaviour they found is not pip's issue and should be reported to the backend. Is that a sufficiently bad user experience that we should try to improve it (experience with setuptools says that it is - why are we assuming that developers of new backends will be so much more conscientious and careful than the setuptools developers? *I* certainly don't feel that the setuptools developers are unusually bad - quite the opposite!)
If that is a sufficiently bad user experience then the only other option is for "us" by some definition of "us" to build the back-end, too -- which is where this all started with setuptool. But my understanding is that goal is was not to force the whole stack to maintained together. And really, the end-user will need to report the problem to the package maintainer, not the build tool maintainer jsu tlike they should no if something doesn't pip install. Having a more robust means of saying "I can't build a sdist" than
"return None" is protecting the user from issues with the backend. So is building via sdist.
but only one particular set of issues -- when there are an infinite number of possible issues... One other thought -- the easiest way to make an sdist it to simply copy the source tree. So: 1) not a big deal to require back-ends to do it -- it's easy to do but 2) back ends may do it the easy and sloppy way, and get build artifacts in the sdist, and then you are back where you started. so requiring the sdist step in no way saves you from poorly written back-ends. It may make it worse, as you _think_ you are getting something clean. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Aug 26, 2017 2:17 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
[removed Guido from CC]
On Aug 26, 2017 02:29, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do.
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Why does the frontend need to know why an sdist was not created? Frontend is asking the backend, given the current state of the world, to either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I would expect build_sdist to make some sanity checks about the world, then make a binary choice about whether sdist creation is a valid goal. If not possible, return None or NotImplemented or False or dict-of-reasons or whatever. Only if creation was *attempted*, and in the exceptional event it then failed, would I expect an Exception. We don't have structured exceptions sadly so they can't really carry much useful information from a protocol perspective above and beyond a simple None or the like anyway. I'd personally like to see some parity between build_sdist and build_wheel in this regard. Maybe the disconnect here is we have a way to specify hard reqs for building a wheel, statically or dynamically, and build_wheel is expected to never fail, but no way to specify hard reqs needed for build_sdist, necessitating this optional signaling path? If we had some definitive way for the frontend to know ahead of time if build_sdist is even expected to work, it could be called with more confidence. This could be a new sdist-related key in [build-system], a new table like [sdist-system].requires, or making the get_requires_for_* less optional, and defaulting to None instead of [ ]. Frontend is responsible for prepping the world, so if it can't get a list of reqs, somehow, for build_sdist, it knows it can't work. Same for build_wheel, because you have to specify the backend itself, so there is at least one requirement! Thus if you are a backend that can produce an sdist without additional requirements beyond build reqs, you should explicitly return empty list from get_requires_for_build_sdist. -- C Anthony
Why does the frontend need to know why an sdist was not created? I was of the opinion that such a distinction is not necessary because building a source distribution doesn't take that much time. However Donald thought that there needed to be a distinction because of the wasted time in attempting to build a wheel that was going to fail anyway. One of the things to consider is that site cythonizing takes time and maybe called for building source distribution. However since I think we're of the agreement that a source distribution should be as close to a checkout as possible, that may not be an issue because cythonizing may not be required to build the sdist. On Aug 26, 2017 3:47 PM, "C Anthony Risinger" <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 2:17 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
[removed Guido from CC]
On Aug 26, 2017 02:29, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do.
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Why does the frontend need to know why an sdist was not created?
Frontend is asking the backend, given the current state of the world, to either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I would expect build_sdist to make some sanity checks about the world, then make a binary choice about whether sdist creation is a valid goal. If not possible, return None or NotImplemented or False or dict-of-reasons or whatever. Only if creation was *attempted*, and in the exceptional event it then failed, would I expect an Exception. We don't have structured exceptions sadly so they can't really carry much useful information from a protocol perspective above and beyond a simple None or the like anyway.
I'd personally like to see some parity between build_sdist and build_wheel in this regard. Maybe the disconnect here is we have a way to specify hard reqs for building a wheel, statically or dynamically, and build_wheel is expected to never fail, but no way to specify hard reqs needed for build_sdist, necessitating this optional signaling path?
If we had some definitive way for the frontend to know ahead of time if build_sdist is even expected to work, it could be called with more confidence.
This could be a new sdist-related key in [build-system], a new table like [sdist-system].requires, or making the get_requires_for_* less optional, and defaulting to None instead of [ ].
Frontend is responsible for prepping the world, so if it can't get a list of reqs, somehow, for build_sdist, it knows it can't work. Same for build_wheel, because you have to specify the backend itself, so there is at least one requirement!
Thus if you are a backend that can produce an sdist without additional requirements beyond build reqs, you should explicitly return empty list from get_requires_for_build_sdist.
--
C Anthony
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I also think that Guido pretty much ruled out Notimplemented. On Aug 26, 2017 4:04 PM, "xoviat" <xoviat@gmail.com> wrote:
Why does the frontend need to know why an sdist was not created?
I was of the opinion that such a distinction is not necessary because building a source distribution doesn't take that much time. However Donald thought that there needed to be a distinction because of the wasted time in attempting to build a wheel that was going to fail anyway. One of the things to consider is that site cythonizing takes time and maybe called for building source distribution. However since I think we're of the agreement that a source distribution should be as close to a checkout as possible, that may not be an issue because cythonizing may not be required to build the sdist.
On Aug 26, 2017 3:47 PM, "C Anthony Risinger" <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 2:17 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
[removed Guido from CC]
On Aug 26, 2017 02:29, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do.
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Why does the frontend need to know why an sdist was not created?
Frontend is asking the backend, given the current state of the world, to either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I would expect build_sdist to make some sanity checks about the world, then make a binary choice about whether sdist creation is a valid goal. If not possible, return None or NotImplemented or False or dict-of-reasons or whatever. Only if creation was *attempted*, and in the exceptional event it then failed, would I expect an Exception. We don't have structured exceptions sadly so they can't really carry much useful information from a protocol perspective above and beyond a simple None or the like anyway.
I'd personally like to see some parity between build_sdist and build_wheel in this regard. Maybe the disconnect here is we have a way to specify hard reqs for building a wheel, statically or dynamically, and build_wheel is expected to never fail, but no way to specify hard reqs needed for build_sdist, necessitating this optional signaling path?
If we had some definitive way for the frontend to know ahead of time if build_sdist is even expected to work, it could be called with more confidence.
This could be a new sdist-related key in [build-system], a new table like [sdist-system].requires, or making the get_requires_for_* less optional, and defaulting to None instead of [ ].
Frontend is responsible for prepping the world, so if it can't get a list of reqs, somehow, for build_sdist, it knows it can't work. Same for build_wheel, because you have to specify the backend itself, so there is at least one requirement!
Thus if you are a backend that can produce an sdist without additional requirements beyond build reqs, you should explicitly return empty list from get_requires_for_build_sdist.
--
C Anthony
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sat, Aug 26, 2017 at 2:06 PM, xoviat <xoviat@gmail.com> wrote:
I also think that Guido pretty much ruled out Notimplemented.
As I've said, I don't think it matters a huge deal whether we use NotImplemented or not. But please don't treat Guido as some kind of pronouncement generating machine where you hurl out-of-context questions at him and then use his response as a club to beat down discussion. It's rude to Guido, it's rude to Nick and Donald (to whom Guido has explicitly delegated his BDFL authority in packaging-related matters), and it's rude to everyone trying to discuss proposals on their merits. -n -- Nathaniel J. Smith -- https://vorpus.org
As I said, I don't care what the particular solution is on this issue. However I'm simply trying to anticipate and resolve potential disagreements that could drag this out for a significant period of time. It is clear that Nick and Donald have a disagreement on this issue which is actually not in and of itself packaging related. The clear way to resolve this was to get a pronouncement. On Aug 26, 2017 4:56 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 2:06 PM, xoviat <xoviat@gmail.com> wrote:
I also think that Guido pretty much ruled out Notimplemented.
As I've said, I don't think it matters a huge deal whether we use NotImplemented or not. But please don't treat Guido as some kind of pronouncement generating machine where you hurl out-of-context questions at him and then use his response as a club to beat down discussion. It's rude to Guido, it's rude to Nick and Donald (to whom Guido has explicitly delegated his BDFL authority in packaging-related matters), and it's rude to everyone trying to discuss proposals on their merits.
-n
-- Nathaniel J. Smith -- https://vorpus.org
With respect to this issue, if everyone told me that I was wrong then I would say I'm obviously wrong. But some people are saying one thing and other people are saying something else. On Aug 26, 2017 5:03 PM, "xoviat" <xoviat@gmail.com> wrote:
As I said, I don't care what the particular solution is on this issue. However I'm simply trying to anticipate and resolve potential disagreements that could drag this out for a significant period of time. It is clear that Nick and Donald have a disagreement on this issue which is actually not in and of itself packaging related. The clear way to resolve this was to get a pronouncement.
On Aug 26, 2017 4:56 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 2:06 PM, xoviat <xoviat@gmail.com> wrote:
I also think that Guido pretty much ruled out Notimplemented.
As I've said, I don't think it matters a huge deal whether we use NotImplemented or not. But please don't treat Guido as some kind of pronouncement generating machine where you hurl out-of-context questions at him and then use his response as a club to beat down discussion. It's rude to Guido, it's rude to Nick and Donald (to whom Guido has explicitly delegated his BDFL authority in packaging-related matters), and it's rude to everyone trying to discuss proposals on their merits.
-n
-- Nathaniel J. Smith -- https://vorpus.org
On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 2:17 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
[removed Guido from CC]
On Aug 26, 2017 02:29, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do.
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Why does the frontend need to know why an sdist was not created?
This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it). But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else. It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-)
Frontend is asking the backend, given the current state of the world, to either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I would expect build_sdist to make some sanity checks about the world, then make a binary choice about whether sdist creation is a valid goal. If not possible, return None or NotImplemented or False or dict-of-reasons or whatever. Only if creation was *attempted*, and in the exceptional event it then failed, would I expect an Exception. We don't have structured exceptions sadly so they can't really carry much useful information from a protocol perspective above and beyond a simple None or the like anyway.
I'd personally like to see some parity between build_sdist and build_wheel in this regard. Maybe the disconnect here is we have a way to specify hard reqs for building a wheel, statically or dynamically, and build_wheel is expected to never fail, but no way to specify hard reqs needed for build_sdist, necessitating this optional signaling path?
Not sure what you mean about hard reqs. The reason for the lack of parity is that we don't currently have any use cases where build_wheel is expected to fail, but this is expected in some sense (not sure what that would even mean), and there's some fallback that the frontend may want to invoke instead. -n -- Nathaniel J. Smith -- https://vorpus.org
Nathaniel: Just to clarify, we're talking about returning none for the build function not the get requirements function. The get requirements function is always expected to succeed and is optional. If while the backend is discovering requirements it finds that it cannot build, then it can return an empty list. It cannot however signal in the get requirements function that it cannot build. At least with what is being proposed. The reason that the proposal works is that the build function can never return none on success. On Aug 26, 2017 5:13 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 2:17 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
[removed Guido from CC]
On Aug 26, 2017 02:29, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that
returns
an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do.
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Why does the frontend need to know why an sdist was not created?
This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist
And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it).
But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else.
It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-)
Frontend is asking the backend, given the current state of the world, to either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I would expect build_sdist to make some sanity checks about the world, then make a binary choice about whether sdist creation is a valid goal. If not possible, return None or NotImplemented or False or dict-of-reasons or whatever. Only if creation was *attempted*, and in the exceptional event it then failed, would I expect an Exception. We don't have structured exceptions sadly so they can't really carry much useful information from a protocol perspective above and beyond a simple None or the like anyway.
I'd personally like to see some parity between build_sdist and build_wheel in this regard. Maybe the disconnect here is we have a way to specify hard reqs for building a wheel, statically or dynamically, and build_wheel is expected to never fail, but no way to specify hard reqs needed for build_sdist, necessitating this optional signaling path?
Not sure what you mean about hard reqs. The reason for the lack of parity is that we don't currently have any use cases where build_wheel is expected to fail, but this is expected in some sense (not sure what that would even mean), and there's some fallback that the frontend may want to invoke instead.
-n
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Aug 26, 2017 5:13 PM, "Nathaniel Smith" <njs@pobox.com> wrote: On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 2:17 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
[removed Guido from CC]
On Aug 26, 2017 02:29, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do.
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not
at
all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Why does the frontend need to know why an sdist was not created?
This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it). But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else. It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-) Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written. What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake. Consider instead, if we make the following small changes: 1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary. 2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty). 3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support). 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static) 6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported. 7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself). This arrangement allows pip to know ahead-of-time if `build_sdist` is appropriate. If no sdist reqs are explicitly acknowledged, then no sdist can be created. Full stop. Example usages: * Backeds that only support wheel creation do not implement `get_requires_for_build_sdist` at all. * Backeds that conditionally support sdist creation implement `get_requires_for_build_sdist` and return None if unsupported. This is where flit signals "impossible" from an unpacked sdist and [...] from VCS. * Backeds that always support sdist creation (setuptools) implement `get_requires_for_build_sdist` and return [...] unconditionally. So for pip's sdist -> unpack -> wheel path, it knows right away if it should build an sdist or skip right to building a wheel because the backend will inform via `get_requires_for_build_sdist` instead of overloading `build_sdist`. I also think this achieves nice parity across sdist and wheel operations, keeps the same hooks optional, adds no special cases, and maybe even less overhead. What say you? -- C Anthony
On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 5:13 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written.
What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake.
Consider instead, if we make the following small changes:
1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary.
That's not a change, that's how it works :-).
2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty).
3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support).
This is what I originally proposed, except you use None where I use NotImplemented, which has the disadvantages I noted earlier. Also, people didn't like the missing get_requires_for_build_* being treated as no-support, which makes sense, since we expect that get_requires_for_build_* won't be used very often. But one can switch the default here without affecting much else. The reason we want to let build_sdist report failure is just for convenience of backends who don't have any other reason to implement get_requires_for_build_sdist.
4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static)
5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static)
build-system.requires contains the requirements that are always installed before we even try importing the backend, so they're available to all backend hooks equally.
6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported.
7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself).
The TOML requires aren't really useful as a signal about whether sdist specifically is supported. Plus I think we probably want to leave no-requires-in-TOML as a valid option for saying "I don't need anything installed" (maybe because the backend is shipped inside the source tree) rather than overloading it to have extra meanings. -n -- Nathaniel J. Smith -- https://vorpus.org
On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 5:13 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written.
What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake.
Consider instead, if we make the following small changes:
1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary.
That's not a change, that's how it works :-).
Is that a change I missed from this thread? I'm reading here: https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301 https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254 https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel and they do not appear to receive the source or wheel directories.
2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty).
3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support).
This is what I originally proposed, except you use None where I use NotImplemented, which has the disadvantages I noted earlier. Also, people didn't like the missing get_requires_for_build_* being treated as no-support, which makes sense, since we expect that get_requires_for_build_* won't be used very often. But one can switch the default here without affecting much else. The reason we want to let build_sdist report failure is just for convenience of backends who don't have any other reason to implement get_requires_for_build_sdist.
Oh OK, good good. Well in that case I agree with you and missed the suggestion. I personally prefer NotImplemented as well here but None seemed mostly just as good and did not elicit as much pushback. It's not too big of deal either way. However, a missing `get_requires_for_build_wheel` technically signaling "unsupported" makes good sense to me because it's always supplemented by the static *and mandatory* `build-system.requires` list. There is no proper way (without breaking the spec) to signal "unsupported" for `build_wheel` since the backend itself (setuptools, wheel, flit) is specified here. "Unsupported" is only signaled when *both* the static and dynamic requires are None (or NotImplemented as mentioned). The kicker here in my offering, is that the presence of `build-system.requires` *does not in any way imply* `build_sdist` support. As written, there is no way to statically set the requirements for sdist support (though this could be changed of course with a new TOML key/table), so you must explicitly signal it with something like: def get_requires_for_build_sdist(*args, **kwds): return [] This means, by default, `build_wheel` is "supported" and `build_sdist` is "unsupported", and both are no fail operations. If called, any exception is fatal to the entire process. If a backend goes through the work of supporting sdist creation, is it really a problem to relay this support with a 3 line function definition? Wheel (could also define nothing at all): def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None Flit: def get_requires_for_build_sdist(source_dir, ...): # I can output an sdist if the source directory is a VCS checkout I understand. requires = get_requires_for_vcs_checkout_or_signal_unsupported(source_dir) return requires Setuptools: def get_requires_for_build_sdist(source_dir, ...): # I'm going to successfully create an sdist or die trying! # There is literally no directory I can't handle so I don't even look. return [] This seems pretty straightforward to me and avoids overloading `build_sdist`, keeping it no fail like `build_wheel`. Semantically, it's really the job of the requirements discovery mechanism to decide one of: a) Zero or more requirements are needed to transform target source_dir into an sdist. b) No requirement enables my backend to transform target source_dir into an sdist. This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let this hook relay it's decision more completely?
4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static)
5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static)
build-system.requires contains the requirements that are always installed before we even try importing the backend, so they're available to all backend hooks equally.
Yes absolutely. If said backend wants to also signal unconditional sdist support, it simply needs to: def get_requires_for_build_sdist(source_dir, ...): return [] and call it a day. The point of my `sdist reqs =` comment was to stress that while the presence of `build-system.requires` inherently signals `build_wheel` support, it *does not* signal `build_sdist` support.
6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported.
7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself).
The TOML requires aren't really useful as a signal about whether sdist specifically is supported. Plus I think we probably want to leave no-requires-in-TOML as a valid option for saying "I don't need anything installed" (maybe because the backend is shipped inside the source tree) rather than overloading it to have extra meanings.
The spec currently states that the `requires` key is mandatory, so `build_wheel` will unconditionally signal readiness. I was more referring to if we decided to add a separate key, like `sdist-requires`, to statically signal `build_sdist` readiness via TOML. You could use such a key to pull in a plugin for the chosen build backend, such as a hypothetical `flit.sdist` package, instead of baking the support into flit. Even without the addition of an `sdist-requires` TOML key, build backends can always signal sdist support dynamically by defining a basic `get_requires_for_build_sdist` returning an empty list. The most basic build backend is still one that defines only `build_wheel` and nothing else. If it also defines `build_sdist`, well great, but pip and friends will not use it unless it also defines an appropriate `get_requires_for_build_sdist`. This is an extremely low bar to me and achieves commonality in signature and behavior across all get_requires_* and build_* hooks. -- C Anthony
and they do not appear to receive the source or wheel directories.
This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let
The source directory is the current directory, if I am not mistaken. this hook relay it's decision more completely? Not trying to speak on behalf of flit here, but if I understand correctly, flit requires git to build a source distribution. Flit knows its build requirements, so it can just return them when get_requires is called. However, it needs to attempt to invoke git to find out whether it can build a source distribution, which if I understand correctly, is a lengthy operation (whether that's actually true is not actually relevant because we are discussion potentially any backend/operation). It's more efficient if git is only invoked once, and if a failure occurs, then under the proposal "None" would be returned from build_sdist rather than the name of the built source distribution. def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None I have to say that the above example is not a good idea. Perhaps I was a bit muddled on that point earlier: the reason that "return None" is a good idea for build sdist is, well: def build_sdist(...): # Indicates success return "x.tar.gz" def build_sdist(...): # Indicates failure return None Those are not truthily equivalent, which is important because it means that someone is unlikely to make a mistake on that matter. 2017-08-26 23:05 GMT-05:00 C Anthony Risinger <c@anthonyrisinger.com>:
On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 5:13 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written.
What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake.
Consider instead, if we make the following small changes:
1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary.
That's not a change, that's how it works :-).
Is that a change I missed from this thread? I'm reading here:
https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301 https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254 https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel
and they do not appear to receive the source or wheel directories.
2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty).
3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support).
This is what I originally proposed, except you use None where I use NotImplemented, which has the disadvantages I noted earlier. Also, people didn't like the missing get_requires_for_build_* being treated as no-support, which makes sense, since we expect that get_requires_for_build_* won't be used very often. But one can switch the default here without affecting much else. The reason we want to let build_sdist report failure is just for convenience of backends who don't have any other reason to implement get_requires_for_build_sdist.
Oh OK, good good. Well in that case I agree with you and missed the suggestion. I personally prefer NotImplemented as well here but None seemed mostly just as good and did not elicit as much pushback. It's not too big of deal either way.
However, a missing `get_requires_for_build_wheel` technically signaling "unsupported" makes good sense to me because it's always supplemented by the static *and mandatory* `build-system.requires` list. There is no proper way (without breaking the spec) to signal "unsupported" for `build_wheel` since the backend itself (setuptools, wheel, flit) is specified here. "Unsupported" is only signaled when *both* the static and dynamic requires are None (or NotImplemented as mentioned). The kicker here in my offering, is that the presence of `build-system.requires` *does not in any way imply* `build_sdist` support. As written, there is no way to statically set the requirements for sdist support (though this could be changed of course with a new TOML key/table), so you must explicitly signal it with something like:
def get_requires_for_build_sdist(*args, **kwds): return []
This means, by default, `build_wheel` is "supported" and `build_sdist` is "unsupported", and both are no fail operations. If called, any exception is fatal to the entire process. If a backend goes through the work of supporting sdist creation, is it really a problem to relay this support with a 3 line function definition?
Wheel (could also define nothing at all):
def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None
Flit:
def get_requires_for_build_sdist(source_dir, ...): # I can output an sdist if the source directory is a VCS checkout I understand. requires = get_requires_for_vcs_checkout_or_signal_unsupported(source_ dir) return requires
Setuptools:
def get_requires_for_build_sdist(source_dir, ...): # I'm going to successfully create an sdist or die trying! # There is literally no directory I can't handle so I don't even look. return []
This seems pretty straightforward to me and avoids overloading `build_sdist`, keeping it no fail like `build_wheel`.
Semantically, it's really the job of the requirements discovery mechanism to decide one of:
a) Zero or more requirements are needed to transform target source_dir into an sdist. b) No requirement enables my backend to transform target source_dir into an sdist.
This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let this hook relay it's decision more completely?
4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static)
5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static)
build-system.requires contains the requirements that are always installed before we even try importing the backend, so they're available to all backend hooks equally.
Yes absolutely. If said backend wants to also signal unconditional sdist support, it simply needs to:
def get_requires_for_build_sdist(source_dir, ...): return []
and call it a day. The point of my `sdist reqs =` comment was to stress that while the presence of `build-system.requires` inherently signals `build_wheel` support, it *does not* signal `build_sdist` support.
6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported.
7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself).
The TOML requires aren't really useful as a signal about whether sdist specifically is supported. Plus I think we probably want to leave no-requires-in-TOML as a valid option for saying "I don't need anything installed" (maybe because the backend is shipped inside the source tree) rather than overloading it to have extra meanings.
The spec currently states that the `requires` key is mandatory, so `build_wheel` will unconditionally signal readiness. I was more referring to if we decided to add a separate key, like `sdist-requires`, to statically signal `build_sdist` readiness via TOML. You could use such a key to pull in a plugin for the chosen build backend, such as a hypothetical `flit.sdist` package, instead of baking the support into flit.
Even without the addition of an `sdist-requires` TOML key, build backends can always signal sdist support dynamically by defining a basic `get_requires_for_build_sdist` returning an empty list.
The most basic build backend is still one that defines only `build_wheel` and nothing else. If it also defines `build_sdist`, well great, but pip and friends will not use it unless it also defines an appropriate `get_requires_for_build_sdist`. This is an extremely low bar to me and achieves commonality in signature and behavior across all get_requires_* and build_* hooks.
--
C Anthony
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sat, Aug 26, 2017 at 11:18 PM, xoviat <xoviat@gmail.com> wrote:
and they do not appear to receive the source or wheel directories.
The source directory is the current directory, if I am not mistaken.
Oh right right, for some reason I was reading the `build_*` hooks as receiving the input directory rather than the output directory. I don't see it stated anywhere that the PWD will be the source directory however. Might be a good addition. In this case, `get_requires_for_*` hooks should also be able to depend on PWD being the applicable source directory (which of course might be an unpacked sdist like in pip's proposed multistage sdist -> unpack -> wheel pipeline).
This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let this hook relay it's decision more completely?
Not trying to speak on behalf of flit here, but if I understand correctly, flit requires git to build a source distribution. Flit knows its build requirements, so it can just return them when get_requires is called. However, it needs to attempt to invoke git to find out whether it can build a source distribution, which if I understand correctly, is a lengthy operation (whether that's actually true is not actually relevant because we are discussion potentially any backend/operation). It's more efficient if git is only invoked once, and if a failure occurs, then under the proposal "None" would be returned from build_sdist rather than the name of the built source distribution.
What is the benefit in making `build_sdist` handle this assertion? Requirements checking is only performed once, earlier than building. Somewhere, somehow, something must look to see if git exists. Is it not more appropriate to check early, while asserting python requirements, vs. actually being told to build an sdist? That is already the expectation on the wheel side of things since it would also fail if necessary system binaries are missing. Checking for git is conflating system reqs and python reqs as "requirements", but I don't see why the get_requires_* hooks can't do both if they so choose.
def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None
I have to say that the above example is not a good idea. Perhaps I was a bit muddled on that point earlier: the reason that "return None" is a good idea for build sdist is, well:
def build_sdist(...): # Indicates success return "x.tar.gz"
def build_sdist(...): # Indicates failure return None
Those are not truthily equivalent, which is important because it means that someone is unlikely to make a mistake on that matter.
What is the difference? Both equally inform the frontend that building an sdist is not going to happen. At the end of the day, I'm not too partial to any of the solutions I've read thus far, and do not have any desire to derail progress. I just don't like the variation in `build_sdist` vs. `build_wheel` because it feels like a kludge to me. I thought perhaps defaulting sdists to "unsupported" and requiring a backend to opt-in was both more elegant and more explicit, allowing tools like flit to better express their conditional sdist support. -- C Anthony
The difference is that an empty list for requires would indicate that building is possible but none would not. Those two values are truthily equivalent. For building, the name returned indicates building was possible, but it's not truthily equivalent to none. On Aug 27, 2017 12:08 AM, "C Anthony Risinger" <c@anthonyrisinger.com> wrote:
On Sat, Aug 26, 2017 at 11:18 PM, xoviat <xoviat@gmail.com> wrote:
and they do not appear to receive the source or wheel directories.
The source directory is the current directory, if I am not mistaken.
Oh right right, for some reason I was reading the `build_*` hooks as receiving the input directory rather than the output directory. I don't see it stated anywhere that the PWD will be the source directory however. Might be a good addition.
In this case, `get_requires_for_*` hooks should also be able to depend on PWD being the applicable source directory (which of course might be an unpacked sdist like in pip's proposed multistage sdist -> unpack -> wheel pipeline).
This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let this hook relay it's decision more completely?
Not trying to speak on behalf of flit here, but if I understand correctly, flit requires git to build a source distribution. Flit knows its build requirements, so it can just return them when get_requires is called. However, it needs to attempt to invoke git to find out whether it can build a source distribution, which if I understand correctly, is a lengthy operation (whether that's actually true is not actually relevant because we are discussion potentially any backend/operation). It's more efficient if git is only invoked once, and if a failure occurs, then under the proposal "None" would be returned from build_sdist rather than the name of the built source distribution.
What is the benefit in making `build_sdist` handle this assertion? Requirements checking is only performed once, earlier than building. Somewhere, somehow, something must look to see if git exists. Is it not more appropriate to check early, while asserting python requirements, vs. actually being told to build an sdist? That is already the expectation on the wheel side of things since it would also fail if necessary system binaries are missing. Checking for git is conflating system reqs and python reqs as "requirements", but I don't see why the get_requires_* hooks can't do both if they so choose.
def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None
I have to say that the above example is not a good idea. Perhaps I was a bit muddled on that point earlier: the reason that "return None" is a good idea for build sdist is, well:
def build_sdist(...): # Indicates success return "x.tar.gz"
def build_sdist(...): # Indicates failure return None
Those are not truthily equivalent, which is important because it means that someone is unlikely to make a mistake on that matter.
What is the difference? Both equally inform the frontend that building an sdist is not going to happen.
At the end of the day, I'm not too partial to any of the solutions I've read thus far, and do not have any desire to derail progress. I just don't like the variation in `build_sdist` vs. `build_wheel` because it feels like a kludge to me. I thought perhaps defaulting sdists to "unsupported" and requiring a backend to opt-in was both more elegant and more explicit, allowing tools like flit to better express their conditional sdist support.
--
C Anthony
On Sat, Aug 26, 2017 at 11:05 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 5:13 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written.
What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake.
Consider instead, if we make the following small changes:
1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary.
That's not a change, that's how it works :-).
Is that a change I missed from this thread? I'm reading here:
https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301 https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254 https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel
and they do not appear to receive the source or wheel directories.
2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty).
3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support).
This is what I originally proposed, except you use None where I use NotImplemented, which has the disadvantages I noted earlier. Also, people didn't like the missing get_requires_for_build_* being treated as no-support, which makes sense, since we expect that get_requires_for_build_* won't be used very often. But one can switch the default here without affecting much else. The reason we want to let build_sdist report failure is just for convenience of backends who don't have any other reason to implement get_requires_for_build_sdist.
Oh OK, good good. Well in that case I agree with you and missed the suggestion. I personally prefer NotImplemented as well here but None seemed mostly just as good and did not elicit as much pushback. It's not too big of deal either way.
However, a missing `get_requires_for_build_wheel` technically signaling "unsupported" makes good sense to me because it's always supplemented by the static *and mandatory* `build-system.requires` list. There is no proper way (without breaking the spec) to signal "unsupported" for `build_wheel` since the backend itself (setuptools, wheel, flit) is specified here. "Unsupported" is only signaled when *both* the static and dynamic requires are None (or NotImplemented as mentioned). The kicker here in my offering, is that the presence of `build-system.requires` *does not in any way imply* `build_sdist` support. As written, there is no way to statically set the requirements for sdist support (though this could be changed of course with a new TOML key/table), so you must explicitly signal it with something like:
def get_requires_for_build_sdist(*args, **kwds): return []
This means, by default, `build_wheel` is "supported" and `build_sdist` is "unsupported", and both are no fail operations. If called, any exception is fatal to the entire process. If a backend goes through the work of supporting sdist creation, is it really a problem to relay this support with a 3 line function definition?
Wheel (could also define nothing at all):
def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None
Flit:
def get_requires_for_build_sdist(source_dir, ...): # I can output an sdist if the source directory is a VCS checkout I understand. requires = get_requires_for_vcs_checkout_or_signal_unsupported(source_ dir) return requires
Setuptools:
def get_requires_for_build_sdist(source_dir, ...): # I'm going to successfully create an sdist or die trying! # There is literally no directory I can't handle so I don't even look. return []
This seems pretty straightforward to me and avoids overloading `build_sdist`, keeping it no fail like `build_wheel`.
Semantically, it's really the job of the requirements discovery mechanism to decide one of:
a) Zero or more requirements are needed to transform target source_dir into an sdist. b) No requirement enables my backend to transform target source_dir into an sdist.
This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let this hook relay it's decision more completely?
4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static)
5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static)
build-system.requires contains the requirements that are always installed before we even try importing the backend, so they're available to all backend hooks equally.
Yes absolutely. If said backend wants to also signal unconditional sdist support, it simply needs to:
def get_requires_for_build_sdist(source_dir, ...): return []
and call it a day. The point of my `sdist reqs =` comment was to stress that while the presence of `build-system.requires` inherently signals `build_wheel` support, it *does not* signal `build_sdist` support.
6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported.
7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself).
The TOML requires aren't really useful as a signal about whether sdist specifically is supported. Plus I think we probably want to leave no-requires-in-TOML as a valid option for saying "I don't need anything installed" (maybe because the backend is shipped inside the source tree) rather than overloading it to have extra meanings.
The spec currently states that the `requires` key is mandatory, so `build_wheel` will unconditionally signal readiness. I was more referring to if we decided to add a separate key, like `sdist-requires`, to statically signal `build_sdist` readiness via TOML. You could use such a key to pull in a plugin for the chosen build backend, such as a hypothetical `flit.sdist` package, instead of baking the support into flit.
Even without the addition of an `sdist-requires` TOML key, build backends can always signal sdist support dynamically by defining a basic `get_requires_for_build_sdist` returning an empty list.
The most basic build backend is still one that defines only `build_wheel` and nothing else. If it also defines `build_sdist`, well great, but pip and friends will not use it unless it also defines an appropriate `get_requires_for_build_sdist`. This is an extremely low bar to me and achieves commonality in signature and behavior across all get_requires_* and build_* hooks.
Also, as I want to avoid focusing too much on the prospect of introducing a new TOML key for sdist requirements (which is likely unnecessary) flit could just as well use `get_requires_for_build_sdist` to dynamically decide if sdist creation is possible with something like: def get_requires_for_build_sdist(source_dir, ...): try: import flit.sdist except ImportError: return None requires = flit.sdist.get_requires_for_vcs_checkout_or_signal_unsupported(source_dir) return requires This `pyproject.toml` supports sdists: [build-system] requires = ["flit", "flit.sdist"] This `pyproject.toml` only supports wheels: [build-system] requires = ["flit"] -- C Anthony
On Aug 26, 2017 5:13 PM, "Nathaniel Smith" <njs@pobox.com> wrote: On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger <c@anthonyrisinger.com> wrote:
On Aug 26, 2017 2:17 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
[removed Guido from CC]
On Aug 26, 2017 02:29, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 03:17, Guido van Rossum <guido@python.org> wrote:
In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.)
So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None.
Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do.
Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not
at
all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes.
Why does the frontend need to know why an sdist was not created?
This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it). But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else. It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-) Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written. What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake. Consider instead, if we make the following small changes: 1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary. 2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty). 3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support). 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static) 6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported. 7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself). This arrangement allows pip to know ahead-of-time if `build_sdist` is appropriate. If no sdist reqs are explicitly acknowledged, then no sdist can be created. Full stop. Example usages: * Backeds that only support wheel creation do not implement `get_requires_for_build_sdist` at all. * Backeds that conditionally support sdist creation implement `get_requires_for_build_sdist` and return None if unsupported. This is where flit signals "impossible" from an unpacked sdist and [...] from VCS. * Backeds that always support sdist creation (setuptools) implement `get_requires_for_build_sdist` and return [...] unconditionally. So for pip's sdist -> unpack -> wheel path, it knows right away if it should build an sdist or skip right to building a wheel because the backend will inform via `get_requires_for_build_sdist` instead of overloading `build_sdist`. I also think this achieves nice parity across sdist and wheel operations, keeps the same hooks optional, adds no special cases, and maybe even less overhead. What say you? -- C Anthony
On 26 August 2017 at 23:13, Nathaniel Smith <njs@pobox.com> wrote:
This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist
While I can't speak for Donald or the other pip devs, I'd like to point out that in https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html I confirmed that I'm basically persuaded now that pip should just "trust the backend" and use build_wheel directly. So this whole debate isn't relevant for pip. We'll need to wait for other currently-hypothetical frontends like tox or twine which might use the build_wheel hook, before we'll have any practical experience to confirm if there's an issue with "return None" as the way for backends to signal that they weren't able to build a wheel. Paul
Return none makes sense, but I don't think the outcome of that or the "configurable src path" decisions will affect the success of the proposal. On Sun, Aug 27, 2017, 05:59 Paul Moore <p.f.moore@gmail.com> wrote:
On 26 August 2017 at 23:13, Nathaniel Smith <njs@pobox.com> wrote:
This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist
While I can't speak for Donald or the other pip devs, I'd like to point out that in https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html I confirmed that I'm basically persuaded now that pip should just "trust the backend" and use build_wheel directly. So this whole debate isn't relevant for pip. We'll need to wait for other currently-hypothetical frontends like tox or twine which might use the build_wheel hook, before we'll have any practical experience to confirm if there's an issue with "return None" as the way for backends to signal that they weren't able to build a wheel.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sun, Aug 27, 2017 at 2:59 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist
... I'd like to point out that in https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html I confirmed that I'm basically persuaded now that pip should just "trust the backend" and use build_wheel directly. So this whole debate isn't relevant for pip.
Nor should it be for any other system. Before I saw this post I was about to make a point about not contorting ourselves to make things works the same way as pip and setuptools currently work -- pip was kind of hacked together to support setuptools, and setuptools grew kinda-organically from distutils (while also shoehorning in other not-really-building functionality). So let's make the "new" API clean and adapt the current tools to that. Specifically, I'm really confused about the sdist concept -- sure, it's a nice idea to have a standard way to create a source distributions, and I can see that tools that, for instance, create uploads for PyPi and the like will want to do that, but I don't see it as an inherent part of building a binary distribution (i.e. wheel), or building and installing a package directly. So: If you want a wheel, the build system should be asked to build a wheel. If you want an sdist, the build system should be asked to build an sdist -- and it can politely decline. Whether the build system first makes an sdist, and then builds a wheel from that it entirely up to the build system. Is there any reason to make this more complicated? We'll need to wait for other
currently-hypothetical frontends like tox or twine which might use the build_wheel hook, before we'll have any practical experience to confirm if there's an issue with "return None" as the way for backends to signal that they weren't able to build a wheel.
Sure -- though I'm with Nathaniel on this one (I think -- kinda hard to keep th threads straight!) getting a NOne can happen for alot of reasons, better to have a more information-rich way to know that something failed. And isn't "be able to build a wheel" a requirement for ANY supported back-end? In which case, failure to build one is a failure that could have been caused by any number of issues -- shouldn't it simply fail with an Exception -- and the back end will hopefully generate nice exceptions for the common failures? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 28 August 2017 at 17:25, Chris Barker <chris.barker@noaa.gov> wrote:
Nor should it be for any other system. Before I saw this post I was about to make a point about not contorting ourselves to make things works the same way as pip and setuptools currently work -- pip was kind of hacked together to support setuptools, and setuptools grew kinda-organically from distutils (while also shoehorning in other not-really-building functionality).
So let's make the "new" API clean and adapt the current tools to that.
Specifically, I'm really confused about the sdist concept -- sure, it's a nice idea to have a standard way to create a source distributions, and I can see that tools that, for instance, create uploads for PyPi and the like will want to do that, but I don't see it as an inherent part of building a binary distribution (i.e. wheel), or building and installing a package directly.
This is *precisely* why I remain uncomfortable about the "just call build_wheel" approach. I agree (as I said) that going sdist->wheel in theory should make no difference, assuming that the backend follows the PEP. And so I'm in that sense persuaded that pip doesn't need to go sdist->wheel. However, I am *strongly* against allowing backends to skip the concept of a sdist altogether. Sufficiently so that I refused to use or support flit prior to it gaining support for building sdists. There's a problem of timing at the moment, in that we have not yet defined a new sdist format, so a sdist is currently loosely defined based on "what distutils makes, with some adhoc modifications to incorporate pyproject.toml. That's deliberate - we wanted to avoid PEP 517 getting bogged down in "sdist 2.0" discussions. But none of that matters to me - Python is an open source project, and a packaging infrastructure that isn't built around publishing sources is not something I want to support. And part of that is that we should be expecting build systems to support source publication, and expect projects to publish sources on PyPI (I do *not* consider a link to a project's github site, or other source repository, as sufficient - people can close their github accounts, projects can be abandoned - but PyPI should *always* contain what's needed to reproduce the distributions on there - and that means sources). So while I concede that pip might not *need* to build wheels via sdist, that does *not* mean I am in any way in favour of omitting "build a sdist" from the PEP 517 spec. I'm uncomfortable enough with allowing backends to have the ability to build a wheel when they can't build a sdist (and yes, that means I have reservations about how flit supports sdists). But for me, having a hook to build a sdist is non-negotiable. We've had debates in the past about whether building a sdist is in scope for the PEP - I do *not* want the discussion to go back to the point where we open up that question again. Paul
participants (12)
-
Brett Cannon -
C Anthony Risinger -
Chris Barker -
Daniel Holth -
Donald Stufft -
Greg Ewing -
Guido van Rossum -
Nathaniel Smith -
Paul Moore -
Phil Austin -
Thomas Kluyver -
xoviat