A smaller step towards de-specializing setuptools/distutils
Hello! So I've been reading the various threads (well trying to, there's a lot going on in all of them) and I'm trying to reconile them all in my head and sorting out how things are going to look if we add them. I'm worried about adding more complication to an already underspecified format and making it a lot harder for tooling to work with these things. I'm wondering if there isn't a smaller step we can take towards allowing better interopability with diffrent build systems while also making the situation clearer when someone is still using distutils or setuptools. For the sake of argument, let's say we do something like this: Standardize a key in setup.cfg that can be used for setup_requires. This will be a static list of items which need to be installed in order to execute the setup.py. Unlike the current key (which is inside of setup.py) you don't have to deal with problems of trying to defer the import of the items or dealing with setuptools at all since your setup.py will not be invoked until after those items are installed. For people who want to continue to use setuptools, this file is trivial to parse with the standard library, so they can actually parse it and stick it into a setup_requires as well which will have the benefit that the tool calling the setup.py can handle those dependencies but still have it working on older versions of those tools without requiring duplication. This can also help work to make setuptools less special. Right now you basically have to just manually install setuptools prior to installing anything that uses it (or at all for something like pip when installing from sdist). With this we could pretty easily make it so that the rule is something like "If there is a setup.cfg:setup_requires, then assume that everything that is needed to execute the setup.py is listed there, else assume an implicit 'setuptools' requirement". This means that end users won't need to have setuptools installed at all, because we'll treat it (mostly) like just another build tool. This would also (for the first time) make it possible for things to depend on a specific version of setuptools instead of having to support every version of setuptools ever. I think that could solve the problem of "bootstrapping" the requirements to execute a setup.cfg and extract that from being implicitly handled by setuptools. That still doesn't handle the problem of making it possible to actually invoke the now installed build system. I think that what we should do is figure out the minimal interface we need from a ``setup.py`` based on what already exists. This would be a much smaller API surface than what exists today and wouldn't (ideally) include anything that is an implementation detail of setuptools. We would also need to standard on what flags an arguments the various commands of setup.py MUST accept. This would of course not require a setup.py to implement _only_ that particular interface so additional commands that setuptools already provide can stay, they just won't be required. Off the top of my head, I think we'd probably want to have the ``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need something like ``egg_info``, but I think we could probably diverge a bit from what setuptools does now and make it ``dist_info`` so it's backed by the same standard that Wheel is already. I think that these four commands can support everything we actually _need_ to support. This isn't everything we actually use today, but since this would be opt-in, we can actually be more restrictive about what build/installation tools would call and what level of fallback we need to support for them. The way it would work when what we have available is a sdist is something like this: We download a sdist and we notice that there is a setup.cfg:setup_requires. This toggles a stricter mode of operation where we no longer attempt to do as many fallbacks or hacks to try and work around various broken shims with setuptools. We read this key and install those items needed to execute the setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` and build a wheel from that [1]. Once we have a wheel built, we'll feed that data back into the resolver [2] and use the runtime dependency information from within that wheel to continue resolving the dependencies. OR We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is not being installed in editable. In this case we'll call ``setup.py sdist`` first, then feed that into the above. OR We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is being installed as an editable. In this case, we'll call ``setup.py build --inplace`` first, then do something to ensure that the inplace directory is on sys.path. This is currently undefined because I don't know exactly what we'd need to do to make this work, but I think it should be possible and will be more consistent. We'll probably also need something like ``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards compat with removing setuptools specifics is something we'd need to figure out. So why do I think something like this would be better? * It uses interfaces that are already there (for the most part) which means that it's easier for people to adapt their current tooling to it, and to do it in a way that will work with existing legacy packages. * It defines an interface that build systems must adhere too, instead of defining an interface that lets you query for how you actually interact with the build system (sort of a plugin system). This removes indirection and in my experience, interfaces are generally less problematic then "plugins". * It doesn't drastically change the mechanics of how the underlying thing works so it's a much smaller iteration on existing concepts rather than throwing everything out and starting from scratch. * It moves us towards an overall installation "path" that I think will be benefical for us and will help to reduce the combinatorial explosion of ways that a set of dependencies can be installed. * It will be easier to integrate setuptools into this new system (since it largely already implements it) which means that it's easier for us to move projects to upgrade piece meal to this new system since it would only require dropping a ``setup.cfg`` with a ``setup_requires``. This also means we could adjust setuptools (like adding a dist_info command) and have the implicit setup.cfg:setup_requires be 'setuptools>=somever' instead of just 'setuptools'. What this doesn't solve: * This essentially doesn't solve any of the dynamic vs static metadata issues with the legacy sdist format, but that's OK because it's just a small layering (a new setup.cfg feature) of a new feature combined with just standardizing what already exists. Drastically altering the sdist format to try and shoe-horn static metadata in there is probably not something that is going to work well in practice (and needs a more thought out, newer format). * Dynamically specified _build_ requirements (or really, build requirements at all other than via setup_requires). You can sort of kludge dynamically specified build requirements by making a sort of meta-package that you put in your static setup_requires that generats dynamic runtime requirements. I think keeping this step simple though and work on enabling breaking the dependency on setuptools/distutils in this step and then waiting to see if that is "enough" or if we need to layer on additional features (like dynamic or otherwise seperately declared build requirements). [1] This isn't related to the Wheel cache. In this stricter mode we would only ever build a wheel and install from that. If caching is on then we'll save that wheel and reuse it next time, if caching is not on then we'll just throw that wheel away at the end of the run. [2] Pretending for a minute that we have a real resolver. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I think that would be very handy, especially making setuptools not a special case. You could get it down to 3 lines in setup.cfg, a file that already exists and already gathers random settings. Case 1, pip both requires a single package, and looks for an entry point inside that package to build. [bootstrap] build-with:buildamajig buildamajig package: class Builder: def metadata(target="directory"): ... def wheel(): ... In buildamajig's setup.py: entry_points = { "pip.builder" : "builder = buildamajig:Builder" } Case 2 [bootstrap] requires=pbr pip makes sure pbr is importable inside setup.py, but uses setuptools to build the package. (Implicit build-with=pip_setuptools_adapter). Case 3 [bootstrap] build-with=buildtool:Foobar pip does not require a dependency, but noticing the : in the argument executes "from buildtool import Foobar ..." with the setup.cfg directory on sys.path; alternative: different key "builder=module:Object". Useful for embedded build scripts or anywhere entry points would be impractical. Case 4 [bootstrap] requires=random-dependency build-with=thing:Klass Legal to add requirements whether or not setup.py is used. On Thu, Oct 29, 2015 at 1:32 PM Donald Stufft <donald@stufft.io> wrote:
Hello!
So I've been reading the various threads (well trying to, there's a lot going on in all of them) and I'm trying to reconile them all in my head and sorting out how things are going to look if we add them. I'm worried about adding more complication to an already underspecified format and making it a lot harder for tooling to work with these things.
I'm wondering if there isn't a smaller step we can take towards allowing better interopability with diffrent build systems while also making the situation clearer when someone is still using distutils or setuptools. For the sake of argument, let's say we do something like this:
Standardize a key in setup.cfg that can be used for setup_requires. This will be a static list of items which need to be installed in order to execute the setup.py. Unlike the current key (which is inside of setup.py) you don't have to deal with problems of trying to defer the import of the items or dealing with setuptools at all since your setup.py will not be invoked until after those items are installed. For people who want to continue to use setuptools, this file is trivial to parse with the standard library, so they can actually parse it and stick it into a setup_requires as well which will have the benefit that the tool calling the setup.py can handle those dependencies but still have it working on older versions of those tools without requiring duplication.
This can also help work to make setuptools less special. Right now you basically have to just manually install setuptools prior to installing anything that uses it (or at all for something like pip when installing from sdist). With this we could pretty easily make it so that the rule is something like "If there is a setup.cfg:setup_requires, then assume that everything that is needed to execute the setup.py is listed there, else assume an implicit 'setuptools' requirement". This means that end users won't need to have setuptools installed at all, because we'll treat it (mostly) like just another build tool. This would also (for the first time) make it possible for things to depend on a specific version of setuptools instead of having to support every version of setuptools ever.
I think that could solve the problem of "bootstrapping" the requirements to execute a setup.cfg and extract that from being implicitly handled by setuptools. That still doesn't handle the problem of making it possible to actually invoke the now installed build system.
I think that what we should do is figure out the minimal interface we need from a ``setup.py`` based on what already exists. This would be a much smaller API surface than what exists today and wouldn't (ideally) include anything that is an implementation detail of setuptools. We would also need to standard on what flags an arguments the various commands of setup.py MUST accept. This would of course not require a setup.py to implement _only_ that particular interface so additional commands that setuptools already provide can stay, they just won't be required. Off the top of my head, I think we'd probably want to have the ``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need something like ``egg_info``, but I think we could probably diverge a bit from what setuptools does now and make it ``dist_info`` so it's backed by the same standard that Wheel is already. I think that these four commands can support everything we actually _need_ to support. This isn't everything we actually use today, but since this would be opt-in, we can actually be more restrictive about what build/installation tools would call and what level of fallback we need to support for them.
The way it would work when what we have available is a sdist is something like this:
We download a sdist and we notice that there is a setup.cfg:setup_requires. This toggles a stricter mode of operation where we no longer attempt to do as many fallbacks or hacks to try and work around various broken shims with setuptools. We read this key and install those items needed to execute the setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` and build a wheel from that [1]. Once we have a wheel built, we'll feed that data back into the resolver [2] and use the runtime dependency information from within that wheel to continue resolving the dependencies.
OR
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is not being installed in editable. In this case we'll call ``setup.py sdist`` first, then feed that into the above.
OR
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is being installed as an editable. In this case, we'll call ``setup.py build --inplace`` first, then do something to ensure that the inplace directory is on sys.path. This is currently undefined because I don't know exactly what we'd need to do to make this work, but I think it should be possible and will be more consistent. We'll probably also need something like ``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards compat with removing setuptools specifics is something we'd need to figure out.
So why do I think something like this would be better?
* It uses interfaces that are already there (for the most part) which means that it's easier for people to adapt their current tooling to it, and to do it in a way that will work with existing legacy packages.
* It defines an interface that build systems must adhere too, instead of defining an interface that lets you query for how you actually interact with the build system (sort of a plugin system). This removes indirection and in my experience, interfaces are generally less problematic then "plugins".
* It doesn't drastically change the mechanics of how the underlying thing works so it's a much smaller iteration on existing concepts rather than throwing everything out and starting from scratch.
* It moves us towards an overall installation "path" that I think will be benefical for us and will help to reduce the combinatorial explosion of ways that a set of dependencies can be installed.
* It will be easier to integrate setuptools into this new system (since it largely already implements it) which means that it's easier for us to move projects to upgrade piece meal to this new system since it would only require dropping a ``setup.cfg`` with a ``setup_requires``. This also means we could adjust setuptools (like adding a dist_info command) and have the implicit setup.cfg:setup_requires be 'setuptools>=somever' instead of just 'setuptools'.
What this doesn't solve:
* This essentially doesn't solve any of the dynamic vs static metadata issues with the legacy sdist format, but that's OK because it's just a small layering (a new setup.cfg feature) of a new feature combined with just standardizing what already exists. Drastically altering the sdist format to try and shoe-horn static metadata in there is probably not something that is going to work well in practice (and needs a more thought out, newer format).
* Dynamically specified _build_ requirements (or really, build requirements at all other than via setup_requires). You can sort of kludge dynamically specified build requirements by making a sort of meta-package that you put in your static setup_requires that generats dynamic runtime requirements. I think keeping this step simple though and work on enabling breaking the dependency on setuptools/distutils in this step and then waiting to see if that is "enough" or if we need to layer on additional features (like dynamic or otherwise seperately declared build requirements).
[1] This isn't related to the Wheel cache. In this stricter mode we would only ever build a wheel and install from that. If caching is on then we'll save that wheel and reuse it next time, if caching is not on then we'll just throw that wheel away at the end of the run.
[2] Pretending for a minute that we have a real resolver.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On October 29, 2015 at 2:54:19 PM, Daniel Holth (dholth@gmail.com) wrote:
I think that would be very handy, especially making setuptools not a special case. You could get it down to 3 lines in setup.cfg, a file that already exists and already gathers random settings.
<snip>
I don’t think we should try to use an entry point (at least not yet). One of the key points is to minimize the conceptual churn and to move forward slowly while still progressing. I don’t think that using a class really buys us much because anyone who is capable of writing that class is also capable of writing a small CLI interface to be used within a setup.py. It feels more like churn for churn’s sake than anything else and some sort of idealogical purity over pragmatism. I also don’t want to base any standard on setuptools entry points because they are not a standard (and if we did, the key should not mention pip at all, pip shouldn’t be special cased any more than setuptools should). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Thu, 29 Oct 2015 at 12:03 Donald Stufft <donald@stufft.io> wrote:
On October 29, 2015 at 2:54:19 PM, Daniel Holth (dholth@gmail.com) wrote:
I think that would be very handy, especially making setuptools not a special case. You could get it down to 3 lines in setup.cfg, a file that already exists and already gathers random settings.
<snip>
I don’t think we should try to use an entry point (at least not yet). One of the key points is to minimize the conceptual churn and to move forward slowly while still progressing. I don’t think that using a class really buys us much because anyone who is capable of writing that class is also capable of writing a small CLI interface to be used within a setup.py. It feels more like churn for churn’s sake than anything else and some sort of idealogical purity over pragmatism.
There is also the familiarity of standardizing the CLI that pip will use when calling setup.py. The one thing that Daniel's case 3 proposal has over Donald's, though, is that all data can be self-contained in setup.cfg. If a build tool chooses to have the user specify all data in setup.cfg then you just copy-and-paste the `[bootstrap]` part and then in some other section they specify the info the build tool wants. Donald's approach could do something similar but it also requires copying around a boilerplate setup.py that is never modified. It's a familiarity vs. simplicity issue. Personally I still like the simple setup.py approach for the familiarity of it as well as the backwards-compatibility/transition bit being so simple with older versions of setuptools.
I also don’t want to base any standard on setuptools entry points because they are not a standard (and if we did, the key should not mention pip at all, pip shouldn’t be special cased any more than setuptools should).
Well, it kind of is a standard thanks to Paul and zipapp: https://docs.python.org/3/library/zipapp.html#module-zipapp
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On October 29, 2015 at 3:23:34 PM, Brett Cannon (brett@python.org) wrote:
On Thu, 29 Oct 2015 at 12:03 Donald Stufft wrote:
On October 29, 2015 at 2:54:19 PM, Daniel Holth (dholth@gmail.com) wrote:
I think that would be very handy, especially making setuptools not a special case. You could get it down to 3 lines in setup.cfg, a file that already exists and already gathers random settings.
I don’t think we should try to use an entry point (at least not yet). One of the key points is to minimize the conceptual churn and to move forward slowly while still progressing. I don’t think that using a class really buys us much because anyone who is capable of writing that class is also capable of writing a small CLI interface to be used within a setup.py. It feels more like churn for churn’s sake than anything else and some sort of idealogical purity over pragmatism.
There is also the familiarity of standardizing the CLI that pip will use when calling setup.py.
The one thing that Daniel's case 3 proposal has over Donald's, though, is that all data can be self-contained in setup.cfg. If a build tool chooses to have the user specify all data in setup.cfg then you just copy-and-paste the `[bootstrap]` part and then in some other section they specify the info the build tool wants. Donald's approach could do something similar but it also requires copying around a boilerplate setup.py that is never modified. It's a familiarity vs. simplicity issue.
Personally I still like the simple setup.py approach for the familiarity of it as well as the backwards-compatibility/transition bit being so simple with older versions of setuptools.
My thoughts are basically: * Defining the standard commands has benefits for the current ecosystem as well as the future (much like why PEP 503 defined the /simple/ interface even though I plan to make a new one in the future). * We can always add the build-with key at a later point because adding things is much much easier than removing things so I think being conservative and taking smaller steps is better than taking medium steps.
I also don’t want to base any standard on setuptools entry points because they are not a standard (and if we did, the key should not mention pip at all, pip shouldn’t be special cased any more than setuptools should).
Well, it kind of is a standard thanks to Paul and zipapp: https://docs.python.org/3/library/zipapp.html#module-zipapp
I’m not sure I follow… Where is the setuptools entry point there? I only see the foo.bar:Whatever syntax, which isn’t what I meant, I meant the actual feature of setuptools entry points which you need to use something like pkg_resources to fetch.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 October 2015 at 19:23, Brett Cannon <brett@python.org> wrote:
I also don’t want to base any standard on setuptools entry points because they are not a standard (and if we did, the key should not mention pip at all, pip shouldn’t be special cased any more than setuptools should).
Well, it kind of is a standard thanks to Paul and zipapp: https://docs.python.org/3/library/zipapp.html#module-zipapp
Wait, what? That doesn't use setuptools entry points... Paul
I find CLIs to be very difficult to implement, should I bootstrap require click? How does my bundled-in-sdist build system code generate executable wrappers for Windows? Plus on the command line all method signatures suck: metadata(arguments: list of strings in any order) wheel(arguments: list of strings in any order) It could be a module instead of a class but a method/function interface would have nice documented method signatures: def wheel(target: string): ... you'd copy the example into your new build system adapter, implement, and be done. Perhaps pip could also call more than one command per subprocess more easily if the need arose. If you want your def wheel() to shell out go ahead... I could live without the entry points, it adds one line to setup.cfg in my strawman. But we are doomed to standardize setuptools-compatible entry points eventually. On Thu, Oct 29, 2015 at 3:02 PM Donald Stufft <donald@stufft.io> wrote:
On October 29, 2015 at 2:54:19 PM, Daniel Holth (dholth@gmail.com) wrote:
I think that would be very handy, especially making setuptools not a special case. You could get it down to 3 lines in setup.cfg, a file that already exists and already gathers random settings.
<snip>
I don’t think we should try to use an entry point (at least not yet). One of the key points is to minimize the conceptual churn and to move forward slowly while still progressing. I don’t think that using a class really buys us much because anyone who is capable of writing that class is also capable of writing a small CLI interface to be used within a setup.py. It feels more like churn for churn’s sake than anything else and some sort of idealogical purity over pragmatism.
I also don’t want to base any standard on setuptools entry points because they are not a standard (and if we did, the key should not mention pip at all, pip shouldn’t be special cased any more than setuptools should).
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On October 29, 2015 at 3:40:04 PM, Daniel Holth (dholth@gmail.com) wrote:
I find CLIs to be very difficult to implement, should I bootstrap require click? How does my bundled-in-sdist build system code generate executable wrappers for Windows? Plus on the command line all method signatures suck:
metadata(arguments: list of strings in any order) wheel(arguments: list of strings in any order)
It could be a module instead of a class but a method/function interface would have nice documented method signatures:
def wheel(target: string): ...
you'd copy the example into your new build system adapter, implement, and be done. Perhaps pip could also call more than one command per subprocess more easily if the need arose. If you want your def wheel() to shell out go ahead…
Sure, your build tool can require click, or it can use argparse or anything it wants. It’s just Python and you no longer need to worry about trying to defer imports, so it becomes just another Python script at that point. Of course, since your build system can have it’s own dependencies now, if people really want a class based API like that it’d only take one person to make a wrapper that implements the CLI based build interface and layers on your build-with key. So if projects want to make a class based interface it’s pretty easy to make that possible. Ideally they’d use a custom section in the setup.cfg like: [cli2class] build-with = foobar:Whatever Like I said before, it’s easy to add stuff, hard to take it away. If someone did that and it worked out well, we could then add official support for that and remove the need to have the shim package without breaking things for anyone who used the shim package. Having the current interface defined is still then useful for everyone else.
I could live without the entry points, it adds one line to setup.cfg in my strawman. But we are doomed to standardize setuptools-compatible entry points eventually.
I’m sure we will end up standardizing setuptools-compatible entry points at some point, but we don’t need to do that right now. Small incremental changes that enable things to move forward without locking us into supporting something that has been largely untested. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On October 29, 2015 at 5:01:32 PM, Marcus Smith (qwcode@gmail.com) wrote:
* This essentially doesn't solve any of the dynamic vs static metadata issues
Nathaniel's grand plan requires dynamic run-time dependencies, so to be clear, this plan doesn't help the case that instigated most of the recent discussion, right?
It should still solve that problem because it allows a project to dynamically decide what dependencies they have (just like they can today) when they create the Wheel (but the Wheel itself must have static dependencies). I believe that still handles the case that Nathaniel needed solved. It just doesn’t solve our (pip/PyPI) desires to have static (or more static) metadata. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Nathaniel's grand plan requires dynamic run-time dependencies, so to be clear, this plan doesn't help the case that instigated most of the recent discussion, right?
It should still solve that problem because it allows a project to dynamically decide what dependencies they have (just like they can today)
help me out here... how can we dynamically construct dependencies as we're building wheels today?
When setup.py builds the wheel it can write any dependencies. On Thu, Oct 29, 2015, 17:09 Marcus Smith <qwcode@gmail.com> wrote:
Nathaniel's grand plan requires dynamic run-time dependencies, so to be
clear, this plan doesn't help the case that instigated most of the recent discussion, right?
It should still solve that problem because it allows a project to dynamically decide what dependencies they have (just like they can today)
help me out here... how can we dynamically construct dependencies as we're building wheels today?
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On October 29, 2015 at 5:09:45 PM, Marcus Smith (qwcode@gmail.com) wrote:
Nathaniel's grand plan requires dynamic run-time dependencies, so to be clear, this plan doesn't help the case that instigated most of the recent discussion, right?
It should still solve that problem because it allows a project to dynamically decide what dependencies they have (just like they can today)
help me out here... how can we dynamically construct dependencies as we're building wheels today?
I’m not sure I understand the confusion… since a wheel is created by executing setup.py, you’d just have your build tool dynamically output different wheels based on the system you’re building on (or whatever axis is causing the dynamic dependencies). An example is like: https://github.com/pypa/twine/blob/a0c87357d9d5d588082c9a59f6efc6f6bc3d3498/... ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
since a wheel is created by executing setup.py, you’d just have your build tool dynamically output different wheels based on the system you’re building on (or whatever axis is causing the dynamic dependencies).
understood, but I guess I was under the [mistaken?] impression, that dynamic support is needed literally in the wheel build step. for example, going with your idea above, that we need to support 3rd party tools being dynamic in the "setup.py bdist_wheel" step, not simply in the "setup.py dist-info" step. and as it is, pip does't understand this possibility.
On October 29, 2015 at 6:32:21 PM, Marcus Smith (qwcode@gmail.com) wrote:
since a wheel is created by executing setup.py, you’d just have your build tool dynamically output different wheels based on the system you’re building on (or whatever axis is causing the dynamic dependencies).
understood, but I guess I was under the [mistaken?] impression, that dynamic support is needed literally in the wheel build step.
for example, going with your idea above, that we need to support 3rd party tools being dynamic in the "setup.py bdist_wheel" step, not simply in the "setup.py dist-info" step.
and as it is, pip does't understand this possibility.
So in the new path, we wouldn’t call ``setup.py dist-info`` for “normal” installations, we’d compile a wheel and inspect that for dependencies. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
for example, going with your idea above, that we need to support 3rd party tools being dynamic in the "setup.py bdist_wheel" step, not simply in the "setup.py dist-info" step.
and as it is, pip does't understand this possibility.
So in the new path, we wouldn’t call ``setup.py dist-info`` for “normal” installations, we’d compile a wheel and inspect that for dependencies.
ok, so a pretty significant change to pip's resolution internals.
On Thu, Oct 29, 2015 at 05:11:52PM -0400, Donald Stufft wrote:
On October 29, 2015 at 5:09:45 PM, Marcus Smith (qwcode@gmail.com) wrote:
help me out here... how can we dynamically construct dependencies as we're building wheels today?
I’m not sure I understand the confusion… since a wheel is created by executing setup.py, you’d just have your build tool dynamically output different wheels based on the system you’re building on (or whatever axis is causing the dynamic dependencies). An example is like:
https://github.com/pypa/twine/blob/a0c87357d9d5d588082c9a59f6efc6f6bc3d3498/...
install_requires = [ "pkginfo >= 1.0", "requests >= 2.3.0", "requests-toolbelt >= 0.4.0", "setuptools >= 0.7.0", ] if sys.version_info[:2] < (2, 7): install_requires += [ "argparse", ] setup( ... install_requires=install_requires, ) But code like this doesn't work! You build a wheel on Python 2.7, you get a twine-1.6.4-py2.py3-none-any.whl[*] in your pip wheel cache, and when you try to install it on Python 2.6, pip tries to use the same wheel, with install_requires computed for Python 2.7 instead of 2.6. Unless you override the wheel dependencies completely in setup.cfg[+]. [*] https://github.com/pypa/twine/blob/master/setup.cfg#L2: [wheel] universal = 1 [+] https://github.com/pypa/twine/blob/master/setup.cfg#L9-L15 [metadata] requires-dist = requests >= 2.3.0 requests-toolbelt >= 0.4.0 pkginfo >= 1.0 setuptools >= 0.7.0 argparse; python_version == '2.6' Marius Gedminas -- Since this protocol deals with Firewalls there are no real security considerations. -- RFC 3093
On October 30, 2015 at 4:07:22 AM, Marius Gedminas (marius@gedmin.as) wrote:
But code like this doesn't work! You build a wheel on Python 2.7, you get a twine-1.6.4-py2.py3-none-any.whl[*] in your pip wheel cache, and when you try to install it on Python 2.6, pip tries to use the same wheel, with install_requires computed for Python 2.7 instead of 2.6. Unless you override the wheel dependencies completely in setup.cfg[+].
It will work once https://github.com/pypa/pip/issues/3025 is fixed, which I plan on making sure is in the next release. It of course won’t work for releasing wheels or whatever, but it’s no worse then we are today. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 30 October 2015 at 08:06, Marius Gedminas <marius@gedmin.as> wrote:
But code like this doesn't work! You build a wheel on Python 2.7, you get a twine-1.6.4-py2.py3-none-any.whl[*] in your pip wheel cache, and when you try to install it on Python 2.6, pip tries to use the same wheel, with install_requires computed for Python 2.7 instead of 2.6. Unless you override the wheel dependencies completely in setup.cfg[+].
But this is simply an incorrect package definition, surely? The setup.cfg override is correct, as would be overriding the wheel to be py27.py3 (with a separate wheel depending on argparse for py26). The original form specifies a universal wheel, but provides metadata that isn't correct for a universal wheel... Maybe it's not as easy as it could be to *do* this, but nevertheless, it is possible to specify statically the correct metadata. Paul
This is a nice idea. It makes it easy to fix problematic `setup.py` files like this [1]: ``` from distutils.core import setup, Extension import numpy setup(ext_modules=[Extension("_cos_doubles", sources=["cos_doubles.c", "cos_doubles.i"], include_dirs=[numpy.get_include()])]) ``` [1] Documented here http://www.scipy-lectures.org/advanced/interfacing_with_c/interfacing_with_c... Just add a setup.cfg with `numpy` in the `setup_requires` key and done. No more `ImportError: No module named numpy` on an empty virtualenv. A small layer on top of this idea could also mean a fix to people/systems who still do `setup.py install` (Debian current tooling was mentioned). Just change the `setup.py` above to: ``` from setuptools import setup, Extension, pip_bootstrap pip_bootstrap('setup.cfg') import numpy setup(ext_modules=[Extension("_cos_doubles", sources=["cos_doubles.c", "cos_doubles.i"], include_dirs=[numpy.get_include()])]) ``` The `pip_boostrap` function would: - Install pip as if it was a `setup_requires` (i.e., it gets dumped in the local directory, or somewhere more hygienic, but not in `site-packages`), maybe using `get-pip` so we can actually remove the `install` functionality from setuptools in the future - run `pip bootstrap setup.cfg`, so that pip installs `numpy` and any other setup_requires declared in setup.cfg (dumped in the local directory like today or somewhere more hygienic, but placing them on sys.path) - proceed as usual if `pip_bootstrap` was actually a (sufficiently magical) module instead of a function, even the second line above could be omitted. Cheers, Leo On 29 October 2015 at 15:31, Donald Stufft <donald@stufft.io> wrote:
Hello!
So I've been reading the various threads (well trying to, there's a lot going on in all of them) and I'm trying to reconile them all in my head and sorting out how things are going to look if we add them. I'm worried about adding more complication to an already underspecified format and making it a lot harder for tooling to work with these things.
I'm wondering if there isn't a smaller step we can take towards allowing better interopability with diffrent build systems while also making the situation clearer when someone is still using distutils or setuptools. For the sake of argument, let's say we do something like this:
Standardize a key in setup.cfg that can be used for setup_requires. This will be a static list of items which need to be installed in order to execute the setup.py. Unlike the current key (which is inside of setup.py) you don't have to deal with problems of trying to defer the import of the items or dealing with setuptools at all since your setup.py will not be invoked until after those items are installed. For people who want to continue to use setuptools, this file is trivial to parse with the standard library, so they can actually parse it and stick it into a setup_requires as well which will have the benefit that the tool calling the setup.py can handle those dependencies but still have it working on older versions of those tools without requiring duplication.
This can also help work to make setuptools less special. Right now you basically have to just manually install setuptools prior to installing anything that uses it (or at all for something like pip when installing from sdist). With this we could pretty easily make it so that the rule is something like "If there is a setup.cfg:setup_requires, then assume that everything that is needed to execute the setup.py is listed there, else assume an implicit 'setuptools' requirement". This means that end users won't need to have setuptools installed at all, because we'll treat it (mostly) like just another build tool. This would also (for the first time) make it possible for things to depend on a specific version of setuptools instead of having to support every version of setuptools ever.
I think that could solve the problem of "bootstrapping" the requirements to execute a setup.cfg and extract that from being implicitly handled by setuptools. That still doesn't handle the problem of making it possible to actually invoke the now installed build system.
I think that what we should do is figure out the minimal interface we need from a ``setup.py`` based on what already exists. This would be a much smaller API surface than what exists today and wouldn't (ideally) include anything that is an implementation detail of setuptools. We would also need to standard on what flags an arguments the various commands of setup.py MUST accept. This would of course not require a setup.py to implement _only_ that particular interface so additional commands that setuptools already provide can stay, they just won't be required. Off the top of my head, I think we'd probably want to have the ``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need something like ``egg_info``, but I think we could probably diverge a bit from what setuptools does now and make it ``dist_info`` so it's backed by the same standard that Wheel is already. I think that these four commands can support everything we actually _need_ to support. This isn't everything we actually use today, but since this would be opt-in, we can actually be more restrictive about what build/installation tools would call and what level of fallback we need to support for them.
The way it would work when what we have available is a sdist is something like this:
We download a sdist and we notice that there is a setup.cfg:setup_requires. This toggles a stricter mode of operation where we no longer attempt to do as many fallbacks or hacks to try and work around various broken shims with setuptools. We read this key and install those items needed to execute the setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` and build a wheel from that [1]. Once we have a wheel built, we'll feed that data back into the resolver [2] and use the runtime dependency information from within that wheel to continue resolving the dependencies.
OR
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is not being installed in editable. In this case we'll call ``setup.py sdist`` first, then feed that into the above.
OR
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is being installed as an editable. In this case, we'll call ``setup.py build --inplace`` first, then do something to ensure that the inplace directory is on sys.path. This is currently undefined because I don't know exactly what we'd need to do to make this work, but I think it should be possible and will be more consistent. We'll probably also need something like ``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards compat with removing setuptools specifics is something we'd need to figure out.
So why do I think something like this would be better?
* It uses interfaces that are already there (for the most part) which means that it's easier for people to adapt their current tooling to it, and to do it in a way that will work with existing legacy packages.
* It defines an interface that build systems must adhere too, instead of defining an interface that lets you query for how you actually interact with the build system (sort of a plugin system). This removes indirection and in my experience, interfaces are generally less problematic then "plugins".
* It doesn't drastically change the mechanics of how the underlying thing works so it's a much smaller iteration on existing concepts rather than throwing everything out and starting from scratch.
* It moves us towards an overall installation "path" that I think will be benefical for us and will help to reduce the combinatorial explosion of ways that a set of dependencies can be installed.
* It will be easier to integrate setuptools into this new system (since it largely already implements it) which means that it's easier for us to move projects to upgrade piece meal to this new system since it would only require dropping a ``setup.cfg`` with a ``setup_requires``. This also means we could adjust setuptools (like adding a dist_info command) and have the implicit setup.cfg:setup_requires be 'setuptools>=somever' instead of just 'setuptools'.
What this doesn't solve:
* This essentially doesn't solve any of the dynamic vs static metadata issues with the legacy sdist format, but that's OK because it's just a small layering (a new setup.cfg feature) of a new feature combined with just standardizing what already exists. Drastically altering the sdist format to try and shoe-horn static metadata in there is probably not something that is going to work well in practice (and needs a more thought out, newer format).
* Dynamically specified _build_ requirements (or really, build requirements at all other than via setup_requires). You can sort of kludge dynamically specified build requirements by making a sort of meta-package that you put in your static setup_requires that generats dynamic runtime requirements. I think keeping this step simple though and work on enabling breaking the dependency on setuptools/distutils in this step and then waiting to see if that is "enough" or if we need to layer on additional features (like dynamic or otherwise seperately declared build requirements).
[1] This isn't related to the Wheel cache. In this stricter mode we would only ever build a wheel and install from that. If caching is on then we'll save that wheel and reuse it next time, if caching is not on then we'll just throw that wheel away at the end of the run.
[2] Pretending for a minute that we have a real resolver.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 30 October 2015 at 06:31, Donald Stufft <donald@stufft.io> wrote: tl;dr: - requiring wheel before dependencies is a Big Deal IMO - I don't care about pypa.yaml vs setup.cfg - argue with the flit folk :) - I don't care about setup.py-as-interface vs described-by-metadata - argue with the flit folk - I *do* want someone to pick choices on the 2 previous points so we can move forward :)
Hello!
So I've been reading the various threads (well trying to, there's a lot going on in all of them) and I'm trying to reconile them all in my head and sorting out how things are going to look if we add them. I'm worried about adding more complication to an already underspecified format and making it a lot harder for tooling to work with these things.
Thats fair enough, and thanks for putting the effort in :).
I'm wondering if there isn't a smaller step we can take towards allowing better interopability with diffrent build systems while also making the situation clearer when someone is still using distutils or setuptools. For the sake of argument, let's say we do something like this:
So - my proposal ended up where it did because: - the flit folk really didn't want to have to have setup.py in their tree at all - they strongly objected to that. - setup.cfg had a similar (but weaker) objection - we obviously need a place for pip to key off of to drive things - building packages with non-trivial build times is something we *really* need to keep out of the inner loop of resolving dependencies. Building to wheel first is a *huge* potential cost, *especially* when we haven't solved the wheels-for-linux problem. I've previously (on this list, 6? 8? months backs) proposed using setup.cfg directly to do build time requirements and use that as the interface. I'm still happy doing that, so if thats the direction you and Nick (or whatever the BDFL-equivalents here) want to take, I can easily strip out the indirection involved in defining the interface to the build tool and just assume setup.py with those setuptools-like commands. The specific complexity in the interface was to avoid the emergent need to update config in many trees when flit etc do change things. Similarly, if you want to add the dependency on defining 'develop' as something pip does itself to this PEP, I can help write that PEP (which I think we should do anywhich way), and pivot from asking for develop to be done, and onto having a build inplace facility. However, the big step you're proposing that I think is fundamentally unsound is that of requiring a wheel be built before dependencies can be queried. Wheels are *expensive* to build in enough cases that I really believe we have a choice between being able to fix issue 988 or utilising wheels as the interface to get at install metadata. Remember too that the numpy ABI discussion is going to cause install dependencies to depend on the version of numpy present at build-time, so the resolver is going to have to re-evaluate the dependencies for those distributions when it selects a new numpy version: so its not even a one-time-per-package cost, its a one-time-per-arc-of-the-N!-graph-paths cost. What you're describing is in all other regards basically my PEP, so +1 sure. -Rob
On 30 October 2015 at 11:50, Robert Collins <robertc@robertcollins.net> wrote: ...
However, the big step you're proposing that I think is fundamentally unsound is that of requiring a wheel be built before dependencies can be queried. Wheels are *expensive* to build in enough cases that I really believe we have a choice between being able to fix issue 988 or utilising wheels as the interface to get at install metadata. Remember too that the numpy ABI discussion is going to cause install dependencies to depend on the version of numpy present at build-time, so the resolver is going to have to re-evaluate the dependencies for those distributions when it selects a new numpy version: so its not even a one-time-per-package cost, its a one-time-per-arc-of-the-N!-graph-paths cost.
On further thought there is one mitigating factor here - we'd have to be building numpy fully anyway for the cases where numpy is a build-dependency for things further up the stack, so even in my most paranoid world we will *anyway* have to pay that cost - but we can avoid the cost of multiple full rebuilds of C extensions built on top of numpy. I think avoiding that cost is still worth it, its just a little more into the grey area rather than being absolutely obviously worth it. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On October 29, 2015 at 6:59:00 PM, Robert Collins (robertc@robertcollins.net) wrote:
On 30 October 2015 at 11:50, Robert Collins wrote: ...
However, the big step you're proposing that I think is fundamentally unsound is that of requiring a wheel be built before dependencies can be queried. Wheels are *expensive* to build in enough cases that I really believe we have a choice between being able to fix issue 988 or utilising wheels as the interface to get at install metadata. Remember too that the numpy ABI discussion is going to cause install dependencies to depend on the version of numpy present at build-time, so the resolver is going to have to re-evaluate the dependencies for those distributions when it selects a new numpy version: so its not even a one-time-per-package cost, its a one-time-per-arc-of-the-N!-graph-paths cost.
On further thought there is one mitigating factor here - we'd have to be building numpy fully anyway for the cases where numpy is a build-dependency for things further up the stack, so even in my most paranoid world we will *anyway* have to pay that cost - but we can avoid the cost of multiple full rebuilds of C extensions built on top of numpy. I think avoiding that cost is still worth it, its just a little more into the grey area rather than being absolutely obviously worth it.
We’d only need to pay that cost on the assumption that we don’t have a Wheel cached already right? Either in the machine local cache (~/.caches/pip/wheels/) or a process local cache (temp directory?). Since opting into the new build system mandates the ability to build wheels, for these items we can always assume we can build a wheel. I mentioned it on IRC, but just for the folks who aren’t on IRC, I’m also not dead set on requiring a wheel build during resolution. I did that because we currently have a bit of a race condition since we use ``setup.py egg_info`` to query the dependencies and then we run ``setup.py bdist_wheel`` to build the thing and the dependencies are not guaranteed to be the same between the invocation of these two commands. If we moved to always building Wheels then we eliminate the race condition and we make the required interface smaller. I wouldn’t be opposed to including something like ``setup.py dist-info`` in the interface if we included an assertion stating that, given no state on the machine changes (no additional packages installed, not invoking from a different Python, etc), that the metadata produced by that command must be equal to the metadata produced in the wheel, and we can put an assertion in pip to ensure it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
(Note: I'm still traveling, so I'm not fully caught up on all the threads, this one just caught my eye) On 30 October 2015 at 00:05, Donald Stufft <donald@stufft.io> wrote:
On October 29, 2015 at 6:59:00 PM, Robert Collins (robertc@robertcollins.net) wrote:
On 30 October 2015 at 11:50, Robert Collins wrote: ...
However, the big step you're proposing that I think is fundamentally unsound is that of requiring a wheel be built before dependencies can be queried. Wheels are *expensive* to build in enough cases that I really believe we have a choice between being able to fix issue 988 or utilising wheels as the interface to get at install metadata. Remember too that the numpy ABI discussion is going to cause install dependencies to depend on the version of numpy present at build-time, so the resolver is going to have to re-evaluate the dependencies for those distributions when it selects a new numpy version: so its not even a one-time-per-package cost, its a one-time-per-arc-of-the-N!-graph-paths cost.
On further thought there is one mitigating factor here - we'd have to be building numpy fully anyway for the cases where numpy is a build-dependency for things further up the stack, so even in my most paranoid world we will *anyway* have to pay that cost - but we can avoid the cost of multiple full rebuilds of C extensions built on top of numpy. I think avoiding that cost is still worth it, its just a little more into the grey area rather than being absolutely obviously worth it.
We’d only need to pay that cost on the assumption that we don’t have a Wheel cached already right? Either in the machine local cache (~/.caches/pip/wheels/) or a process local cache (temp directory?). Since opting into the new build system mandates the ability to build wheels, for these items we can always assume we can build a wheel.
I mentioned it on IRC, but just for the folks who aren’t on IRC, I’m also not dead set on requiring a wheel build during resolution. I did that because we currently have a bit of a race condition since we use ``setup.py egg_info`` to query the dependencies and then we run ``setup.py bdist_wheel`` to build the thing and the dependencies are not guaranteed to be the same between the invocation of these two commands. If we moved to always building Wheels then we eliminate the race condition and we make the required interface smaller. I wouldn’t be opposed to including something like ``setup.py dist-info`` in the interface if we included an assertion stating that, given no state on the machine changes (no additional packages installed, not invoking from a different Python, etc), that the metadata produced by that command must be equal to the metadata produced in the wheel, and we can put an assertion in pip to ensure it.
I think this is the key. If the *core* build-and-dependency-resolution is defined in terms of: * building and caching wheels (so they get built at most once per venv, and potentially per machine) * inspecting their metadata for dependencies then "dist-info" can be introduced later as an optimisation that *just* generates the wheel metadata, without actually doing the full binary build. For wheels we're downloading, we won't need to build them, and for wheels we're installing, we'll need to build them anyway. That means we can move defining the interface for a dist-info subcommand off the critical path for decoupling the build system, and instead use the existing directory based metadata format defined for wheel files. However, I also think there's one refinement we can make that lets us drop the need for a copy-and-paste "setup.py", *without* needing to define a programmatic build system API: let setup.cfg define a module name to invoke with "python -m <module>" instead of running "setup.py". That way, instead of the static setup.py needing to be copied and pasted into each project, there's just another line in the config file like: [bootstrap] setup_requires = my_build_system setup_cli_module = my_build_system.setup_cli If setup_cli_module isn't specified, then pip will expect to find a setup.py in the project directory. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2 November 2015 at 12:45, Nick Coghlan <ncoghlan@gmail.com> wrote:
(Note: I'm still traveling, so I'm not fully caught up on all the threads, this one just caught my eye) ...
We’d only need to pay that cost on the assumption that we don’t have a Wheel cached already right? Either in the machine local cache (~/.caches/pip/wheels/) or a process local cache (temp directory?). Since opting into the new build system mandates the ability to build wheels, for these items we can always assume we can build a wheel.
I mentioned it on IRC, but just for the folks who aren’t on IRC, I’m also not dead set on requiring a wheel build during resolution. I did that because we currently have a bit of a race condition since we use ``setup.py egg_info`` to query the dependencies and then we run ``setup.py bdist_wheel`` to build the thing and the dependencies are not guaranteed to be the same between the invocation of these two commands. If we moved to always building Wheels then we eliminate the race condition and we make the required interface smaller. I wouldn’t be opposed to including something like ``setup.py dist-info`` in the interface if we included an assertion stating that, given no state on the machine changes (no additional packages installed, not invoking from a different Python, etc), that the metadata produced by that command must be equal to the metadata produced in the wheel, and we can put an assertion in pip to ensure it.
I think this is the key. If the *core* build-and-dependency-resolution is defined in terms of:
* building and caching wheels (so they get built at most once per venv, and potentially per machine) * inspecting their metadata for dependencies
then "dist-info" can be introduced later as an optimisation that *just* generates the wheel metadata, without actually doing the full binary build. For wheels we're downloading, we won't need to build them, and for wheels we're installing, we'll need to build them anyway.
... and for wheels we *might* install we'll have to build them unnecessarily. I'm very worried that folk are underestimating the cost of complex operations in a resolver context.
That means we can move defining the interface for a dist-info subcommand off the critical path for decoupling the build system, and instead use the existing directory based metadata format defined for wheel files.
We can do that anyway, if we choose to encode the existing interface rather than aim at a clean one. See https://mail.python.org/pipermail/distutils-sig/2015-October/027464.html for more discussion.
However, I also think there's one refinement we can make that lets us drop the need for a copy-and-paste "setup.py", *without* needing to define a programmatic build system API: let setup.cfg define a module name to invoke with "python -m <module>" instead of running "setup.py".
This is now back to a bootstrap API which is what my PEP describes, just with different syntax wrapped around it. You can equally take my PEP and rather than piping metadata on stdout, have it write a directory and consume that. But - my PEP is stalled on the decision between what flit folk have asked for, and what Donald is concerned about (see the mail in the link above). -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On November 1, 2015 at 6:45:16 PM, Nick Coghlan (ncoghlan@gmail.com) wrote:
However, I also think there's one refinement we can make that lets us drop the need for a copy-and-paste "setup.py", *without* needing to define a programmatic build system API: let setup.cfg define a module name to invoke with "python -m " instead of running "setup.py”.
I think we should wait on this. We can always add it later, we can’t (easily) remove it. Defining the ``setup.py`` interface like I did for the /simple/ interface has benefits even completely removed from the goal of supporting alternative build systems. Once we get the details sorted out for how it affects the world of packaging to sanely allow alternative build systems, then we can figure out what it would look like to allow invocation without a setup.py script. Defining a brand new interface is a lot harder than defining the existing interface. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Sun, Nov 1, 2015 at 8:21 PM Donald Stufft <donald@stufft.io> wrote:
On November 1, 2015 at 6:45:16 PM, Nick Coghlan (ncoghlan@gmail.com) wrote:
However, I also think there's one refinement we can make that lets us drop the need for a copy-and-paste "setup.py", *without* needing to define a programmatic build system API: let setup.cfg define a module name to invoke with "python -m " instead of running "setup.py”.
I think we should wait on this. We can always add it later, we can’t (easily) remove it. Defining the ``setup.py`` interface like I did for the /simple/ interface has benefits even completely removed from the goal of supporting alternative build systems. Once we get the details sorted out for how it affects the world of packaging to sanely allow alternative build systems, then we can figure out what it would look like to allow invocation without a setup.py script.
Defining a brand new interface is a lot harder than defining the existing interface.
One problem with setup.py is that pip doesn't like it when egg_info produces a dist-info directory. Is it impossible to define setup.py dist-info as "write current-format wheel metadata to a target directory"? Or we could just standardize the egg-info requires.txt. A plain text list of requirements, one per line, needing no extras or markers is all that pip needs at this phase. It doesn't even mind if you put that into a .dist-info directory in the target folder.
However, the big step you're proposing that I think is fundamentally unsound is that of requiring a wheel be built before dependencies can be queried.
so, Robert, to be clear, you think 3rd party build tools can get the dynamic-dependency support they need just by implementing something dynamic via "setup,py dist-info"? (and it doesn't need to happen literally in the wheel build step?)
On Thu, Oct 29, 2015 at 4:09 PM, Marcus Smith <qwcode@gmail.com> wrote:
However, the big step you're proposing that I think is fundamentally unsound is that of requiring a wheel be built before dependencies can be queried.
so, Robert, to be clear, you think 3rd party build tools can get the dynamic-dependency support they need just by implementing something dynamic via "setup,py dist-info"? (and it doesn't need to happen literally in the wheel build step?)
Sure -- 'dist-info' would be run in the same environment as bdist_wheel, a few seconds before bdist_wheel was run (unless the output from dist-info convinces us that we need a different version of this package and so we skip running bdist_wheel entirely). So at least in principle, anything bdist_wheel can do/know/figure-out is something that dist-info could also do/know/figure-out. The question about whether to provide a 'dist-info' step is whether the optimization it provides is worth the extra complexity it imposes on build systems and the opportunities for what Donald called "race conditions" (= accidental or intentional skew between the output of dist-info and the actual metadata in the eventual wheel). See also the long email I just sent in its own thread going into more detail on what this trade-off entails :-) -n -- Nathaniel J. Smith -- http://vorpus.org
so, Robert, to be clear, you think 3rd party build tools can get the dynamic-dependency support they need just by implementing something dynamic via "setup,py dist-info"? (and it doesn't need to happen literally in the wheel build step?)
Sure -- 'dist-info' would be run in the same environment as bdist_wheel, a few seconds before bdist_wheel was run (unless the output from dist-info convinces us that we need a different version of this package and so we skip running bdist_wheel entirely). So at least in principle, anything bdist_wheel can do/know/figure-out is something that dist-info could also do/know/figure-out. The question about whether to provide a 'dist-info' step is whether the optimization it provides is worth the extra complexity it imposes on build systems and the opportunities for what Donald called "race conditions" (= accidental or intentional skew between the output of dist-info and the actual metadata in the eventual wheel).
thankyou!! for the clarity : )
participants (10)
-
Brett Cannon -
Daniel Holth -
Donald Stufft -
Leonardo Rochael Almeida -
Marcus Smith -
Marius Gedminas -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
Robert Collins