This is kind-of related to the othe thread: "Remove distutils, was: ..." But more specific, so I thought I'd start a new one. Here are my thoughts: We had distutils -- there was a lot it didn't do that the "Masses" needed, so setuptools was born. It proved to be useful to a lot of people, and grew a large userbase.... But a lot was "wrong" with setuptools -- most prominently (in my mind anyway) that it put too many kind-sorta orthogonal stuff into one package: building, installing, distributing, managing version, managing dependencies, managing non-python resources, (and others??). And we didn't like how easy-install installed things :-) So distribute, and pip, and wheel, and now a new backward compatible setuptools was born. But we still have a bunch of should be orthogonal stuff tangled up together. In particular, I find that I often find easy-install getting invoked when I don't want ot to, and I get those darn eggs scattered all over the place, and easy_install.pth, and ???? I think if I am really careful about what I invoke when, this could be avoided, but the reality is that I've been dealing with this for years, and am trying desperately to do things the "right, modern" way, and I still get ugliness. I seriously doubt that I am alone. So -- here's my thought: I think we have it pretty well mapped out what functionality belongs where: one system for building packages (setuptools?) one system for installing packages and managing dependencies (pip) one system (really standard) for metadata and distributing packages (wheel) [I'm just whipping this out off the top of my head, I'm sure we'd need to more clearly define what belongs where] So why not have a setuptools-lite that only does the building? We need to keep the full over-burdened setuptools around, because lot sof folks are using those features. But for those of us that are doing something fairly new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd love to have a setuptools-lite that only did what I really need, and was guaranteed NOT to accidentally introduce easy_install, etc... This seems to me to be a way to go forward -- as it is we'll have people using setuptools features that they "shouldn't" forever, and never be able to move to a cleaner system. Or maybe a flag: import setuptools setuptools.use_only_modern() That would make the dependencies easier -- i.e. pip depends on some of setuptools being there -- hard to say that it needs either setuptools OR setuptools_lite. Maybe I'm missing something here, but if the goal is for there to be one way to do things, let's have a tool chain that only does things one way..... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
why does that have to be in setuptools ?! if we want a new light system to begin with, shouldn't it be far more sustainable to use just implementations of the new standards rather than leaving all of setuptools there is no viable way in setuptools to get rid of the legacy ina sane and fast manner, it would drag out over years and both distutils and setuptools are very very sub-par from the maintenance perspective On 10/21/2015 06:42 PM, Chris Barker wrote:
This is kind-of related to the othe thread:
"Remove distutils, was: ..."
But more specific, so I thought I'd start a new one.
Here are my thoughts:
We had distutils -- there was a lot it didn't do that the "Masses" needed, so setuptools was born. It proved to be useful to a lot of people, and grew a large userbase....
But a lot was "wrong" with setuptools -- most prominently (in my mind anyway) that it put too many kind-sorta orthogonal stuff into one package: building, installing, distributing, managing version, managing dependencies, managing non-python resources, (and others??). And we didn't like how easy-install installed things :-)
So distribute, and pip, and wheel, and now a new backward compatible setuptools was born.
But we still have a bunch of should be orthogonal stuff tangled up together. In particular, I find that I often find easy-install getting invoked when I don't want ot to, and I get those darn eggs scattered all over the place, and easy_install.pth, and ????
I think if I am really careful about what I invoke when, this could be avoided, but the reality is that I've been dealing with this for years, and am trying desperately to do things the "right, modern" way, and I still get ugliness. I seriously doubt that I am alone.
So -- here's my thought:
I think we have it pretty well mapped out what functionality belongs where:
one system for building packages (setuptools?) one system for installing packages and managing dependencies (pip) one system (really standard) for metadata and distributing packages (wheel)
[I'm just whipping this out off the top of my head, I'm sure we'd need to more clearly define what belongs where]
So why not have a setuptools-lite that only does the building? We need to keep the full over-burdened setuptools around, because lot sof folks are using those features. But for those of us that are doing something fairly new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd love to have a setuptools-lite that only did what I really need, and was guaranteed NOT to accidentally introduce easy_install, etc...
This seems to me to be a way to go forward -- as it is we'll have people using setuptools features that they "shouldn't" forever, and never be able to move to a cleaner system.
Or maybe a flag:
import setuptools setuptools.use_only_modern()
That would make the dependencies easier -- i.e. pip depends on some of setuptools being there -- hard to say that it needs either setuptools OR setuptools_lite.
Maybe I'm missing something here, but if the goal is for there to be one way to do things, let's have a tool chain that only does things one way.....
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt < opensource@ronnypfannschmidt.de> wrote:
why does that have to be in setuptools ?!
if we want a new light system to begin with, shouldn't it be far more sustainable to use just implementations of the new standards rather than leaving all of setuptools
there is no viable way in setuptools to get rid of the legacy ina sane and fast manner, it would drag out over years
agreed. I have never met a person who had to deal substantially with distutils code and enjoyed the experience. The whole architecture is fundamentally flawed. I wrote this a long time ago, but I still stand by most arguments: https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observation... David
and both distutils and setuptools are very very sub-par from the maintenance perspective
On 10/21/2015 06:42 PM, Chris Barker wrote:
This is kind-of related to the othe thread:
"Remove distutils, was: ..."
But more specific, so I thought I'd start a new one.
Here are my thoughts:
We had distutils -- there was a lot it didn't do that the "Masses" needed, so setuptools was born. It proved to be useful to a lot of people, and grew a large userbase....
But a lot was "wrong" with setuptools -- most prominently (in my mind anyway) that it put too many kind-sorta orthogonal stuff into one package: building, installing, distributing, managing version, managing dependencies, managing non-python resources, (and others??). And we didn't like how easy-install installed things :-)
So distribute, and pip, and wheel, and now a new backward compatible setuptools was born.
But we still have a bunch of should be orthogonal stuff tangled up together. In particular, I find that I often find easy-install getting invoked when I don't want ot to, and I get those darn eggs scattered all over the place, and easy_install.pth, and ????
I think if I am really careful about what I invoke when, this could be avoided, but the reality is that I've been dealing with this for years, and am trying desperately to do things the "right, modern" way, and I still get ugliness. I seriously doubt that I am alone.
So -- here's my thought:
I think we have it pretty well mapped out what functionality belongs where:
one system for building packages (setuptools?) one system for installing packages and managing dependencies (pip) one system (really standard) for metadata and distributing packages (wheel)
[I'm just whipping this out off the top of my head, I'm sure we'd need to more clearly define what belongs where]
So why not have a setuptools-lite that only does the building? We need to keep the full over-burdened setuptools around, because lot sof folks are using those features. But for those of us that are doing something fairly new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd love to have a setuptools-lite that only did what I really need, and was guaranteed NOT to accidentally introduce easy_install, etc...
This seems to me to be a way to go forward -- as it is we'll have people using setuptools features that they "shouldn't" forever, and never be able to move to a cleaner system.
Or maybe a flag:
import setuptools setuptools.use_only_modern()
That would make the dependencies easier -- i.e. pip depends on some of setuptools being there -- hard to say that it needs either setuptools OR setuptools_lite.
Maybe I'm missing something here, but if the goal is for there to be one way to do things, let's have a tool chain that only does things one way.....
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Wed, Oct 21, 2015 at 12:32 PM, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt < opensource@ronnypfannschmidt.de> wrote:
why does that have to be in setuptools ?!
if we want a new light system to begin with, shouldn't it be far more sustainable to use just implementations of the new standards rather than leaving all of setuptools
there is no viable way in setuptools to get rid of the legacy ina sane and fast manner, it would drag out over years
agreed. I have never met a person who had to deal substantially with distutils code and enjoyed the experience.
The whole architecture is fundamentally flawed. I wrote this a long time ago, but I still stand by most arguments: https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observation...
I've (luckily?) never had to deal with distutils code... I am definitely digging the post. First time I've read it, but I am definitely more pro-standard-interface-and-let-tools-do-with-it-what-they-will than I was a few minutes ago. Would pip's freeze format work a la the cabal file, or is it missing too much information? -Wayne
On Wed, Oct 21, 2015 at 6:44 PM, Wayne Werner <waynejwerner@gmail.com> wrote:
On Wed, Oct 21, 2015 at 12:32 PM, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt < opensource@ronnypfannschmidt.de> wrote:
why does that have to be in setuptools ?!
if we want a new light system to begin with, shouldn't it be far more sustainable to use just implementations of the new standards rather than leaving all of setuptools
there is no viable way in setuptools to get rid of the legacy ina sane and fast manner, it would drag out over years
agreed. I have never met a person who had to deal substantially with distutils code and enjoyed the experience.
The whole architecture is fundamentally flawed. I wrote this a long time ago, but I still stand by most arguments: https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observation...
I've (luckily?) never had to deal with distutils code... I am definitely digging the post. First time I've read it, but I am definitely more pro-standard-interface-and-let-tools-do-with-it-what-they-will than I was a few minutes ago.
Would pip's freeze format work a la the cabal file, or is it missing too much information?
the main cabal file is its own DSL that describes the package metadata (so more a replacement of setup.py). I used a similar idea when I wrote bento (https://cournape.github.io/Bento/), which ended up being a mistake. If I were to write a similar tool today, I would just use templated yaml. A big difference compared to 6-7 years ago is the progress made on pip, wheels, etc... A lot of the current efforts in python package (metadata formalization, etc...) are about enabling tools like bento, flit, etc... to coexist: as long as they produced wheels/(wheel)sdist/etc... that pip can consume, you have a clear separate of concern, and an healthy competition. David
-Wayne
We're not just grumpy, we have seen The One Build System idea tried several times with bad results. Instead, if you have the inclination, time, and ability, you could try to build a new competing build system like http://flit.readthedocs.org/ did, or you could help try to figure out how to improve support for said competing build systems in pip, or you could even try to make a pip replacement that is better for distributing end-user applications. Ideally one of these new build systems will be compelling enough that new packages will choose to use it instead of distutils in the same way that people choose Django over the cgi module. On Wed, Oct 21, 2015 at 2:08 PM Wayne Werner <waynejwerner@gmail.com> wrote:
On Wed, Oct 21, 2015 at 12:32 PM, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt < opensource@ronnypfannschmidt.de> wrote:
why does that have to be in setuptools ?!
if we want a new light system to begin with, shouldn't it be far more sustainable to use just implementations of the new standards rather than leaving all of setuptools
there is no viable way in setuptools to get rid of the legacy ina sane and fast manner, it would drag out over years
agreed. I have never met a person who had to deal substantially with distutils code and enjoyed the experience.
The whole architecture is fundamentally flawed. I wrote this a long time ago, but I still stand by most arguments: https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observation...
I've (luckily?) never had to deal with distutils code... I am definitely digging the post. First time I've read it, but I am definitely more pro-standard-interface-and-let-tools-do-with-it-what-they-will than I was a few minutes ago.
Would pip's freeze format work a la the cabal file, or is it missing too much information?
-Wayne _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Wed, 21 Oct 2015 18:29:17 +0000 Daniel Holth <dholth@gmail.com> wrote:
We're not just grumpy, we have seen The One Build System idea tried several times with bad results. Instead, if you have the inclination, time, and ability, you could try to build a new competing build system like http://flit.readthedocs.org/ did, or you could help try to figure out how to improve support for said competing build systems in pip, or you could even try to make a pip replacement that is better for distributing end-user applications.
That doesn't seem to follow at all. First sentence you're talking about a build system, second you're talking about distributing end-user applications. Building and distributing are different things. "flit" certainly does nothing for non-trivial build chains. Regards Antoine.
On 21 October 2015 at 19:32, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt <opensource@ronnypfannschmidt.de> wrote:
why does that have to be in setuptools ?!
if we want a new light system to begin with, shouldn't it be far more sustainable to use just implementations of the new standards rather than leaving all of setuptools
there is no viable way in setuptools to get rid of the legacy ina sane and fast manner, it would drag out over years
agreed. I have never met a person who had to deal substantially with distutils code and enjoyed the experience.
The whole architecture is fundamentally flawed. I wrote this a long time ago, but I still stand by most arguments: https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observation...
It's also worth reading Greg Ward's post on how distutils came to be: http://gerg.ca/blog/post/2013/distutils-history/ Something else not a lot of folks realise is that setuptools was originally built to handle development for the Open Source Application Foundation's Chandler project: https://en.wikipedia.org/wiki/Chandler_%28software%29 Neither setuptools nor distutils has particularly robust test suties, and nor do they have nicely structured public APIs, so refactoring either of them heavily is almost guaranteed to break something. OSAF's and Chandler's original needs are also far removed from the way we're using setuptools today (although the greatest of the differences relate to the differences between pip and easy_install moreso than anything on the build side). Despite those limitations, tens of thousands of software projects have been successfully published with both of them, so anyone trying to push for improvements needs to account for the popularity of pkg_resources for plugin management, the widespread use of setuptools based executable script wrappers, and the intertwined combination of distutils and setuptools that needs to be managed in order to successfully rebuild the whole of PyPI from source. Cheers, Nick. P.S. Anyone that decides to work on improving the packaging ecosystem also needs to account for the fact that they will hear relatively frequent complaints from *everybody*, as newcomers will complain "Why isn't it fixed yet?", veterans will complain "What have you broken today?", but almost nobody will offer to actually pay anyone to fix anything other than their own specific corner case, rather than investing in properly overhauling the entire system (which is as formidable a political challenge as it as a technical one). The core problem is that the status quo is merely clumsy and inelegant, rather than completely unusable. For all their flaws, both hobbyists and professional developers *can* use the current tools to distribute software (and they're still easier to deal with than writing directly for the native installer toolsets on each target platform), while in any organisation that uses Python heavily, setup scripts for new projects are likely to be either generated from templates or copied from those for old projects, rather than needing to be written from scratch each time. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 23 October 2015 at 23:12, Nick Coghlan <ncoghlan@gmail.com> wrote:
The core problem is that the status quo is merely clumsy and inelegant, rather than completely unusable. For all their flaws, both hobbyists and professional developers *can* use the current tools to distribute software (and they're still easier to deal with than writing directly for the native installer toolsets on each target platform), while in any organisation that uses Python heavily, setup scripts for new projects are likely to be either generated from templates or copied from those for old projects, rather than needing to be written from scratch each time.
And now that I've got my "Mr grumpy pants" reaction to the earlier parts of the thread out of my system (sorry about that), I'd like to revisit Chris's idea of having a *simple* replacement for setuptools as a build system. In particular, what if there was a build system aimed specifically at beginners that, at least initially, *only* supported the creation of sdist's and wheels for pure Python source publication (no "install" or "upload" commands), without support for binary extensions, and defaulted to publishing everything in the current directory and below, while relying on twine to actually do the uploads to PyPI? Folks dealing with C extensions (including via cffi or Cython), using pkg_resources for plugins, installing command line wrappers, or otherwise doing something more than publishing a pure Python library would still need to "graduate" to dealing with the complexities of setuptools or one of its competitors, but we could potentially get it out of the initial learning curve for students publishing list-of-list printers :) One particularly nice aspect of that is anyone interested in working on it could potentially pitch it to the PSF's Education Working Group as a project worth investigating as part of the Education bundle. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On October 25, 2015 at 7:17:12 PM, Nick Coghlan (ncoghlan@gmail.com) wrote:
And now that I've got my "Mr grumpy pants" reaction to the earlier parts of the thread out of my system (sorry about that), I'd like to revisit Chris's idea of having a *simple* replacement for setuptools as a build system.
I’m not *against* it, but it has some problems: * Older versions of pip and setuptools will not be able to install it at all. This includes anyplace setuptools is used implicitly such as ``setup.py install`` from another package that depends on it or setup_requires or test_requires etc. * A lot of downstream folks will probably be mad at us for introducing a second way of doing things (particularly since it won’t come with the agile build system support that we have on the road map that will make this a lot better for non setuptools build systems). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Mr. grumpy pants, Now you're trolling us by describing flit. Older versions of pip could be able to install it, just add a setup.py shim. Nearly trivial. A new simple build system could come with the agile build system support. It's just a matter of timing. On Sun, Oct 25, 2015 at 9:49 PM Donald Stufft <donald@stufft.io> wrote:
On October 25, 2015 at 7:17:12 PM, Nick Coghlan (ncoghlan@gmail.com) wrote:
And now that I've got my "Mr grumpy pants" reaction to the earlier parts of the thread out of my system (sorry about that), I'd like to revisit Chris's idea of having a *simple* replacement for setuptools as a build system.
I’m not *against* it, but it has some problems:
* Older versions of pip and setuptools will not be able to install it at all. This includes anyplace setuptools is used implicitly such as ``setup.py install`` from another package that depends on it or setup_requires or test_requires etc.
* A lot of downstream folks will probably be mad at us for introducing a second way of doing things (particularly since it won’t come with the agile build system support that we have on the road map that will make this a lot better for non setuptools build systems).
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 27 Oct 2015 13:30, "Daniel Holth" <dholth@gmail.com> wrote:
Mr. grumpy pants,
Now you're trolling us by describing flit.
I'd managed to miss Flit's creation, so I simply wasn't aware of it. Now that I've had a chance to remedy that oversight, yes, flit sounds exactly like what I meant, so it could be worth recommending it ahead of a full distutils/setuptools based setup.py for simple projects. Cheers, Nick.
On Wed, Oct 28, 2015 at 9:04 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 27 Oct 2015 13:30, "Daniel Holth" <dholth@gmail.com> wrote:
Mr. grumpy pants,
Now you're trolling us by describing flit.
I'd managed to miss Flit's creation, so I simply wasn't aware of it.
Now that I've had a chance to remedy that oversight, yes, flit sounds exactly like what I meant, so it could be worth recommending it ahead of a full distutils/setuptools based setup.py for simple projects.
IMO, flit should be officially recommended until it at least supports building sdist. Not having this feature is a significant pain point for downstream distributors, David
On October 28, 2015 at 10:55:54 AM, David Cournapeau (cournape@gmail.com) wrote:
On Wed, Oct 28, 2015 at 9:04 AM, Nick Coghlan wrote:
On 27 Oct 2015 13:30, "Daniel Holth" wrote:
Mr. grumpy pants,
Now you're trolling us by describing flit.
I'd managed to miss Flit's creation, so I simply wasn't aware of it.
Now that I've had a chance to remedy that oversight, yes, flit sounds exactly like what I meant, so it could be worth recommending it ahead of a full distutils/setuptools based setup.py for simple projects.
IMO, flit should be officially recommended until it at least supports building sdist. Not having this feature is a significant pain point for downstream distributors,
Assuming you meant “shouldn’t” I agree. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Well it sounds like we're ready to actually fix that, by defining a reasonable universal Python sdist format. On Wed, Oct 28, 2015 at 10:57 AM Donald Stufft <donald@stufft.io> wrote:
On October 28, 2015 at 10:55:54 AM, David Cournapeau (cournape@gmail.com) wrote:
On Wed, Oct 28, 2015 at 9:04 AM, Nick Coghlan wrote:
On 27 Oct 2015 13:30, "Daniel Holth" wrote:
Mr. grumpy pants,
Now you're trolling us by describing flit.
I'd managed to miss Flit's creation, so I simply wasn't aware of it.
Now that I've had a chance to remedy that oversight, yes, flit sounds exactly like what I meant, so it could be worth recommending it ahead
of a
full distutils/setuptools based setup.py for simple projects.
IMO, flit should be officially recommended until it at least supports building sdist. Not having this feature is a significant pain point for downstream distributors,
Assuming you meant “shouldn’t” I agree.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Wed, Oct 28, 2015 at 11:41 AM Daniel Holth <dholth@gmail.com> wrote:
Well it sounds like we're ready to actually fix that, by defining a reasonable universal Python sdist format.
On Wed, Oct 28, 2015 at 10:57 AM Donald Stufft <donald@stufft.io> wrote:
On October 28, 2015 at 10:55:54 AM, David Cournapeau (cournape@gmail.com) wrote:
On Wed, Oct 28, 2015 at 9:04 AM, Nick Coghlan wrote:
On 27 Oct 2015 13:30, "Daniel Holth" wrote:
Mr. grumpy pants,
Now you're trolling us by describing flit.
I'd managed to miss Flit's creation, so I simply wasn't aware of it.
Now that I've had a chance to remedy that oversight, yes, flit sounds exactly like what I meant, so it could be worth recommending it ahead
of a
full distutils/setuptools based setup.py for simple projects.
IMO, flit should be officially recommended until it at least supports building sdist. Not having this feature is a significant pain point for downstream distributors,
Assuming you meant “shouldn’t” I agree.
I feel like you are suggesting that the person who spent their time developing some software and then gave it to away, and then chose a build system that was convenient for them, is causing problems. We can prevent them from causing those problems by asking them to use the sdist file format. How much pain would we be looking at with accepting a github archive of a tag as the official source? Something like an extra debhelper script per build system?
On October 28, 2015 at 3:05:57 PM, Daniel Holth (dholth@gmail.com) wrote:
I feel like you are suggesting that the person who spent their time developing some software and then gave it to away, and then chose a build system that was convenient for them, is causing problems. We can prevent them from causing those problems by asking them to use the sdist file format. How much pain would we be looking at with accepting a github archive of a tag as the official source? Something like an extra debhelper script per build system?
No, I’m suggesting that the PyPA needs to be conservative with it’s recommendations and that we need to consider the broader ecosystem impact. Individual projects or people may have different criteria they use to determine what is an acceptable solution, but we have to walk the fine line between trying to do what’s best for many different groups of people (and we sometimes have to make things worse for X to make it better for Y). I don’t have any problems with flit, or if someone wants to use it. I have a problem with making it an official recommendation prior to the foundations being laid to make it what I consider to be a reasonable, ecosystem wide, solution. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 28 Oct 2015 20:10, "Donald Stufft" <donald@stufft.io> wrote:
No, I’m suggesting that the PyPA needs to be conservative with it’s
recommendations and that we need to consider the broader ecosystem impact. Individual projects or people may have different criteria they use to determine what is an acceptable solution, but we have to walk the fine line between trying to do what’s best for many different groups of people (and we sometimes have to make things worse for X to make it better for Y). Right. While consuming prebuilt binaries is an acceptable approach in many cases, it's not acceptable for Linux distros since it can easily lead to problems with things like reproducible builds and licensing compliance (especially for copyleft licenses with on demand source provision requirements). That said, for Fedora RPMs, we don't need sdist per se - we only need ready access to the "original sources". It's just that wheels don't count, since they're a nominally binary format, and "C ABI = None, Platform = None" in the filename isn't currently an entirely reliable indicator of a pure Python wheel file. (Even if those markers *were* entirely reliable, the "certain kinds of wheel files can be treated as if they were source archives" gets really messy conceptual - it's akin to the "8-bit text is an ill-defined subset of arbitrary 8-bit data" represented by the Python 2 str type). A HTTPS source control URL together with a commit hash *can* count, though, even in the absence of a source archive uploaded to PyPI, as the VCS information is enough for us to retrieve the original sources and put them in the SRPM. (I'm not as familiar with Debian's policies as I am with those for Fedora et al, but as far as I am aware, they want a reference to the original sources for similar reasons of build reproducibility, license compliance, and code auditability).
I don’t have any problems with flit, or if someone wants to use it. I have a problem with making it an official recommendation prior to the foundations being laid to make it what I consider to be a reasonable, ecosystem wide, solution.
Right, any recommended solution needs to provide access to the original sources *in addition to* any already built wheel files. While Linux distros could technically cope with arbitrary source trees, I think we're close enough to having a cleaner sdist format defined that we can wait until it's defined and flit produces it in addition to a wheel file before switching the official recommendation for new users. Cheers, Nick.
Nick Coghlan <ncoghlan@gmail.com> writes:
That said, for Fedora RPMs, we don't need sdist per se - we only need ready access to the "original sources".
There's a temporal element to that, too. While “ready access” to the source might be clear enough at the moment of the wheel's release, it is less clear in ten years time when the original source for a package still in Fedora is needed again. In practice, a URL to (what one hopes is) the source is not enough to provude assurance the source will be available long in the future. Only a known tarball (or equivalent fixed single-file archive form) of the actual source is going to provide that.
It's just that wheels don't count, since they're a nominally binary format, and "C ABI = None, Platform = None" in the filename isn't currently an entirely reliable indicator of a pure Python wheel file.
One needn't say that wheel is “nominally” anything; it is sufficient to ask “is this what a recipient would need to have the source in a form suitable for further modification and redistribution?” A wheel distribution is fairly clearly *not* the preferred form of the work for a recipient to have to exercise freedom to modify and redistribute. The wheel distgribution was generated from source files edited by the developer, and so *those* file are the source form of the work, the wheel distribution is not.
A HTTPS source control URL together with a commit hash *can* count, though, even in the absence of a source archive uploaded to PyPI, as the VCS information is enough for us to retrieve the original sources and put them in the SRPM.
As I point out above, that's only reliably true if it is immediately turned into a more reliably-archived form than a URL to some hosting provider somewhere. So, in practice, the URL is not enough to provide good assurance one has the source form of the work for redistribution.
(I'm not as familiar with Debian's policies as I am with those for Fedora et al, but as far as I am aware, they want a reference to the original sources for similar reasons of build reproducibility, license compliance, and code auditability).
Not merely a reference, but (as explained above) The actual source form of the work, persistent over long periods of time as the known corresponding source form of that version of the work.
Right, any recommended solution needs to provide access to the original sources *in addition to* any already built wheel files.
Yes, thanks. -- \ “Try adding “as long as you don't breach the terms of service – | `\ according to our sole judgement” to the end of any cloud | _o__) computing pitch.” —Simon Phipps, 2010-12-11 | Ben Finney
Sorry all -- on vacation and only semi-online for a while... I'd managed to miss Flit's creation, so I simply wasn't aware of it.
Now that I've had a chance to remedy that oversight, yes, flit sounds exactly like what I meant, so it could be worth recommending it ahead of a full distutils/setuptools based setup.py for simple projects.
neither did I - and it does look intriguing, though not (yet) what I had in mind. "my" idea of setuptool-lite would be a drop-in replacement for the parts of setuptools that we want to continue to support. i.e. not nearly as crippled as flit. That being said, *I* don't have any use for pkg_resources, so maybe flit is pretty close :-) but sdists are key. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Oct 23, 2015 at 2:32 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 23 October 2015 at 23:12, Nick Coghlan <ncoghlan@gmail.com> wrote:
The core problem is that the status quo is merely clumsy and inelegant, rather than completely unusable. For all their flaws, both hobbyists and professional developers *can* use the current tools to distribute software (and they're still easier to deal with than writing directly for the native installer toolsets on each target platform), while in any organisation that uses Python heavily, setup scripts for new projects are likely to be either generated from templates or copied from those for old projects, rather than needing to be written from scratch each time.
And now that I've got my "Mr grumpy pants" reaction to the earlier parts of the thread out of my system (sorry about that), I'd like to revisit Chris's idea of having a *simple* replacement for setuptools as a build system.
In particular, what if there was a build system aimed specifically at beginners that, at least initially, *only* supported the creation of sdist's and wheels for pure Python source publication (no "install" or "upload" commands), without support for binary extensions, and defaulted to publishing everything in the current directory and below, while relying on twine to actually do the uploads to PyPI?
This isn't terribly far from what flit already is. -n -- Nathaniel J. Smith -- http://vorpus.org
On Wed, Oct 21, 2015 at 10:03 AM, Ronny Pfannschmidt < opensource@ronnypfannschmidt.de> wrote:
why does that have to be in setuptools ?!
it doesn't have to be in setuptools at all -- I suppose I should have defined more clearly what I meant: setuptools_lite would be a package that does all things that setuptools does that we currently think it *should* do -- and nothing else. whether it's in setuptools as a special mode, forked from setuptools, or written from scratch is all implementation detail. perhaps it would be better with a different name -- maybe "buildtools"? But I thought for acceptance by the community, maybe saying: replace all your "import setuptools" with "import setuptool_lite" would be clear what the intent was -- i.e. not YET ANOTHER brand new build system... Oh and it would use the same API for the parts that it still supported. Given all that, if I were to do it, and I'm not sure I can... I would probably start with setuptools and rip stuff out rather than start from scratch. and both distutils and setuptools are very very sub-par
well, see the other discussion about the role / necessity of distutils... but yes, it's certainly possible to make a new build system completely independent of distutils. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
i would very strongly suggest to completely avoid distutils as base for a new tool being based on distutils is one of the main reasons why setuptools is so painful to develop/maintain, its always a huge mental cost and pain to work into that huge pile of historic spaghetti and its even more pain to come up with useful tests -- Ronny On 10/21/2015 09:05 PM, Chris Barker wrote:
On Wed, Oct 21, 2015 at 10:03 AM, Ronny Pfannschmidt <opensource@ronnypfannschmidt.de <mailto:opensource@ronnypfannschmidt.de>> wrote:
why does that have to be in setuptools ?!
it doesn't have to be in setuptools at all -- I suppose I should have defined more clearly what I meant:
setuptools_lite would be a package that does all things that setuptools does that we currently think it *should* do -- and nothing else.
whether it's in setuptools as a special mode, forked from setuptools, or written from scratch is all implementation detail.
perhaps it would be better with a different name -- maybe "buildtools"?
But I thought for acceptance by the community, maybe saying:
replace all your "import setuptools" with "import setuptool_lite" would be clear what the intent was -- i.e. not YET ANOTHER brand new build system...
Oh and it would use the same API for the parts that it still supported. Given all that, if I were to do it, and I'm not sure I can... I would probably start with setuptools and rip stuff out rather than start from scratch.
and both distutils and setuptools are very very sub-par
well, see the other discussion about the role / necessity of distutils... but yes, it's certainly possible to make a new build system completely independent of distutils.
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>
On October 21, 2015 at 3:13:11 PM, Chris Barker (chris.barker@noaa.gov) wrote:
replace all your "import setuptools" with "import setuptool_lite" would be clear what the intent was -- i.e. not YET ANOTHER brand new build system...
Moving from one “one true build system” to another “one true build system” is unlikely. The problem is that the kind of things people want to do on the build side are basically infinite and trying to make a single tool to rule them all has historically boiled down to either the tool is good at one particular use case and really really terrible for every other use case OR it’s just terrible (but not really really terrible!) at every use case. A far better approach IMO is the one we’re taking. Define standard *formats* and let end users select whatever tool they want to create things that adhere to that format. This lets people create really focused tools for particular use cases, you can have a dead simple one for pure python things that never need to worry about the complexity of building C libraries (or even the complexity of needing to interface with Fortran) but the people who do need the extra complexity can use tools that enable them to do that complexity in a sane way. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Wed, Oct 21, 2015 at 12:26 PM, Donald Stufft <donald@stufft.io> wrote:
On October 21, 2015 at 3:13:11 PM, Chris Barker (chris.barker@noaa.gov) wrote:
replace all your "import setuptools" with "import setuptool_lite" would
be
clear what the intent was -- i.e. not YET ANOTHER brand new build system...
Moving from one “one true build system” to another “one true build system” is unlikely.
sure -- but I'm not even talking about another special purpose build system. A far better approach IMO is the one we’re taking. Define standard
*formats* and let end users select whatever tool they want to create things that adhere to that format.
yes -- fabulous -- but you need API as well as format, yes? i.e if pip is going to be able to find a source package and build and install it, it needs to know how to do that, ideally without knowing what the build tool is. So that's API, yes? (maybe plain old setup.py build" is a fine API... As I understand it, the current "API" that pip knows about is: "some subset of what setuptools provides" So my proposal is here is to provide a way for users to easily use jsut that subset. This would: - give folks like me a way to avoid easy-install, etc... - give all of us a way to start moving our packages to a structure that will be better suited to the upcoming orthogonal build-install toolchains. - give those folks involved in the planning a concrete way to know, for sure what people really need (or really use, anyway), both for features and API. In short -- aside from the distraction issue, this is a way to help us get to the future we want, not an alternative to that future. This lets people create really focused tools for particular use cases, you
can have a dead simple one for pure python things that never need to worry about the complexity of building C libraries
which distutils isn't so bad at, eh? :-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 21 October 2015 at 20:41, Chris Barker <chris.barker@noaa.gov> wrote:
As I understand it, the current "API" that pip knows about is:
"some subset of what setuptools provides"
So my proposal is here is to provide a way for users to easily use jsut that subset.
https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface All you need to do is write a setup.py, which can do *anything you like*, doesn't need to be based on distutils, setuptools, or whatever (but it needs to survive the setuptools injection pip currently does, but that's not a big deal). As long as it provides this API, that's fine. The only proviso is that no-one has ever tried this in anger, so it's largely untested so far. Feel free to put it through its paces and provide PRs for the docs or for pip to improve it :-) Paul
"some subset of what setuptools provides"
So my proposal is here is to provide a way for users to easily use jsut that subset.
https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface
note above: *easily* use just that subset. All you need to do is write a setup.py, which can do *anything you
like*, doesn't need to be based on distutils, setuptools, or whatever (but it needs to survive the setuptools injection pip currently does, but that's not a big deal).
not if you don't use setuptools at all-- sure. But this is the big project -- make a whole new build system. I'm suggesting that we could prototype that interface without having to create a fully functional alternative to setuptool+distutils first. Maybe I'll look into the pip-injecting-setuptools code and see if I can come up with anyway to hijack that. But If we take the "cripple setuptools" approach, rather than the "replace setuptools", than pip can still inject it... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Wed, 21 Oct 2015 21:41:35 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
On 21 October 2015 at 20:41, Chris Barker <chris.barker@noaa.gov> wrote:
As I understand it, the current "API" that pip knows about is:
"some subset of what setuptools provides"
So my proposal is here is to provide a way for users to easily use jsut that subset.
https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface
All you need to do is write a setup.py, which can do *anything you like*,
Reading this, the CLI options which have to be implemented are completely tied to setuptools' own view of the world. `--single-version-externally-managed`? `--install-headers`? Why should a random build system care about that gunk? What should it do with it? I think Nathaniel's PEP, for all its shortcomings, looked much saner than that piece of ad-hoc specification (a.k.a. "here's the random set of things we're currently doing, let's make a spec out of it"). This is like the Microsoft OOXML of Python packages distribution. Regards Antoine.
On Wed, Oct 21, 2015 at 10:25 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 21 Oct 2015 21:41:35 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
On 21 October 2015 at 20:41, Chris Barker <chris.barker@noaa.gov> wrote:
As I understand it, the current "API" that pip knows about is:
"some subset of what setuptools provides"
So my proposal is here is to provide a way for users to easily use jsut that subset.
https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface
All you need to do is write a setup.py, which can do *anything you like*,
Reading this, the CLI options which have to be implemented are completely tied to setuptools' own view of the world. `--single-version-externally-managed`? `--install-headers`? Why should a random build system care about that gunk? What should it do with it?
I agree it is not ideal, but it is not *that* hard. Regarding `--single-version-externally-managed`, if you don't care about setuptools interoperability, I believe it is safe to just ignore the flag (or more exactly do like what distutils does, that is installed directly in site-packages, and make `--single-version-externally-managed` a no-op). install_headers, I am not sure how many projects depend on that. The only projects I know that install headers (numpy, etc...) are not using the setuptools mechanism. If that's deemed useful, I could write a simple implementation that does not use distutils/setuptools but support that interface as an example (I have something similar enough times that I unfortunately know the ropes :) ).
I think Nathaniel's PEP, for all its shortcomings, looked much saner than that piece of ad-hoc specification (a.k.a. "here's the random set of things we're currently doing, let's make a spec out of it"). This is like the Microsoft OOXML of Python packages distribution.
Regards
Antoine.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Wed, Oct 21, 2015 at 5:52 PM David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 10:25 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 21 Oct 2015 21:41:35 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
On 21 October 2015 at 20:41, Chris Barker <chris.barker@noaa.gov> wrote:
As I understand it, the current "API" that pip knows about is:
"some subset of what setuptools provides"
So my proposal is here is to provide a way for users to easily use jsut that subset.
https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface
All you need to do is write a setup.py, which can do *anything you like*,
Reading this, the CLI options which have to be implemented are completely tied to setuptools' own view of the world. `--single-version-externally-managed`? `--install-headers`? Why should a random build system care about that gunk? What should it do with it?
I agree it is not ideal, but it is not *that* hard. Regarding `--single-version-externally-managed`, if you don't care about setuptools interoperability, I believe it is safe to just ignore the flag (or more exactly do like what distutils does, that is installed directly in site-packages, and make `--single-version-externally-managed` a no-op). install_headers, I am not sure how many projects depend on that. The only projects I know that install headers (numpy, etc...) are not using the setuptools mechanism.
If that's deemed useful, I could write a simple implementation that does not use distutils/setuptools but support that interface as an example (I have something similar enough times that I unfortunately know the ropes :) )
Also, it's no longer necessary to implement 'setup.py install' at all; you can just implement bdist_wheel and pip will go from there. Of course if pip can beat us to a better interface that would be great too, but a setup.py shim would be a fun little project that someone could do alone.
On October 21, 2015 at 6:15:50 PM, Daniel Holth (dholth@gmail.com) wrote:
On Wed, Oct 21, 2015 at 5:52 PM David Cournapeau wrote:
On Wed, Oct 21, 2015 at 10:25 PM, Antoine Pitrou wrote:
On Wed, 21 Oct 2015 21:41:35 +0100 Paul Moore wrote:
On 21 October 2015 at 20:41, Chris Barker wrote:
As I understand it, the current "API" that pip knows about is:
"some subset of what setuptools provides"
So my proposal is here is to provide a way for users to easily use jsut that subset.
https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface
All you need to do is write a setup.py, which can do *anything you like*,
Reading this, the CLI options which have to be implemented are completely tied to setuptools' own view of the world. `--single-version-externally-managed`? `--install-headers`? Why should a random build system care about that gunk? What should it do with it?
I agree it is not ideal, but it is not *that* hard. Regarding `--single-version-externally-managed`, if you don't care about setuptools interoperability, I believe it is safe to just ignore the flag (or more exactly do like what distutils does, that is installed directly in site-packages, and make `--single-version-externally-managed` a no-op). install_headers, I am not sure how many projects depend on that. The only projects I know that install headers (numpy, etc...) are not using the setuptools mechanism.
If that's deemed useful, I could write a simple implementation that does not use distutils/setuptools but support that interface as an example (I have something similar enough times that I unfortunately know the ropes :) )
Also, it's no longer necessary to implement 'setup.py install' at all; you can just implement bdist_wheel and pip will go from there.
Well, it is if you want it to work with older pips or pip when wheel isn’t also installed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 21 October 2015 at 22:25, Antoine Pitrou <solipsis@pitrou.net> wrote:
Reading this, the CLI options which have to be implemented are completely tied to setuptools' own view of the world. `--single-version-externally-managed`? `--install-headers`? Why should a random build system care about that gunk? What should it do with it?
Sorry, maybe I didn't explain properly. That's what you can do right now. It's certainly completely tied to setuptools' view of the world, because that's what it was written for. But just because we supply setuptools' options doesn't mean you have to use them (you can just ignore most of the junk). Changing pip to work based on a more tool-neutral API is a bigger piece of work, which has backward compatibility issues. It's doable, but then we're back to the whole process of moving forward with a better solution. As I've already said, we have a plan for that, it's just happening slowly (apparently too slowly for some people, but rather than helping with it they seem happier to propose alternative approaches that we've typically considered in the past and discarded, but hey, it's their free time and they can spend it how they like...)
I think Nathaniel's PEP, for all its shortcomings, looked much saner than that piece of ad-hoc specification (a.k.a. "here's the random set of things we're currently doing, let's make a spec out of it"). This is like the Microsoft OOXML of Python packages distribution.
Absolutely 100%. That spec is just an attempt to document what's there, because people keep saying "why do we have to use setuptools?" and the answer is "you don't, you can ignore it as long as you emulate this tiny portion of its API (and most of *that* you can ignore)". Nathaniel's PEP is much closer to what we need - people coming up with a concrete way forward that builds on what we have. And the pushback he got was where his proposal didn't take into account issues we knew needed solving (and the real-time discussion they had seems to have helped bring people's understanding a lot closer together, even though progress seems to have stalled since then. Paul
On 22 October 2015 at 10:53, Paul Moore <p.f.moore@gmail.com> wrote:
On 21 October 2015 at 22:25, Antoine Pitrou <solipsis@pitrou.net> wrote:
Reading this, the CLI options which have to be implemented are completely tied to setuptools' own view of the world. `--single-version-externally-managed`? `--install-headers`? Why should a random build system care about that gunk? What should it do with it?
Sorry, maybe I didn't explain properly. That's what you can do right now. It's certainly completely tied to setuptools' view of the world, because that's what it was written for. But just because we supply setuptools' options doesn't mean you have to use them (you can just ignore most of the junk).
Changing pip to work based on a more tool-neutral API is a bigger piece of work, which has backward compatibility issues. It's doable, but then we're back to the whole process of moving forward with a better solution. As I've already said, we have a plan for that, it's just happening slowly (apparently too slowly for some people, but rather than helping with it they seem happier to propose alternative approaches that we've typically considered in the past and discarded, but hey, it's their free time and they can spend it how they like...)
I think Nathaniel's PEP, for all its shortcomings, looked much saner than that piece of ad-hoc specification (a.k.a. "here's the random set of things we're currently doing, let's make a spec out of it"). This is like the Microsoft OOXML of Python packages distribution.
Absolutely 100%. That spec is just an attempt to document what's there, because people keep saying "why do we have to use setuptools?" and the answer is "you don't, you can ignore it as long as you emulate this tiny portion of its API (and most of *that* you can ignore)".
Nathaniel's PEP is much closer to what we need - people coming up with a concrete way forward that builds on what we have. And the pushback he got was where his proposal didn't take into account issues we knew needed solving (and the real-time discussion they had seems to have helped bring people's understanding a lot closer together, even though progress seems to have stalled since then.
Nathaniel said he was going to follow up on it with the improvements from the discussion. If he can't, someone else certainly can. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On Wed, Oct 21, 2015 at 2:53 PM, Paul Moore <p.f.moore@gmail.com> wrote:
As I've already said, we have a plan for that, it's just happening slowly (apparently too slowly for some people, but rather than helping with it they seem happier to propose alternative approaches that we've typically considered in the past and discarded,
I'm not exactly sure who (or which proposals) that was aimed at, but my idea, at least is not an alternative approach at all -- it's small idea for how to help get there, sooner than later. It seems the plan is to: finish hammering out the spec. Wait for _someone_ to make a new build system. Then wait how long? decades? for the entire user community to go to some new build system(s). Then we can finally deprecate the cruft of setuptools. Actually -- I like that plan -- and I'll probably be one of the early folks to jump on the bandwagon of the new build system. But in the meantime, a way to get my project to build without the setuptools cruft would be a nice option. BTW -- this is not just about letting pip work with more than one build system -- it's also about letting more than one package manage work with the existing (and other future) build systems. In conda, for instance, the convention is to call: python setup.py intstall. It turns out that now, we _should_ be calling: pip install ./ to actually get the right thing done. Really? A setuptools lite would do the right thing with "setup.py install" Hmm, now that I think about it, maybe I can just banish setuptools from my setup.py, stick with distutils, and then make sure to use pip to actually invoke anything -- so it will inject setuptools, but only use the bits we need.... I'll have to try that..... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 22 October 2015 at 11:33, Chris Barker <chris.barker@noaa.gov> wrote:
On Wed, Oct 21, 2015 at 2:53 PM, Paul Moore <p.f.moore@gmail.com> wrote:
As I've already said, we have a plan for that, it's just happening slowly (apparently too slowly for some people, but rather than helping with it they seem happier to propose alternative approaches that we've typically considered in the past and discarded,
I'm not exactly sure who (or which proposals) that was aimed at, but my idea, at least is not an alternative approach at all -- it's small idea for how to help get there, sooner than later.
It seems the plan is to:
finish hammering out the spec.
Wait for _someone_ to make a new build system.
Then wait how long? decades? for the entire user community to go to some new build system(s).
Then we can finally deprecate the cruft of setuptools.
Actually -- I like that plan -- and I'll probably be one of the early folks to jump on the bandwagon of the new build system.
But in the meantime, a way to get my project to build without the setuptools cruft would be a nice option.
BTW -- this is not just about letting pip work with more than one build system -- it's also about letting more than one package manage work with the existing (and other future) build systems.
In conda, for instance, the convention is to call:
python setup.py intstall.
It turns out that now, we _should_ be calling:
pip install ./
to actually get the right thing done. Really?
A setuptools lite would do the right thing with "setup.py install"
Hmm, now that I think about it, maybe I can just banish setuptools from my setup.py, stick with distutils, and then make sure to use pip to actually invoke anything -- so it will inject setuptools, but only use the bits we need....
I'll have to try that.....
You won't gain anything from that. distutils doesn't track installed files either. You *must* use 'pip install .' or 'setup.py bdist_wheel' + 'pip install .' with the current ecosystem state, -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On Wed, Oct 21, 2015 at 4:52 PM, Robert Collins <robertc@robertcollins.net> wrote:
Hmm, now that I think about it, maybe I can just banish setuptools from my setup.py, stick with distutils, and then make sure to use pip to actually invoke anything -- so it will inject setuptools, but only use the bits we need....
I'll have to try that.....
You won't gain anything from that. distutils doesn't track installed files either. You *must* use 'pip install .' or 'setup.py bdist_wheel' + 'pip install .' with the current ecosystem state,
I guess I wasn't clear -- the idea was to force myself to use pip install, rather than ever doing a plain setup.py install or setup.py develop so: pip install ./ or pip install -e ./ this way, pip will inject the parts of setuptools I really need, but hopefully not any other cruft. We'll see. -CHB
-Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 22 October 2015 at 12:59, Chris Barker <chris.barker@noaa.gov> wrote:
On Wed, Oct 21, 2015 at 4:52 PM, Robert Collins <robertc@robertcollins.net> wrote:
Hmm, now that I think about it, maybe I can just banish setuptools from my setup.py, stick with distutils, and then make sure to use pip to actually invoke anything -- so it will inject setuptools, but only use the bits we need....
I'll have to try that.....
You won't gain anything from that. distutils doesn't track installed files either. You *must* use 'pip install .' or 'setup.py bdist_wheel' + 'pip install .' with the current ecosystem state,
I guess I wasn't clear -- the idea was to force myself to use pip install, rather than ever doing a plain
setup.py install or setup.py develop
so: pip install ./ or pip install -e ./
this way, pip will inject the parts of setuptools I really need, but hopefully not any other cruft.
Its precisely the same as doing """ import setuptools from distutils import setup ... setup() """ Except that other people may still run setup.py directly, and then you'll be picking up the pieces :). -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On Wed, Oct 21, 2015 at 7:27 PM, Robert Collins <robertc@robertcollins.net> wrote:
I guess I wasn't clear -- the idea was to force myself to use pip install, rather than ever doing a plain
setup.py install or setup.py develop
so: pip install ./ or pip install -e ./
this way, pip will inject the parts of setuptools I really need, but hopefully not any other cruft.
Its precisely the same as doing
""" import setuptools from distutils import setup
... setup()
"""
Except that other people may still run setup.py directly, and then you'll be picking up the pieces :).
exactly! That is the point -- pip simply injects the full-on setuptools, but it calls it in specific ways -- i.e. --single-version-externally-managed and all that. But, you also wouldn't get things like setuptools find_packages and the like -- though I"m not sure I've ever liked that... The trick is to make it clear to your users that this particular setup.py is not expected to work right unless run from pip.... - CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Wed, Oct 21, 2015 at 11:42 AM, Chris Barker <chris.barker@noaa.gov> wrote:
But a lot was "wrong" with setuptools -- most prominently (in my mind anyway) that it put too many kind-sorta orthogonal stuff into one package: building, installing, distributing, managing version, managing dependencies, managing non-python resources, (and others??). And we didn't like how easy-install installed things :-)
So distribute, and pip, and wheel, and now a new backward compatible setuptools was born.
But we still have a bunch of should be orthogonal stuff tangled up together. In particular, I find that I often find easy-install getting invoked when I don't want ot to, and I get those darn eggs scattered all over the place, and easy_install.pth, and ????
I think if I am really careful about what I invoke when, this could be avoided, but the reality is that I've been dealing with this for years, and am trying desperately to do things the "right, modern" way, and I still get ugliness. I seriously doubt that I am alone.
So -- here's my thought:
I think we have it pretty well mapped out what functionality belongs where:
one system for building packages (setuptools?) one system for installing packages and managing dependencies (pip) one system (really standard) for metadata and distributing packages (wheel)
[I'm just whipping this out off the top of my head, I'm sure we'd need to more clearly define what belongs where]
So why not have a setuptools-lite that only does the building? We need to keep the full over-burdened setuptools around, because lot sof folks are using those features. But for those of us that are doing something fairly new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd love to have a setuptools-lite that only did what I really need, and was guaranteed NOT to accidentally introduce easy_install, etc...
This seems to me to be a way to go forward -- as it is we'll have people using setuptools features that they "shouldn't" forever, and never be able to move to a cleaner system.
Or maybe a flag:
import setuptools setuptools.use_only_modern()
That would make the dependencies easier -- i.e. pip depends on some of setuptools being there -- hard to say that it needs either setuptools OR setuptools_lite.
Maybe I'm missing something here, but if the goal is for there to be one way to do things, let's have a tool chain that only does things one way.....
I've been thinking about packaging in general and some of the complaints that I've seen and heard raised, as well as some that I have harbored personally, like having a venv per app that I've installed, or my own env where I install several things. I actually don't know if I've ran into the issues with easy_install, but I haven't been involved in installing things enough to know :) I do like the idea of having distinct pieces to the toolchain, though. Of course maybe it's also a good idea to have something that integrates all the pieces, too. -Wayne
On 21 October 2015 at 17:42, Chris Barker <chris.barker@noaa.gov> wrote:
So why not have a setuptools-lite that only does the building? We need to keep the full over-burdened setuptools around, because lot sof folks are using those features. But for those of us that are doing something fairly new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd love to have a setuptools-lite that only did what I really need, and was guaranteed NOT to accidentally introduce easy_install, etc...
In general, this sounds like a reasonable idea. But, there are the usual problems: 1. Someone needs to do the work. 2. Backward compatibility (as in, there's bound to be some projects out there using whatever functionality you can think of that "nobody would ever need"...) And a third issue specific to this idea. 3. You can't call it setuptools, and pip "injects" the real setuptools into projects when building/installing. Pretending to be setuptools was what made distribute such a PITA to deal with, and I don't imagine we'd ever want to go there again. Sorry, I'm being a downbeat grump today. The reality is that far and away the best shot we've had at getting out of this mess is the plan we're currently working to, which is to standardise interfaces, and once we have some standards, we can start decoupling the various bits and *then* replacing them as needed. We need people working on that in the first instance, work is currently stalled by lack of time from the key participants. So anyone with energy for moving this whole area forward would be really welcome to help out with the interoperability standards (metadata 2.0, defining a sdist format, pinning down pip's interfaces to build systems, things like that). But if you want to rework the existing toolset, you're probably going to be on your own (and there's unfortunately a risk that you'll pull attention away from other work via debates on this list etc). I really appreciate the enthusiasm we're seeing from people on the list at the moment, it's a matter of harnessing that into the right direction. As a thought, does anyone feel like working on an "interoperability standards roadmap" that provides a reference for where the PyPA see things going? It would be useful to keep focus, but the information's currently scattered about various mailing lists and discussions - some of the posts around the recent "new source distribution format" threads might be a good place to start. Paul
On Wed, Oct 21, 2015 at 10:17 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 21 October 2015 at 17:42, Chris Barker <chris.barker@noaa.gov> wrote:
So why not have a setuptools-lite that only does the building?
In general, this sounds like a reasonable idea. But, there are the usual problems:
1. Someone needs to do the work.
well, yeah -- that's the always the big problem.
2. Backward compatibility (as in, there's bound to be some projects out there using whatever functionality you can think of that "nobody would ever need"...)
actually, this is exactly the problem I'm trying to address with this idea. Lots of people use setuptools, and you can bet the someone, somewhere, is using every little feature (and counting on a bug) than it has. Which is why distribute merged into setuptools -- I wasn't involved in the conversation at that point, but I think it more or less started as a "new, better, setuptools replacement", and then turned into a "maintained and updated setuptools". What I do remember from those days is that setuptools itself had fallen into a maintenance hole -- very frustrating. But fixing that means that distribute lost its goal of actually making it better. The idea here is to provide an upgrade path -- if you want to do thing the new way (the sane way :-) ), then drop in setuptools_lite, and see what breaks -- then decide to remove or refactor the things that setuptools_lite doesn't support, or just go back to setuptools. 3. You can't call it setuptools, and pip "injects" the real setuptools
into projects when building/installing. Pretending to be setuptools was what made distribute such a PITA to deal with, and I don't imagine we'd ever want to go there again.
no -- it would be setuptools_lite -- different name :-) Or it would be a somewhat-disabled setuptools -- which would be the distribute problem again. But I _think_ the problem with distribute was that people WANTED it to have all the functionality of setuptools -- and going back to setuptools wasn't a good option because it was no longer well maintained (see above). Granted, I wasn't in the trenches then, or even lurking -- but I was a user, and trying to teach python newbies what to do -- and yes, it did kind of suck. As I understand it, pip injects setuptools because pip really needs a build tool that does things that distutils doesn't. And there are no other build tools. But that architecture really sucks -- pip should not have to rely on injecting a particular package, but rather be able to rely on a given API to be available. I have no idea how we can get from here to there cleanly, but it would it be that big a deal for pip to look for either setuptools or setuptools_lite, and neither are there, inject setuptools_lite. The whole point is that setuptools_lite would have everything that pip needs. Sorry, I'm being a downbeat grump today. not at all -- this has been an ugly mess for a long time -- we really, really need the folks with experience to keep things in check :-)
The reality is that far and away the best shot we've had at getting out of this mess is the plan we're currently working to, which is to standardise interfaces, and once we have some standards, we can start decoupling the various bits and *then* replacing them as needed.
well, yes. But my idea here is to have something for users to do in the interim -- right now, I HAVE to use setuptools -- it's the only thing that knows how to find the compiler on Windows, and pip needs it to install. So there is no way for me not to find it really easy to accidentally use setuptools features that I don't want to use, and "shouldn't" use. So while we are waiting for the grand future of clear APIs and well-defined components, we can start getting people ready for that.... As I write this, I'm starting to think that despite my post a half hour ago, probably the best way to do this is to add a __future__ feature to setuptools: import setuptools setuptools.__future__() or some such -- all it would do is remove and/or turn off stuff setuptools should no longer be doing. And this might actually help with the "standardise interfaces" part of the project. Essentially a working prototype of what the build tool needs to support. If turn off a feature, we may find that it's needed -- then we can decide where that feature needs to live. But if you want to
rework the existing toolset, you're probably going to be on your own (and there's unfortunately a risk that you'll pull attention away from other work via debates on this list etc).
Fair enough -- though I"m thinking of this as both a way to get something useful sooner than later, AND a way to maybe test out some of the API ideas. After all, there have been efforts to just build something new -- distutils2, etc -- maybe we need a more transitional, try it as you go approach.
As a thought, does anyone feel like working on an "interoperability standards roadmap" that provides a reference for where the PyPA see things going?
That would be great, yes! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 21 October 2015 at 20:32, Chris Barker <chris.barker@noaa.gov> wrote:
I have no idea how we can get from here to there cleanly, but it would it be that big a deal for pip to look for either setuptools or setuptools_lite, and neither are there, inject setuptools_lite.
Yes it would, unfortunately. The problem is that setup.py is not introspectable - pip simply can't "detect" what is imported (short of fragile approaches such as searching the source for "import\s*setuptools$" or something) without running it. So pip just unilaterally injects setuptools at the moment, even if it's already there. One of the points about having a setup_requires element in static metadata is precisely so that pip *can* introspect what build system is in use and not have to make assumptions/inject. Paul
On Wed, 21 Oct 2015 at 10:17 Paul Moore <p.f.moore@gmail.com> wrote:
On 21 October 2015 at 17:42, Chris Barker <chris.barker@noaa.gov> wrote:
So why not have a setuptools-lite that only does the building? We need to keep the full over-burdened setuptools around, because lot sof folks are using those features. But for those of us that are doing something fairly new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd love to have a setuptools-lite that only did what I really need, and was guaranteed NOT to accidentally introduce easy_install, etc...
In general, this sounds like a reasonable idea. But, there are the usual problems:
1. Someone needs to do the work. 2. Backward compatibility (as in, there's bound to be some projects out there using whatever functionality you can think of that "nobody would ever need"...)
And a third issue specific to this idea.
3. You can't call it setuptools, and pip "injects" the real setuptools into projects when building/installing. Pretending to be setuptools was what made distribute such a PITA to deal with, and I don't imagine we'd ever want to go there again.
Sorry, I'm being a downbeat grump today. The reality is that far and away the best shot we've had at getting out of this mess is the plan we're currently working to, which is to standardise interfaces, and once we have some standards, we can start decoupling the various bits and *then* replacing them as needed. We need people working on that in the first instance, work is currently stalled by lack of time from the key participants. So anyone with energy for moving this whole area forward would be really welcome to help out with the interoperability standards (metadata 2.0, defining a sdist format, pinning down pip's interfaces to build systems, things like that). But if you want to rework the existing toolset, you're probably going to be on your own (and there's unfortunately a risk that you'll pull attention away from other work via debates on this list etc).
I really appreciate the enthusiasm we're seeing from people on the list at the moment, it's a matter of harnessing that into the right direction. As a thought, does anyone feel like working on an "interoperability standards roadmap" that provides a reference for where the PyPA see things going? It would be useful to keep focus, but the information's currently scattered about various mailing lists and discussions - some of the posts around the recent "new source distribution format" threads might be a good place to start.
Would it makes sense to start a roadmap doc/repo under the PyPA account so the current grand vision is written down in a very high-level overview and then what the issues needing solving for each bit are along with any in-progress work are kept in a single place (I don't care if this is one big doc or there is the overview doc and then an individual doc per part of the grand vision)? Putting it up on GitHub allows for quick PRs on some Markdown doc that people can do quickly after an email thread reaches some form of a conclusion.
participants (13)
-
Antoine Pitrou -
Ben Finney -
Brett Cannon -
Chris Barker -
Daniel Holth -
David Cournapeau -
Donald Stufft -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
Robert Collins -
Ronny Pfannschmidt -
Wayne Werner