Re: [Distutils] Remove distutils, was: red, green, refactor ...
Am 21.10.2015 um 13:28 schrieb Ionel Cristian Mărieș:
On Wed, Oct 21, 2015 at 1:42 PM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
Why not deprecate and remove distutils?
Should we or can we? There was an attempt at that, called distutils2. I'd love to hear what people have learnt from that.
And if now-days walking around with pants instead of only underwear is the way to go, should we still have underwear under the pants? I think yes :-) In other words, setuptools uses distutils.
ok, at the moment setuptools uses distutils. Why not melt them together into **one** underwear-pants-module? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/
On 21 October 2015 at 12:52, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
(1) because there's no-one with anything like the time or energy to do such a thing (2) because doing so would almost certainly break vast numbers of existing setup.py scripts Paul
On Wed, 21 Oct 2015 13:07:10 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
On 21 October 2015 at 12:52, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
(1) because there's no-one with anything like the time or energy to do such a thing (2) because doing so would almost certainly break vast numbers of existing setup.py scripts
(3) because distutils is part of the standard library and setuptools is most likely unwelcome there :-) Regards Antoine.
On Wed, Oct 21, 2015 at 3:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
(3) because distutils is part of the standard library and setuptools is most likely unwelcome there :-)
Well, it's not dead yet. Sorry, I mean, good enough to be included in the stdlib :-) But seriously, is it correct to assume that it's not there for largely the same reasons pip ain't in stdlib? (and yes, getpip is in stdlib) Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Wed, 21 Oct 2015 15:30:51 +0300 Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Wed, Oct 21, 2015 at 3:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
(3) because distutils is part of the standard library and setuptools is most likely unwelcome there :-)
Well, it's not dead yet. Sorry, I mean, good enough to be included in the stdlib :-)
But seriously, is it correct to assume that it's not there for largely the same reasons pip ain't in stdlib? (and yes, getpip is in stdlib)
setuptools inclusion was discussed long ago AFAIK, and it was refused. For pip the reasons are different (the much shorter release cycle, the external dependencies required). See e.g.: https://mail.python.org/pipermail/python-dev/2006-April/063966.html https://mail.python.org/pipermail/python-dev/2006-April/063886.html (incidentally, the latter link shows a previous attempt to refactor distutils was shot down because of backwards compatibility; which is also why distutils2 was canned, despite bringing concrete positive enhancements; had those attempts not been shot down, the situation right now would be significantly better than it is...) Regards Antoine.
On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
Am 21.10.2015 um 13:28 schrieb Ionel Cristian Mărieș:
On Wed, Oct 21, 2015 at 1:42 PM, Thomas Güttler < guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
Why not deprecate and remove distutils?
Should we or can we? There was an attempt at that, called distutils2. I'd love to hear what people have learnt from that.
And if now-days walking around with pants instead of only underwear is the way to go, should we still have underwear under the pants? I think yes :-) In other words, setuptools uses distutils.
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
What do you hope getting from that ? distutils is in the stdlib, so cannot change easily, and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python. On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend. David
On 21 October 2015 at 14:55, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
What do you hope getting from that ? distutils is in the stdlib, so cannot change easily, and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python.
It's more useful to go the other direction and vendor a modern version of distutils inside setuptools: https://bitbucket.org/pypa/setuptools/issues/417/adopt-distutils distutils can then optionally be replaced wholesale at runtime, rather than having the internals be monkeypatched.
On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend.
We still need a migration path to modern metadata standards for everyone using distutils and setuptools - that's the side of things that caused major problems for both distribute and distutils2. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 21 Oct 2015 17:05:29 +0200 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 21 October 2015 at 14:55, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
What do you hope getting from that ? distutils is in the stdlib, so cannot change easily, and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python.
It's more useful to go the other direction and vendor a modern version of distutils inside setuptools:
It seems it would only add a bit more craziness to the current landscape. What happens to projects which have a need to monkeypatch distutils? How does it interact with the vendored version? etc. Regards Antoine.
Am 21.10.2015 um 17:15 schrieb Antoine Pitrou:
On Wed, 21 Oct 2015 17:05:29 +0200 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 21 October 2015 at 14:55, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
What do you hope getting from that ? distutils is in the stdlib, so cannot change easily, and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python.
It's more useful to go the other direction and vendor a modern version of distutils inside setuptools:
It seems it would only add a bit more craziness to the current landscape. What happens to projects which have a need to monkeypatch distutils? How does it interact with the vendored version? etc.
What are the needs to monkeypatch distutils? I guess there are good reasons to do so. So why not implement this directly there, so that no monkey patching is needed any more? What are "vendored version" in this context? Regards, Thomas Güttler -- http://www.thomas-guettler.de/
On Thu, Oct 22, 2015 at 5:04 PM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
On Wed, 21 Oct 2015 17:05:29 +0200 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 21 October 2015 at 14:55, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
What do you hope getting from that ? distutils is in the stdlib, so
cannot
change easily, and even if putting setuptools in the stdlib were
Am 21.10.2015 um 17:15 schrieb Antoine Pitrou: possible,
you would now need to handle different versions of setuptools for different versions of python.
It's more useful to go the other direction and vendor a modern version of distutils inside setuptools:
It seems it would only add a bit more craziness to the current landscape. What happens to projects which have a need to monkeypatch distutils? How does it interact with the vendored version? etc.
What are the needs to monkeypatch distutils?
That's the only way to extend distutils in a meaningful way, because of the broken command concept (e.g. the command dependency graph is defined inside the classes instead of being defined by an external scheduler). As anybody who has dealt with that codebase will tell you, It is impossible to fix distutils without breaking most non trivial uses of it. David
The strategy that we are successfully pursuing is to standardize around file formats that the build tools generate and the installers and runtime can consume. For example we can convert a setuptools egg or a bdist_wininst .exe to a wheel, and in the future we will be able to convert e-mail style PKG-INFO to json. It's much easier to test whether a tool is producing correct metadata than to decide whether the API is compatible.
Am 21.10.2015 um 17:29 schrieb Daniel Holth:
The strategy that we are successfully pursuing is to standardize around file formats that the build tools generate and the installers and runtime can consume....
I ask myself: Why a standard? I see that a standard is very important if there will be several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...) But here a single implementation for creating and installing packages would be enough. Is a standard really needed? Regards, Thomas Güttler -- http://www.thomas-guettler.de/
On Wed, Oct 21, 2015 at 9:34 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
I ask myself: Why a standard? I see that a standard is very important if there will be several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...)
But here a single implementation for creating and installing packages would be enough.
Is a standard really needed?
Yes -- because of exactly what you say above -- we really can't have a SINGLE build system that will well support everything -- the common use-caes, sure (distutils already does that), but when it comes to bulding complex packages like numpy, sciPy, etc, it's really inadequate. If we standardize the format, interfaces and metadata, and properly separate the components, then someone can use some other build system for their package, and still be able to use the same distribution, dependency mangement, and installation tools. or, for that matter swap out one of those other components too. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Am 21.10.2015 um 18:46 schrieb Chris Barker:
On Wed, Oct 21, 2015 at 9:34 AM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
I ask myself: Why a standard? I see that a standard is very important if there will be several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...)
But here a single implementation for creating and installing packages would be enough.
Is a standard really needed?
Yes -- because of exactly what you say above -- we really can't have a SINGLE build system that will well support everything -- the common use-caes, sure (distutils already does that), but when it comes to bulding complex packages like numpy, sciPy, etc, it's really inadequate.
What happens if the common use cases are inadequate? My guess: re-inventing the same stuff over and over again. Once in numpy, once in scipy ... Why should it be impossible to get all the needs of numpy and scipy into setuptools? I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ... Even for complex packages. This data gets processed by setuptools. I don't see a need for more than one library doing this. Plugins are nice and can solve edge cases. Regards, Thomas Güttler -- http://www.thomas-guettler.de/
On Wed, Oct 21, 2015 at 10:10 PM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
On Wed, Oct 21, 2015 at 9:34 AM, Thomas Güttler < guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
I ask myself: Why a standard? I see that a standard is very important if there will be several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...)
But here a single implementation for creating and installing
Am 21.10.2015 um 18:46 schrieb Chris Barker: packages would be enough.
Is a standard really needed?
Yes -- because of exactly what you say above -- we really can't have a
SINGLE build system that will well support everything --
the common use-caes, sure (distutils already does that), but when it comes to bulding complex packages like numpy, sciPy, etc, it's really inadequate.
What happens if the common use cases are inadequate?
My guess: re-inventing the same stuff over and over again. Once in numpy, once in scipy ...
Why should it be impossible to get all the needs of numpy and scipy into setuptools?
I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ...
Even for complex packages.
This data gets processed by setuptools. I don't see a need for more than one library doing this. Plugins are nice and can solve edge cases.
setuptools is one bloated piece of code that contains too much features. I'd prefer more lean and mean package.
Am 22.10.2015 um 09:28 schrieb anatoly techtonik:
On Wed, Oct 21, 2015 at 10:10 PM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
Am 21.10.2015 um 18:46 schrieb Chris Barker: > On Wed, Oct 21, 2015 at 9:34 AM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de> <mailto:guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>>> wrote: > > I ask myself: Why a standard? I see that a standard is very important if there will be > several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...) > > But here a single implementation for creating and installing packages would be enough. > > Is a standard really needed? > > > Yes -- because of exactly what you say above -- we really can't have a SINGLE build system that will well support everything -- > the common use-caes, sure (distutils already does that), but when it comes to bulding complex packages like numpy, sciPy, etc, it's really inadequate.
What happens if the common use cases are inadequate?
My guess: re-inventing the same stuff over and over again. Once in numpy, once in scipy ...
Why should it be impossible to get all the needs of numpy and scipy into setuptools?
I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ...
Even for complex packages.
This data gets processed by setuptools. I don't see a need for more than one library doing this. Plugins are nice and can solve edge cases.
setuptools is one bloated piece of code that contains too much features. I'd prefer more lean and mean package.
Yes, your are rigth. I'd like a lean and mean package, too. Regards, Thomas Güttler -- http://www.thomas-guettler.de/
On Thu, Oct 22, 2015 at 8:47 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
I have a dream: For packaging and building package provides only
**data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ...
Even for complex packages.
It's a nice dream -- but I think there is more or less a consensus on this list that you really can't do everything completely declaratively. I know I would find it very frustrating to have to hard-code everything up front. But we do have a bad situtation now where you can't know the metadata until you've run setup.py -- not good. Maybe we need to add a configure step: a configure_package.py that would return the full set of metadata -- or maybe a dict that could be passed right on to: setup(**configuration) In the simple cases, this would simply return a static dict, but it would be a hook where people could do whatever they need. Perhaps sdists would have had the configure step already run -- not sure about that one off the bat, but it would be kind of like distributing a source tarball with autotools already run. I think this may be a bit where the sdist conversation was going (at least on one side-track...): you might have sdists that are customized to the platform: "this sdist is for numpy running on Windows with OpenBLAS" that sort of thing. I THINK, once you'd broken it down to that level, you could have static configuration. -Chris
This data gets processed by setuptools. I don't see a need for more
than one library doing this. Plugins are nice and can solve edge cases.
setuptools is one bloated piece of code that contains too much features.
I'd prefer more lean and mean package.
Yes, your are rigth. I'd like a lean and mean package, too.
Regards, Thomas Güttler
-- http://www.thomas-guettler.de/ _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Thu, Oct 22, 2015 at 11:27 AM, Chris Barker <chris.barker@noaa.gov> wrote:
On Thu, Oct 22, 2015 at 8:47 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
I have a dream: For packaging and building package provides only
**data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ...
Even for complex packages.
It's a nice dream -- but I think there is more or less a consensus on this list that you really can't do everything completely declaratively. I know I would find it very frustrating to have to hard-code everything up front.
I've seen this sentiment mentioned several times... Having only had experience with sdist-style packages which are dead simple to define, do you have any examples of some specific thing that's a pain to do declaratively? -W
On Oct 26, 2015 6:59 AM, "Wayne Werner" <waynejwerner@gmail.com> wrote:
On Thu, Oct 22, 2015 at 11:27 AM, Chris Barker <chris.barker@noaa.gov>
On Thu, Oct 22, 2015 at 8:47 AM, Thomas Güttler <
guettliml@thomas-guettler.de> wrote:
I have a dream: For packaging and building package provides only
**data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ...
Even for complex packages.
It's a nice dream -- but I think there is more or less a consensus on
wrote: this list that you really can't do everything completely declaratively. I know I would find it very frustrating to have to hard-code everything up front.
I've seen this sentiment mentioned several times... Having only had
experience with sdist-style packages which are dead simple to define, do you have any examples of some specific thing that's a pain to do declaratively? Test the system's implementation of the math.h 'tanh' function to decide whether it correctly implements C99 annex F style handling of edge cases and thus can be used safely, or whether we need to fall back to our internal version of tanh instead, and define some appropriate C preprocessor macros depending on the result. -n
Test if the provided system C/C++ compiler supports OpenMP and do not use the openmp compiler flag to fallback to a sequential build if not. -- Olivier Grisel
Am 26.10.2015 um 20:07 schrieb Nathaniel Smith:
On Oct 26, 2015 6:59 AM, "Wayne Werner" <waynejwerner@gmail.com <mailto:waynejwerner@gmail.com>> wrote:
On Thu, Oct 22, 2015 at 11:27 AM, Chris Barker <chris.barker@noaa.gov <mailto:chris.barker@noaa.gov>> wrote:
On Thu, Oct 22, 2015 at 8:47 AM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ...
Even for complex packages.
It's a nice dream -- but I think there is more or less a consensus on this list that you really can't do everything completely declaratively. I know I would find it very frustrating to have to hard-code everything up front.
I've seen this sentiment mentioned several times... Having only had experience with sdist-style packages which are dead simple to define, do you have any examples of some specific thing that's a pain to do declaratively?
Test the system's implementation of the math.h 'tanh' function to decide whether it correctly implements C99 annex F style handling of edge cases and thus can be used safely, or whether we need to fall back to our internal version of tanh instead, and define some appropriate C preprocessor macros depending on the result.
With a plugin-system it would be easy to handle this declaratively. We need a way to extend the supported key-value pairs. This would be done in check_tanh plugin package. Then you need to build-depend on the new check_tanh package. I guess that most developers feel that this is very complicated. I feel the same, but I **think** it is not. The structure is straight forward. It just feels complicated. The check_tanh package can be create in just a few minutes. Regards, Thomas Güttler -- http://www.thomas-guettler.de/
On Tue, Oct 27, 2015 at 12:13 AM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
Am 26.10.2015 um 20:07 schrieb Nathaniel Smith:
On Oct 26, 2015 6:59 AM, "Wayne Werner" <waynejwerner@gmail.com <mailto:waynejwerner@gmail.com>> wrote:
On Thu, Oct 22, 2015 at 11:27 AM, Chris Barker <chris.barker@noaa.gov <mailto:chris.barker@noaa.gov>> wrote:
On Thu, Oct 22, 2015 at 8:47 AM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ...
Even for complex packages.
It's a nice dream -- but I think there is more or less a consensus on this list that you really can't do everything completely declaratively. I know I would find it very frustrating to have to hard-code everything up front.
I've seen this sentiment mentioned several times... Having only had experience with sdist-style packages which are dead simple to define, do you have any examples of some specific thing that's a pain to do declaratively?
Test the system's implementation of the math.h 'tanh' function to decide whether it correctly implements C99 annex F style handling of edge cases and thus can be used safely, or whether we need to fall back to our internal version of tanh instead, and define some appropriate C preprocessor macros depending on the result.
With a plugin-system it would be easy to handle this declaratively.
We need a way to extend the supported key-value pairs. This would be done in check_tanh plugin package.
Then you need to build-depend on the new check_tanh package.
I guess that most developers feel that this is very complicated. I feel the same, but I **think** it is not. The structure is straight forward. It just feels complicated. The check_tanh package can be create in just a few minutes.
Plugins are arbitrary code, that contain plenty of ifs, defs, and method calls... In any case, the next thread over is discussing exactly the question of how we can make it easy for you to write your own build system and see how declarative you can make it, with plugins or without :-). -n -- Nathaniel J. Smith -- http://vorpus.org
Am 21.10.2015 um 18:46 schrieb Chris Barker:
On Wed, Oct 21, 2015 at 9:34 AM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
I ask myself: Why a standard? I see that a standard is very important if there will be several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...)
But here a single implementation for creating and installing packages would be enough.
Is a standard really needed?
Yes -- because of exactly what you say above -- we really can't have a SINGLE build system that will well support everything -- the common use-caes, sure (distutils already does that), but when it comes to bulding complex packages like numpy, sciPy, etc, it's really inadequate.
Yes, you are right: "we really can't have a SINGLE build system that will well support everything". But maybe a single build systems which has a well documented plugin API can solve all needs? Regards, Thomas Güttler -- http://www.thomas-guettler.de/
another example is that without a standard there is no viable way to replace a grown messy legacy tool with a much less messy tool over time creating a new trimmed down standard and supporting it from the legacy tool as well as from the new tool works out much better than trying to trim down the legacy tool over time On 10/21/2015 06:34 PM, Thomas Güttler wrote:
Am 21.10.2015 um 17:29 schrieb Daniel Holth:
The strategy that we are successfully pursuing is to standardize around file formats that the build tools generate and the installers and runtime can consume.... I ask myself: Why a standard? I see that a standard is very important if there will be several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...)
But here a single implementation for creating and installing packages would be enough.
Is a standard really needed?
Regards, Thomas Güttler
Am 21.10.2015 um 19:04 schrieb Ronny Pfannschmidt:
another example is that without a standard there is no viable way to replace a grown messy legacy tool with a much less messy tool over time creating a new trimmed down standard and supporting it from the legacy tool as well as from the new tool works out much better than trying to trim down the legacy tool over time
I prefer the code of unittests to the not testable sentences in a standard. But maybe I am missing something. Regards, Thomas Güttler -- http://www.thomas-guettler.de/
Am 21.10.2015 um 17:29 schrieb Daniel Holth:
The strategy that we are successfully pursuing is to standardize around file formats that the build tools generate and the installers and runtime can consume. For example we can convert a setuptools egg or a bdist_wininst .exe to a wheel, and in the future we will be able to convert e-mail style PKG-INFO to json. It's much easier to test whether a tool is producing correct metadata than to decide whether the API is compatible.
I don't see any reason to convert a setuptools egg or a bdist_wininst .exe to a wheel. Why should this be done? The egg is something created from sources. Why not create the wheel from the sources? It is like SRPM (source RPMs). I never understood why they exist. Why not create the target from the sources? Regards, Thomas -- http://www.thomas-guettler.de/
From: Nick Coghlan <ncoghlan@gmail.com>
We still need a migration path to modern metadata standards for everyone using distutils and setuptools - that's the side of things that caused major problems for both distribute and distutils2.
I've thought about the migration path with distil, which uses a declarative metadata format (superset of PEP-426 as it was a while ago) where the actual metadata is extracted from setup.py files in releases already on PyPI. This works well for simple distributions which don't do anything fancy in setup.py, such as extending build commands to create files on the fly which are then installed etc. In my early testing, distil (which uses no distutils or setuptools code) could install and package distributions which were pure-Python as well as package, build and install ones with C extensions and even Fortran extensions. I haven't done much work on it for a while due to other commitments, but when I had the time to work on it, I noted precious little interest from distutils-sig (yes, I know we're all volunteers, but one person can only do so much). IMO the distil approach comes the closest to a least-work migration path: with distutils2 / packaging that was pulled from 3.3, almost everyone would have to convert their releases to setup.cfg, with limited support for automating this and no support for existing stuff on PyPI, which made it a non-starter. Plus, setup.cfg never supported build-side metadata IIRC, only installation-side. With distil, the metadata for build and installation is *automatically* extracted from setup.py with no work required from packagers e.g. for existing releases, and the extraction only fails if the setup.py code does certain operations which can't be inferred from the actual call to setup() in setup.py (I know that those hard-to-infer operations occur in a lot of releases - but that's the nature of setup.py, where anything goes - I still wanted to see how much progress could be made for the simple cases). The metadata extracted has been useful in other contexts, e.g. Brett Cannon's caniusepython3 uses it to work out package dependencies. With distil, the entire dependency tree is calculated before any package is downloaded at all - something people seem to want - but, it seems, not to the extent of actually doing anything concrete, even to try out code and report problems ... distil isn't perfect and I've said it's more like a proof of concept, but it's quite usable for the "non-fancy" distributions. As long as people are free to put whatever code they want in setup.py, we're not really going to have a simple migration path, because a sizeable number of packagers will carry on doing so, and that'll be hard to convert to declarative metadata automagically (not to mention the situation with existing releases). And as long as there is so much inertia stopping interested parties from trying out new approaches, it seems like the pace of development of better tools will continue to be glacial. Regards, Vinay Sajip
On Thu, Oct 22, 2015 at 1:50 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I've thought about the migration path with distil, which uses a declarative metadata format (superset of PEP-426 as it was a while ago) where the actual metadata is extracted from setup.py files in releases already on PyPI. This works well for simple distributions which don't do anything fancy in setup.py, such as extending build commands to create files on the fly which are then installed etc. In my early testing, distil (which uses no distutils or setuptools code) could install and package distributions which were pure-Python as well as package, build and install ones with C extensions and even Fortran extensions. I haven't done much work on it for a while due to other commitments, but when I had the time to work on it, I noted precious little interest from distutils-sig (yes, I know we're all volunteers, but one person can only do so much).
What does distil do to achieve this? A setuptools polyfill? Just parsing setup.py and inspecting file layout? Is there any code for this or you're just talking about an idea? I couldn't find any distil project in https://bitbucket.org/vinay.sajip/ ... Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On October 21, 2015 at 7:05:59 PM, Ionel Cristian Mărieș (contact@ionelmc.ro) wrote:
On Thu, Oct 22, 2015 at 1:50 AM, Vinay Sajip wrote:
I've thought about the migration path with distil, which uses a declarative metadata format (superset of PEP-426 as it was a while ago) where the actual metadata is extracted from setup.py files in releases already on PyPI. This works well for simple distributions which don't do anything fancy in setup.py, such as extending build commands to create files on the fly which are then installed etc. In my early testing, distil (which uses no distutils or setuptools code) could install and package distributions which were pure-Python as well as package, build and install ones with C extensions and even Fortran extensions. I haven't done much work on it for a while due to other commitments, but when I had the time to work on it, I noted precious little interest from distutils-sig (yes, I know we're all volunteers, but one person can only do so much).
What does distil do to achieve this? A setuptools polyfill? Just parsing setup.py and inspecting file layout?
Is there any code for this or you're just talking about an idea? I couldn't find any distil project in https://bitbucket.org/vinay.sajip/ …
If I remember correctly distil was not OSS at all, it was distributed as a runnable zip file and if you unzipped it there wasn’t a license at all (or perhaps it was all rights reserved?). I forget the exact details but I recall unzipping it to take a look, seeing it wasn’t OSS and closing it after that. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Am 22.10.2015 um 01:10 schrieb Donald Stufft:
On October 21, 2015 at 7:05:59 PM, Ionel Cristian Mărieș (contact@ionelmc.ro) wrote:
On Thu, Oct 22, 2015 at 1:50 AM, Vinay Sajip wrote:
I've thought about the migration path with distil, which uses a declarative metadata format (superset of PEP-426 as it was a while ago) where the actual metadata is extracted from setup.py files in releases already on PyPI. This works well for simple distributions which don't do anything fancy in setup.py, such as extending build commands to create files on the fly which are then installed etc. In my early testing, distil (which uses no distutils or setuptools code) could install and package distributions which were pure-Python as well as package, build and install ones with C extensions and even Fortran extensions. I haven't done much work on it for a while due to other commitments, but when I had the time to work on it, I noted precious little interest from distutils-sig (yes, I know we're all volunteers, but one person can only do so much).
What does distil do to achieve this? A setuptools polyfill? Just parsing setup.py and inspecting file layout?
Is there any code for this or you're just talking about an idea? I couldn't find any distil project in https://bitbucket.org/vinay.sajip/ …
If I remember correctly distil was not OSS at all, it was distributed as a runnable zip file and if you unzipped it there wasn’t a license at all (or perhaps it was all rights reserved?). I forget the exact details but I recall unzipping it to take a look, seeing it wasn’t OSS and closing it after that.
Hi Donald, If distil is good software, it's sad that there is no licence. Maybe there is an mail adress to contact the orignal author? Most authors feel proud if they get positive feedback. Regards, Thomas Güttler -- http://www.thomas-guettler.de/
Am 21.10.2015 um 17:05 schrieb Nick Coghlan:
On 21 October 2015 at 14:55, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
What do you hope getting from that ? distutils is in the stdlib, so cannot change easily, and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python.
It's more useful to go the other direction and vendor a modern version of distutils inside setuptools: https://bitbucket.org/pypa/setuptools/issues/417/adopt-distutils
distutils can then optionally be replaced wholesale at runtime, rather than having the internals be monkeypatched.
On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend.
We still need a migration path to modern metadata standards for everyone using distutils and setuptools - that's the side of things that caused major problems for both distribute and distutils2.
I guess you have a rough migration path in your mind? I guess some people here are interested. Regards, Thomas Güttler -- http://www.thomas-guettler.de/
On 22 October 2015 at 18:07, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
Am 21.10.2015 um 17:05 schrieb Nick Coghlan:
On 21 October 2015 at 14:55, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
What do you hope getting from that ? distutils is in the stdlib, so cannot change easily, and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python.
It's more useful to go the other direction and vendor a modern version of distutils inside setuptools: https://bitbucket.org/pypa/setuptools/issues/417/adopt-distutils
distutils can then optionally be replaced wholesale at runtime, rather than having the internals be monkeypatched.
On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend.
We still need a migration path to modern metadata standards for everyone using distutils and setuptools - that's the side of things that caused major problems for both distribute and distutils2.
I guess you have a rough migration path in your mind? I guess some people here are interested.
The people here are the ones already building it: ensuring setuptools can generate any required metadata, using pip to inject setuptools into the build process for projects using plain distutils. Providing the option for folks that don't want to use distutils or setuptools to be able to use something else and have pip handle that reliably would be a nice addition, especially if it's done in a beginner friendly way that can become the new default build tool recommendation, but ideas for improvement that start with expecting tens of thousands of project owners to change the build system they use aren't going to work. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Am 23.10.2015 um 21:12 schrieb Nick Coghlan:
On 22 October 2015 at 18:07, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
Am 21.10.2015 um 17:05 schrieb Nick Coghlan:
On 21 October 2015 at 14:55, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
What do you hope getting from that ? distutils is in the stdlib, so cannot change easily, and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python.
It's more useful to go the other direction and vendor a modern version of distutils inside setuptools: https://bitbucket.org/pypa/setuptools/issues/417/adopt-distutils
distutils can then optionally be replaced wholesale at runtime, rather than having the internals be monkeypatched.
On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend.
We still need a migration path to modern metadata standards for everyone using distutils and setuptools - that's the side of things that caused major problems for both distribute and distutils2.
I guess you have a rough migration path in your mind? I guess some people here are interested.
The people here are the ones already building it: ensuring setuptools can generate any required metadata, using pip to inject setuptools into the build process for projects using plain distutils.
Providing the option for folks that don't want to use distutils or setuptools to be able to use something else and have pip handle that reliably would be a nice addition, especially if it's done in a beginner friendly way that can become the new default build tool recommendation, but ideas for improvement that start with expecting tens of thousands of project owners to change the build system they use aren't going to work.
yes, you are right. Several small steps are better than one big. -- http://www.thomas-guettler.de/
Am 21.10.2015 um 14:55 schrieb David Cournapeau:
On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
Am 21.10.2015 um 13:28 schrieb Ionel Cristian Mărieș:
On Wed, Oct 21, 2015 at 1:42 PM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de> <mailto:guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>>> wrote:
Why not deprecate and remove distutils?
Should we or can we? There was an attempt at that, called distutils2. I'd love to hear what people have learnt from that.
And if now-days walking around with pants instead of only underwear is the way to go, should we still have underwear under the pants? I think yes :-) In other words, setuptools uses distutils.
ok, at the moment setuptools uses distutils.
Why not melt them together into **one** underwear-pants-module?
What do you hope getting from that ? distutils is in the stdlib, so cannot change easily , and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python.
I hope to be getting this: Packaging Python will be KISS. If setuptools in inside Python, then different versions of Python each have their matching version included. If a package has the need for newer version of setuptool, it provides the hint somewhere: required_setuptools_version>=N.M Then the matching version from pypi gets downloaded first. Looks straight forward for me.
On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend.
If the underlying design is fundamentally difficult to extend, what should be done? Build tools on top of it? I guess this is not a good idea. Or start from scratch and deprecate the current underlying design in the future? What do you think? Regards, Thomas -- http://www.thomas-guettler.de/
On Thu, 22 Oct 2015 18:15:37 +0200 Thomas Güttler <guettliml@thomas-guettler.de> wrote:
On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend.
If the underlying design is fundamentally difficult to extend, what should be done?
Build tools on top of it? I guess this is not a good idea.
Or start from scratch and deprecate the current underlying design in the future?
Providing official hooks would be a way to provide extension facilities. Of course this assumes someone would actually maintain distutils, and "the community" would stop making a ruckus everytime an extremely minor behaviour change is considered (otherwise you get what happened to distutils2). Regards Antoine.
On Thu, Oct 22, 2015 at 9:15 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
If setuptools in inside Python, then different versions of Python each have their matching version included.
If a package has the need for newer version of setuptool, it provides the hint somewhere: required_setuptools_version>=N.M
Then the matching version from pypi gets downloaded first.
Isn't this what ensurepip already does??? Sure, it's called ensure *pip* but pip requires setuptools, so you get that out of the box. If in the future we have some nifty replacement for both setuptools and distutils, pip can just grab that, too... I think we're at the "someone needs to actually write the code" stage.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Am 22.10.2015 um 18:31 schrieb Chris Barker:
On Thu, Oct 22, 2015 at 9:15 AM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
If setuptools in inside Python, then different versions of Python each have their matching version included.
If a package has the need for newer version of setuptool, it provides the hint somewhere: required_setuptools_version>=N.M
Then the matching version from pypi gets downloaded first.
Isn't this what ensurepip already does??? Sure, it's called ensure *pip* but pip requires setuptools, so you get that out of the box.
ok, nice. I was thinking about a new key-value pair in setup.py or setup.cfg. ensurepip is a package. Is there already a way to pull the matching pip or setuptools version via a setup.py/setup.cfg? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/
On 22 October 2015 at 18:15, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
Am 21.10.2015 um 14:55 schrieb David Cournapeau:
What do you hope getting from that ? distutils is in the stdlib, so cannot change easily , and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python.
I hope to be getting this: Packaging Python will be KISS.
If setuptools in inside Python, then different versions of Python each have their matching version included.
No, this is what got us into the problem with distutils: we want the packaging system to be *version independent*, as many projects want to support multiple versions of Python. Having the build system be runtime version dependent turns out to be a huge pain. This is why Microsoft versions Visual Studio separately from Windows, Apple versions Xcode separately from Mac OS X, and Red Hat versions the RHEL Developer Toolset separately from RHEL itself. However, we also have more than a decade's worth of existing projects on PyPI that we can't force to switch to a new build system, so we need to ensure any improvements we make benefit *existing* projects, even if they never get updated again. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (15)
-
anatoly techtonik -
Antoine Pitrou -
Chris Barker -
Daniel Holth -
David Cournapeau -
Donald Stufft -
Ionel Cristian Mărieș -
Nathaniel Smith -
Nick Coghlan -
Olivier Grisel -
Paul Moore -
Ronny Pfannschmidt -
Thomas Güttler -
Vinay Sajip -
Wayne Werner