it's happened - wheels without sdists (flit)
I was wondering when someone would attempt this. Simple .ini file and some conventions, and you have a pip-installable wheel. No bdist_wheel, setup.py, MANIFEST or setuptools involved. I like it. Still alpha/beta. https://github.com/takluyver/flit
On Mar 30, 2015, at 10:19 AM, Daniel Holth <dholth@gmail.com> wrote:
I was wondering when someone would attempt this. Simple .ini file and some conventions, and you have a pip-installable wheel. No bdist_wheel, setup.py, MANIFEST or setuptools involved. I like it. Still alpha/beta.
https://github.com/takluyver/flit _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Wheels without sdists are likely a generally bad idea, downstream redistributors are not going to like them. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
The approach doesn't exclude the possibility of a source distribution, it's only a few weeks old. I would suggest that if you have to choose between having a setup.py, not having a setup.py, and not having the package on pypi at all because the packager can't figure out setup.py, prefer the second option. On Mon, Mar 30, 2015 at 10:32 AM, Donald Stufft <donald@stufft.io> wrote:
On Mar 30, 2015, at 10:19 AM, Daniel Holth <dholth@gmail.com> wrote:
I was wondering when someone would attempt this. Simple .ini file and some conventions, and you have a pip-installable wheel. No bdist_wheel, setup.py, MANIFEST or setuptools involved. I like it. Still alpha/beta.
https://github.com/takluyver/flit _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Wheels without sdists are likely a generally bad idea, downstream redistributors are not going to like them.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Mar 30, 2015 at 9:46 AM, Daniel Holth <dholth@gmail.com> wrote:
The approach doesn't exclude the possibility of a source distribution, it's only a few weeks old. I would suggest that if you have to choose between having a setup.py, not having a setup.py, and not having the package on pypi at all because the packager can't figure out setup.py, prefer the second option.
Is figuring out setup.py still a thing? Between cookiecutter laying out your project with a setup.py and the Packaging Guide, how many people still have trouble setting up a setup.py for a package? It almost seems like this is a solution wanting for a problem.
Yes, setup.py should die. Flit is one example, and you can understand it not by copy/pasting, but by spending half an hour reading its complete source code. On Mon, Mar 30, 2015 at 10:50 AM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
On Mon, Mar 30, 2015 at 9:46 AM, Daniel Holth <dholth@gmail.com> wrote:
The approach doesn't exclude the possibility of a source distribution, it's only a few weeks old. I would suggest that if you have to choose between having a setup.py, not having a setup.py, and not having the package on pypi at all because the packager can't figure out setup.py, prefer the second option.
Is figuring out setup.py still a thing? Between cookiecutter laying out your project with a setup.py and the Packaging Guide, how many people still have trouble setting up a setup.py for a package? It almost seems like this is a solution wanting for a problem.
On Mar 30, 2015, at 10:58 AM, Xavier Fernandez <xav.fernandez@gmail.com> wrote:
Wheels without sdists are likely a generally bad idea, downstream redistributors are not going to like them.
Why do you think that ? Wheels seem way simpler/saner than all the possible things setup.py can do.
Wheels are simpler than a setup.py, because wheels are a binary format and Wheels don’t need to handle things like build software because it’s already been built. However downstream redistributors will not accept a Wheel as the source for a package because it is a binary format. It doesn’t matter if you can unzip it and there is pure python there, it is still a binary format. So if you release only Wheels you’re essentially saying that downstream redistributors will never package your software (or any software that depends on it). A few issues that Wheel only has: * If your project has a C extension, downstream redistributes need access to the source code not the compiled code (as does anyone wanting to use the project on platform you didn’t release for). * If your project has tests that don’t get installed, they should get shipped as part of the sdist so that downstream can run them as part of their packaging activities to ensure they didn’t break anything in the test suite. However if you’re installing from Wheel you can’t do that. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Mar 30, 2015 at 9:59 AM, Daniel Holth <dholth@gmail.com> wrote:
Yes, setup.py should die. Flit is one example, and you can understand it not by copy/pasting, but by spending half an hour reading its complete source code.
In other words, no one should read the docs because that's a waste of time? Because a lot of time has been poured into the packaging docs and if they're not sufficient, then instead of improving them, people should write undocumented tools that force people to read the source? I'm not sure how that's better than what we already have. Further, how does flit deal with C-extensions? How does flit deal with C-extensions distributed on Linux? How does it generate the appropriate wheels for that?
On Mar 30, 2015, at 10:59 AM, Daniel Holth <dholth@gmail.com> wrote:
Yes, setup.py should die. Flit is one example, and you can understand it not by copy/pasting, but by spending half an hour reading its complete source code.
I don’t have a problem with flit (although I’m not sure it’s easier, it appears it took the setup.py keyword args and turned them into ini file directives). I do have a problem with any solution which doesn’t include sdist support. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Fair enough, I didn't think of compiled wheels :) And having a clean way to run tests for the provided wheel is indeed an other good point. On Mon, Mar 30, 2015 at 5:04 PM, Donald Stufft <donald@stufft.io> wrote:
On Mar 30, 2015, at 10:58 AM, Xavier Fernandez <xav.fernandez@gmail.com> wrote:
Wheels without sdists are likely a generally bad idea, downstream
redistributors are not going to like them.
Why do you think that ? Wheels seem way simpler/saner than all the possible things setup.py can do.
Wheels are simpler than a setup.py, because wheels are a binary format and Wheels don’t need to handle things like build software because it’s already been built. However downstream redistributors will not accept a Wheel as the source for a package because it is a binary format. It doesn’t matter if you can unzip it and there is pure python there, it is still a binary format. So if you release only Wheels you’re essentially saying that downstream redistributors will never package your software (or any software that depends on it).
A few issues that Wheel only has:
* If your project has a C extension, downstream redistributes need access to the source code not the compiled code (as does anyone wanting to use the project on platform you didn’t release for).
* If your project has tests that don’t get installed, they should get shipped as part of the sdist so that downstream can run them as part of their packaging activities to ensure they didn’t break anything in the test suite. However if you’re installing from Wheel you can’t do that.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
setup.py as implemented with distutils/setuptools has a bit of a Goldilocks problem: it's just right for a medium-complexity project but when your project is very simple it's too hard, and when you get to the point where you are trying to extend distutils by writing a 10,000 line extension, yikes. So it's fantastic to be able to just avoid distutils entirely if it isn't the right size for your project. This example, flit, does not invoke any code from distutils, setuptools or bdist_wheel to do its thing. A source release could just be an archive of the repository. On Mon, Mar 30, 2015 at 10:59 AM, Daniel Holth <dholth@gmail.com> wrote:
Yes, setup.py should die. Flit is one example, and you can understand it not by copy/pasting, but by spending half an hour reading its complete source code.
On Mon, Mar 30, 2015 at 10:50 AM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
On Mon, Mar 30, 2015 at 9:46 AM, Daniel Holth <dholth@gmail.com> wrote:
The approach doesn't exclude the possibility of a source distribution, it's only a few weeks old. I would suggest that if you have to choose between having a setup.py, not having a setup.py, and not having the package on pypi at all because the packager can't figure out setup.py, prefer the second option.
Is figuring out setup.py still a thing? Between cookiecutter laying out your project with a setup.py and the Packaging Guide, how many people still have trouble setting up a setup.py for a package? It almost seems like this is a solution wanting for a problem.
On Mon, Mar 30, 2015 at 10:18 AM, Daniel Holth <dholth@gmail.com> wrote:
setup.py as implemented with distutils/setuptools has a bit of a Goldilocks problem: it's just right for a medium-complexity project but when your project is very simple it's too hard, and when you get to the point where you are trying to extend distutils by writing a 10,000 line extension, yikes. So it's fantastic to be able to just avoid distutils entirely if it isn't the right size for your project. This example, flit, does not invoke any code from distutils, setuptools or bdist_wheel to do its thing.
A source release could just be an archive of the repository.
You still have not answered how reading flit's source code to get it working is better than using cookiecutter to generate a project, and using `python setup.py bdist_wheel sdist` (which is well-documented, and has tons of answered questions on sites like StackOverflow to help in case of a problem).
On 30 March 2015 at 16:06, Donald Stufft <donald@stufft.io> wrote:
On Mar 30, 2015, at 10:59 AM, Daniel Holth <dholth@gmail.com> wrote:
Yes, setup.py should die. Flit is one example, and you can understand it not by copy/pasting, but by spending half an hour reading its complete source code.
I don’t have a problem with flit (although I’m not sure it’s easier, it appears it took the setup.py keyword args and turned them into ini file directives). I do have a problem with any solution which doesn’t include sdist support.
Personally, I could see a benefit to something that allowed me to write my setup.py as import fancytool fancytool.setup() and got everything from a static file. But otherwise worked just like distutils/setuptools (i.e. fancytool.setup() calls setuptools.setup() behind the scenes). I'd be happy if it only handled pure-python packages, and if it didn't cover complicated things. I wouldn't be worried that I had to manually install fancytool before my setup.py worked (no setup_requires stuff here, thanks!) I don't see "don't use setuptools behind the scenes" as a necessary goal. I *do* see "make the UI simple for 90% of projects" as a worthwhile goal. I like the idea and goal behind flit. I'm not sure the implementation strategy is sufficiently compatible with existing practices, though. Paul
On Mon, Mar 30, 2015 at 6:05 PM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
In other words, no one should read the docs because that's a waste of time? Because a lot of time has been poured into the packaging docs and if they're not sufficient, then instead of improving them, people should write undocumented tools that force people to read the source? I'm not sure how that's better than what we already have.
A waste of time? No. But it sure is poor investment of time for newbie users. It's a futile struggle to document (or read about) all the features of distutils and setuptools, not because it can't be documented, but because there's too much ground to cover and PyPA's packaging.python.org approach is to just give an overview of what's available and avoid giving any real best practice recommendations as much as possible. There may be good reasons for that but that's not a sensible approach to giving users a "pitfall free" learning path. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Mon, Mar 30, 2015 at 10:23 AM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Mon, Mar 30, 2015 at 6:05 PM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
In other words, no one should read the docs because that's a waste of time? Because a lot of time has been poured into the packaging docs and if they're not sufficient, then instead of improving them, people should write undocumented tools that force people to read the source? I'm not sure how that's better than what we already have.
A waste of time? No. But it sure is poor investment of time for newbie users. It's a futile struggle to document (or read about) all the features of distutils and setuptools, not because it can't be documented, but because there's too much ground to cover and PyPA's packaging.python.org approach is to just give an overview of what's available and avoid giving any real best practice recommendations as much as possible. There may be good reasons for that but that's not a sensible approach to giving users a "pitfall free" learning path.
So for new python programmers (or newbie users in general) reading the entire source of another package to understand it is a better experience?
On Mon, Mar 30, 2015 at 11:22 AM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
On Mon, Mar 30, 2015 at 10:18 AM, Daniel Holth <dholth@gmail.com> wrote:
setup.py as implemented with distutils/setuptools has a bit of a Goldilocks problem: it's just right for a medium-complexity project but when your project is very simple it's too hard, and when you get to the point where you are trying to extend distutils by writing a 10,000 line extension, yikes. So it's fantastic to be able to just avoid distutils entirely if it isn't the right size for your project. This example, flit, does not invoke any code from distutils, setuptools or bdist_wheel to do its thing.
A source release could just be an archive of the repository.
You still have not answered how reading flit's source code to get it working is better than using cookiecutter to generate a project, and using `python setup.py bdist_wheel sdist` (which is well-documented, and has tons of answered questions on sites like StackOverflow to help in case of a problem).
It's simple. distutils, setuptools, and bdist_wheel are all terrible! They solve a 57,000 package strong backwards compatibility problem, and bdist_wheel specifically helps correct the tightly coupled build-install design flaw in distutils, but if you can avoid them great! You probably would not have to actually read the source code to flit in order to use it, but if it came down to that then it would not take very long :)
On 30/03/15 16:05, Ian Cordasco wrote:
On Mon, Mar 30, 2015 at 9:59 AM, Daniel Holth <dholth@gmail.com <mailto:dholth@gmail.com>> wrote:
Yes, setup.py should die. Flit is one example, and you can understand it not by copy/pasting, but by spending half an hour reading its complete source code.
In other words, no one should read the docs because that's a waste of time? Because a lot of time has been poured into the packaging docs and if they're not sufficient, then instead of improving them, people should write undocumented tools that force people to read the source? I'm not sure how that's better than what we already have.
You're attacking a strawman. Flit does have documentation. What Daniel was trying to say is that flit is small enough to understand by just reading the source code. Regards, Ian F
On 30 March 2015 at 16:26, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
A waste of time? No. But it sure is poor investment of time for newbie users. It's a futile struggle to document (or read about) all the features of distutils and setuptools, not because it can't be documented, but because there's too much ground to cover and PyPA's packaging.python.org approach is to just give an overview of what's available and avoid giving any real best practice recommendations as much as possible. There may be good reasons for that but that's not a sensible approach to giving users a "pitfall free" learning path.
So for new python programmers (or newbie users in general) reading the entire source of another package to understand it is a better experience?
Note that if you want a copy-and-paste level "quick start" setup.py, https://github.com/pypa/sampleproject is basically what you want. The packaging user guide uses it as its recommended starting point. If you don't like it, that probably means you already have enough knowledge to write your own version :-) Paul
On Mar 30, 2015, at 11:18 AM, Daniel Holth <dholth@gmail.com> wrote:
setup.py as implemented with distutils/setuptools has a bit of a Goldilocks problem: it's just right for a medium-complexity project but when your project is very simple it's too hard, and when you get to the point where you are trying to extend distutils by writing a 10,000 line extension, yikes. So it's fantastic to be able to just avoid distutils entirely if it isn't the right size for your project. This example, flit, does not invoke any code from distutils, setuptools or bdist_wheel to do its thing.
A source release could just be an archive of the repository.
An archive of the repository is not the same thing as a source release. Honestly, most of my setup.py’s look basically the same as a flit ini file, just inside of python instead of ini. For example, I’m not sure how something like https://github.com/pypa/packaging/blob/master/setup.py or https://github.com/pypa/warehouse/blob/master/setup.py or https://github.com/pypa/twine/blob/master/setup.py or https://github.com/pypa/readme/blob/master/setup.py would be improved by moving it to a ini file instead of a python file. The current toolchain absolutely has some problems, but I’m not convinced that shuffling around the same data into different locations is the answer to those problems. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Mar 30, 2015 at 11:34 AM, Donald Stufft <donald@stufft.io> wrote:
On Mar 30, 2015, at 11:18 AM, Daniel Holth <dholth@gmail.com> wrote:
setup.py as implemented with distutils/setuptools has a bit of a Goldilocks problem: it's just right for a medium-complexity project but when your project is very simple it's too hard, and when you get to the point where you are trying to extend distutils by writing a 10,000 line extension, yikes. So it's fantastic to be able to just avoid distutils entirely if it isn't the right size for your project. This example, flit, does not invoke any code from distutils, setuptools or bdist_wheel to do its thing.
A source release could just be an archive of the repository.
An archive of the repository is not the same thing as a source release.
Honestly, most of my setup.py’s look basically the same as a flit ini file, just inside of python instead of ini. For example, I’m not sure how something like https://github.com/pypa/packaging/blob/master/setup.py or https://github.com/pypa/warehouse/blob/master/setup.py or https://github.com/pypa/twine/blob/master/setup.py or https://github.com/pypa/readme/blob/master/setup.py would be improved by moving it to a ini file instead of a python file.
The current toolchain absolutely has some problems, but I’m not convinced that shuffling around the same data into different locations is the answer to those problems.
The way to solve the problems is to allow anyone to try by providing good hooks that do not require extending distutils.
On Mar 30, 2015, at 11:36 AM, Daniel Holth <dholth@gmail.com> wrote:
On Mon, Mar 30, 2015 at 11:34 AM, Donald Stufft <donald@stufft.io> wrote:
On Mar 30, 2015, at 11:18 AM, Daniel Holth <dholth@gmail.com> wrote:
setup.py as implemented with distutils/setuptools has a bit of a Goldilocks problem: it's just right for a medium-complexity project but when your project is very simple it's too hard, and when you get to the point where you are trying to extend distutils by writing a 10,000 line extension, yikes. So it's fantastic to be able to just avoid distutils entirely if it isn't the right size for your project. This example, flit, does not invoke any code from distutils, setuptools or bdist_wheel to do its thing.
A source release could just be an archive of the repository.
An archive of the repository is not the same thing as a source release.
Honestly, most of my setup.py’s look basically the same as a flit ini file, just inside of python instead of ini. For example, I’m not sure how something like https://github.com/pypa/packaging/blob/master/setup.py or https://github.com/pypa/warehouse/blob/master/setup.py or https://github.com/pypa/twine/blob/master/setup.py or https://github.com/pypa/readme/blob/master/setup.py would be improved by moving it to a ini file instead of a python file.
The current toolchain absolutely has some problems, but I’m not convinced that shuffling around the same data into different locations is the answer to those problems.
The way to solve the problems is to allow anyone to try by providing good hooks that do not require extending distutils.
Sure. Which is why that’s what we’ve essentially been doing. Like I said I have no problems with flit itself other than I think it needs to produce sdists to be a reasonable solution. I don’t personally think it’s much easier to work with than a simple setup.py (and in my experience, I often want a simple.py plus a little extra) but I have no problem with its existence as long as it learns how to produce sdists. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I think the point was not to say that documentation is useless (and there is some: http://flit.readthedocs.org/en/latest/ ) but that the code/implementation is much simpler than the combination of distutils/setuptools/bdist_wheel. On Mon, Mar 30, 2015 at 5:26 PM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
On Mon, Mar 30, 2015 at 10:23 AM, Ionel Cristian Mărieș < contact@ionelmc.ro> wrote:
On Mon, Mar 30, 2015 at 6:05 PM, Ian Cordasco <graffatcolmingov@gmail.com
wrote:
In other words, no one should read the docs because that's a waste of time? Because a lot of time has been poured into the packaging docs and if they're not sufficient, then instead of improving them, people should write undocumented tools that force people to read the source? I'm not sure how that's better than what we already have.
A waste of time? No. But it sure is poor investment of time for newbie users. It's a futile struggle to document (or read about) all the features of distutils and setuptools, not because it can't be documented, but because there's too much ground to cover and PyPA's packaging.python.org approach is to just give an overview of what's available and avoid giving any real best practice recommendations as much as possible. There may be good reasons for that but that's not a sensible approach to giving users a "pitfall free" learning path.
So for new python programmers (or newbie users in general) reading the entire source of another package to understand it is a better experience?
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 30 March 2015 at 16:34, Donald Stufft <donald@stufft.io> wrote:
The current toolchain absolutely has some problems, but I’m not convinced that shuffling around the same data into different locations is the answer to those problems.
Moving the data to a location that isn't Turing-complete would be a benefit. Changing the processing backend to something different once the declarative UI is in common use would be a lot easier than where we are now. Even a 90% solution would be worthwhile here - we don't *have* to cater for every use case in one go. Paul
On Mar 30, 2015, at 11:41 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 March 2015 at 16:34, Donald Stufft <donald@stufft.io> wrote:
The current toolchain absolutely has some problems, but I’m not convinced that shuffling around the same data into different locations is the answer to those problems.
Moving the data to a location that isn't Turing-complete would be a benefit. Changing the processing backend to something different once the declarative UI is in common use would be a lot easier than where we are now. Even a 90% solution would be worthwhile here - we don't *have* to cater for every use case in one go.
Paul
Well, parts of it are turing complete, since it pulls the version number out of the module itself and that’s just Python too. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Mar 30, 2015 at 6:39 PM, Xavier Fernandez <xav.fernandez@gmail.com> wrote:
I think the point was not to say that documentation is useless (and there is some: http://flit.readthedocs.org/en/latest/ ) but that the code/implementation is much simpler than the combination of distutils/setuptools/bdist_wheel.
On Mon, Mar 30, 2015 at 5:26 PM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
So for new python programmers (or newbie users in general) reading the entire source of another package to understand it is a better experience?
To put that in context, flit goes for less than 600 SLOC while distutils+setuptools+wheel amount to over 20000 SLOC. At that ratio arguments for distutils+setuptools+wheel documentation seem unreasonable.
On 30 March 2015 at 16:45, Donald Stufft <donald@stufft.io> wrote:
Well, parts of it are turing complete, since it pulls the version number out of the module itself and that’s just Python too.
Sorry, I wasn't specifically looking at flit there. But I'm in the camp that says just put the version in your ini file and in your module, and don't worry that you have it in 2 places. If managing version numbers is the biggest showstopper in moving to declarative metadata, then we've won :-) Paul
This is also a great time to be getting working setup-requires. A pip-compatible flit sdist could declare flit as a setup requirement, and its setup.py could translate install/bdist_wheel to the appropriate flit calls. On Mon, Mar 30, 2015 at 11:47 AM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Mon, Mar 30, 2015 at 6:39 PM, Xavier Fernandez <xav.fernandez@gmail.com> wrote:
I think the point was not to say that documentation is useless (and there is some: http://flit.readthedocs.org/en/latest/ ) but that the code/implementation is much simpler than the combination of distutils/setuptools/bdist_wheel.
On Mon, Mar 30, 2015 at 5:26 PM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
So for new python programmers (or newbie users in general) reading the entire source of another package to understand it is a better experience?
To put that in context, flit goes for less than 600 SLOC while distutils+setuptools+wheel amount to over 20000 SLOC. At that ratio arguments for distutils+setuptools+wheel documentation seem unreasonable.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Mon, Mar 30, 2015 at 10:47 AM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Mon, Mar 30, 2015 at 6:39 PM, Xavier Fernandez <xav.fernandez@gmail.com
wrote:
I think the point was not to say that documentation is useless (and there is some: http://flit.readthedocs.org/en/latest/ ) but that the code/implementation is much simpler than the combination of distutils/setuptools/bdist_wheel.
On Mon, Mar 30, 2015 at 5:26 PM, Ian Cordasco <graffatcolmingov@gmail.com
wrote:
So for new python programmers (or newbie users in general) reading the entire source of another package to understand it is a better experience?
To put that in context, flit goes for less than 600 SLOC while distutils+setuptools+wheel amount to over 20000 SLOC. At that ratio arguments for distutils+setuptools+wheel documentation seem unreasonable.
To be clear, no one should ever be advocating to "just read the source" as a form of documentation. This is why the Packaging guide exists (because no one should ever be expected to read the distutils, setuptools, or wheel source to use it). Code is never as self-documenting as people like to believe. And since we're talking about new users (without defining what they're new to) reading the source should only be for educational purposes. cookiecutter will serve new users better than flit or anything else. cookiecutter will teach new users good package structure and take care of the (possibly hard parts) of a setup.py. Then, when the "new user" goes to publish it, there's tons of prior documentation on how to do it. If they run into problems using flit they have the skimpy documentation or the source. Yeah, it's "easy" to read 600 SLOC for you, but what about for some "new user"? Are they new to python? Why do they have to care about reading the source if something else will "just work" as documented for their "simple" use case?
On Mar 30, 2015, at 11:52 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 March 2015 at 16:45, Donald Stufft <donald@stufft.io> wrote:
Well, parts of it are turing complete, since it pulls the version number out of the module itself and that’s just Python too.
Sorry, I wasn't specifically looking at flit there. But I'm in the camp that says just put the version in your ini file and in your module, and don't worry that you have it in 2 places. If managing version numbers is the biggest showstopper in moving to declarative metadata, then we've won :-)
Paul
Honestly, I don’t think that setup.py as a development interface is that bad. It gets really bad when we start sticking it inside of a sdist and using that as part of the installation metadata. It’s not unusual for me to want (or need) to do something a little bit different in a project, or something that the original authors didn’t quite intend to do. This is perfectly valid and fine inside of a file that only ever gets executed on a developer machine. However it *needs* to be “compiled” down to a static file when creating a sdist. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Mar 30, 2015 at 11:55 AM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
On Mon, Mar 30, 2015 at 10:47 AM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Mon, Mar 30, 2015 at 6:39 PM, Xavier Fernandez <xav.fernandez@gmail.com> wrote:
I think the point was not to say that documentation is useless (and there is some: http://flit.readthedocs.org/en/latest/ ) but that the code/implementation is much simpler than the combination of distutils/setuptools/bdist_wheel.
On Mon, Mar 30, 2015 at 5:26 PM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
So for new python programmers (or newbie users in general) reading the entire source of another package to understand it is a better experience?
To put that in context, flit goes for less than 600 SLOC while distutils+setuptools+wheel amount to over 20000 SLOC. At that ratio arguments for distutils+setuptools+wheel documentation seem unreasonable.
To be clear, no one should ever be advocating to "just read the source" as a form of documentation. This is why the Packaging guide exists (because no one should ever be expected to read the distutils, setuptools, or wheel source to use it).
Code is never as self-documenting as people like to believe. And since we're talking about new users (without defining what they're new to) reading the source should only be for educational purposes. cookiecutter will serve new users better than flit or anything else. cookiecutter will teach new users good package structure and take care of the (possibly hard parts) of a setup.py. Then, when the "new user" goes to publish it, there's tons of prior documentation on how to do it. If they run into problems using flit they have the skimpy documentation or the source.
Yeah, it's "easy" to read 600 SLOC for you, but what about for some "new user"? Are they new to python? Why do they have to care about reading the source if something else will "just work" as documented for their "simple" use case?
No one has advocated reading the source code instead of reading the documentation.
On Mon, Mar 30, 2015 at 6:55 PM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
Then, when the "new user" goes to publish it, there's tons of prior documentation on how to do it. If they run into problems using flit they have the skimpy documentation or the source.
Now it might have skimpy docs and no users, but that's largely a product of time. I think `flit` should be judged on what it can be in the future, not all what it's right now. To put it in picture, the argument you're making is like comparing the amazon rainforest to a banana milkshake recipe. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Mon, Mar 30, 2015 at 11:14 AM, Daniel Holth <dholth@gmail.com> wrote:
On Mon, Mar 30, 2015 at 11:55 AM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
On Mon, Mar 30, 2015 at 10:47 AM, Ionel Cristian Mărieș <
wrote:
On Mon, Mar 30, 2015 at 6:39 PM, Xavier Fernandez <xav.fernandez@gmail.com> wrote:
I think the point was not to say that documentation is useless (and
is some: http://flit.readthedocs.org/en/latest/ ) but that the code/implementation is much simpler than the combination of distutils/setuptools/bdist_wheel.
On Mon, Mar 30, 2015 at 5:26 PM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
So for new python programmers (or newbie users in general) reading the entire source of another package to understand it is a better
experience?
To put that in context, flit goes for less than 600 SLOC while distutils+setuptools+wheel amount to over 20000 SLOC. At that ratio arguments for distutils+setuptools+wheel documentation seem unreasonable.
To be clear, no one should ever be advocating to "just read the source" as a form of documentation. This is why the Packaging guide exists (because no one should ever be expected to read the distutils, setuptools, or wheel source to use it).
Code is never as self-documenting as people like to believe. And since we're talking about new users (without defining what they're new to) reading
contact@ionelmc.ro> there the
source should only be for educational purposes. cookiecutter will serve new users better than flit or anything else. cookiecutter will teach new users good package structure and take care of the (possibly hard parts) of a setup.py. Then, when the "new user" goes to publish it, there's tons of prior documentation on how to do it. If they run into problems using flit they have the skimpy documentation or the source.
Yeah, it's "easy" to read 600 SLOC for you, but what about for some "new user"? Are they new to python? Why do they have to care about reading the source if something else will "just work" as documented for their "simple" use case?
No one has advocated reading the source code instead of reading the documentation.
Thankfully this is a publicly archived list. Quoting yourself:
Flit is one example, and you can understand it not by copy/pasting, but by spending half an hour reading its complete source code.
In which you advocate reading the source of a tool over using setup.py which has countless resources written about it on the internet.
On Mon, Mar 30, 2015 at 11:15 AM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Mon, Mar 30, 2015 at 6:55 PM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:
Then, when the "new user" goes to publish it, there's tons of prior documentation on how to do it. If they run into problems using flit they have the skimpy documentation or the source.
Now it might have skimpy docs and no users, but that's largely a product of time. I think `flit` should be judged on what it can be in the future, not all what it's right now. To put it in picture, the argument you're making is like comparing the amazon rainforest to a banana milkshake recipe.
Well to make a better comparison, we're discussing a healthy diet to a sugar laced treat. The healthy diet is sustainable because it is well documented and has tons of users with experience with it, but has pitfalls in that it can be expensive at times, while the sugar laced treat is good for a quick blood sugar spike but will leave you thoroughly unsatisfied and eventually wanting for the healthy diet. New users, (like some people who prefer high sugar diets) may prefer the initial simplicity, but at the cost of having to do a lot more work up front. Those who follow a healthy diet (and ostensibly exercise regiment) will have tooling to not have to worry about how to construct a healthy diet. New users who find and work with those already on a healthy diet will learn the tools that help them avoid the pitfalls of starting on a healthy diet (e.g., tools that generate setup.py for you and maintain the 98% use case, which is inevitably where new users fall*) and will be better off for the long term. * Note, new users still has yet to be defined by anyone advocating that this is better for new users because of mystical reasons (like being able to read the source code).
On Mar 30, 2015, at 11:33 AM, Ian Foote <ian@feete.org> wrote:
On 30/03/15 16:05, Ian Cordasco wrote:
On Mon, Mar 30, 2015 at 9:59 AM, Daniel Holth <dholth@gmail.com <mailto:dholth@gmail.com>> wrote: Yes, setup.py should die. Flit is one example, and you can understand it not by copy/pasting, but by spending half an hour reading its complete source code.
In other words, no one should read the docs because that's a waste of time? Because a lot of time has been poured into the packaging docs and if they're not sufficient, then instead of improving them, people should write undocumented tools that force people to read the source? I'm not sure how that's better than what we already have.
You're attacking a strawman. Flit does have documentation. What Daniel was trying to say is that flit is small enough to understand by just reading the source code.
Meh, comparisons by SLOC are silly in either direction. The size of the code base isn’t an interesting discriminator in judging quality. Python does not try to be a language that minimizes the number of lines of code and when looking at straight SLOC (in abstract of these two projects) it’s entirely possible to have something harder to understand with less SLOC and easier to understand with more. In particular looking at these two pieces of software, trying to compare SLOC is even more silly given the two toolchains don’t even come close to the same feature set. Ian C, is pointing out that being able to understand it “just by reading the source code” (whatever that means, what software can’t you understand by reading the source code?) isn’t an extremely useful discriminator either because it’s premised on the fact that there is going to be a common situation where you *need* to read the source code to understand it. So the statement is in itself not particularly useful because it’s premised on something that is fundamentally user hostile. An end user shouldn’t care if it’s written in 600 lines, 6000 lines, or 6 million lines. What matters to them is the interface it provides and the documentation around it and often the “ease” of reading the code is used to excuse poor (or non existent) documentation. IOW, using SLOC or size or anything of the like as a discriminator is akin to saying “well it can fit on a floppy” as a measure of quality. None of this is particularly centered around flit (or distutils) so I’m not speaking about either project there. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mar 30, 2015, at 11:56 AM, Donald Stufft wrote:
Honestly, I don’t think that setup.py as a development interface is that bad.
Especially when you cargo cult most of a new project's basic infrastructure[*] from one that's already working. Sweat out the first one, then reuse. ;) Cheers, -Barry [*] For me, not just setup.py but also tox.ini, coverage.ini, MANIFEST.in, other helpers, readmes, license files, source templates, etc.
On 31 Mar 2015 01:17, "Xavier Fernandez" <xav.fernandez@gmail.com> wrote:
Fair enough, I didn't think of compiled wheels :) And having a clean way to run tests for the provided wheel is indeed an
other good point. To elaborate on Donald's answer, one of our general requirements downstream in distro land is the ability to rebuild the entire distro from source without human intervention (for example: https://beaker-project.org/docs-develop/user-guide/beaker-provided-tasks.htm... for Fedora & derivatives) We need to build from source not just to ensure our binaries match the published source code, but also because our build systems are designed to let us *patch* the packages before we build them. This is what lets us backport security updates, bug fixes, and sometimes even entire features without needing to rebase a package on a new upstream release of a project. We also like to be able to automatically *generate* package build instructions from upstream metadata. Those tools aren't perfect today (too much info is missing from current generation upstream metadata), but they're a lot better than nothing. Historically those tools have relied on "setup.py install" working on an unpacked sdist. These days in Fedora we aim to rely on "pip install" working instead of invoking setup.py directly, but there are over 15000 Python packages in Fedora (and a similar number in Debian), and Fedora's policy only switched to preferring indirection through pip for Python package builds last year (AFAIK Debian still favours invoking setup.py directly, hopefully Barry will correct me if I'm wrong). But as long as those source package builds keep working downstream, we don't really care what's happening behind the scenes in the upstream build systems. It's mildly irritating when packaging a new release of an already included project turns into a yak shaving expedition to package a whole new build system as a dependency, but that's just one of the consequences of running an open source integration project rather than writing everything from scratch yourself. It's also one of the reasons why d2to1 is such a neat hack - that presents as plain distutils at sdist build time, while allowing the use of distutils2 features at package definition time upstream. Cheers, Nick.
On Mon, Mar 30, 2015 at 5:04 PM, Donald Stufft <donald@stufft.io> wrote:
On Mar 30, 2015, at 10:58 AM, Xavier Fernandez <xav.fernandez@gmail.com>
Wheels without sdists are likely a generally bad idea, downstream redistributors are not going to like them.
Why do you think that ? Wheels seem way simpler/saner than all the
Wheels are simpler than a setup.py, because wheels are a binary format
and Wheels don’t need to
handle things like build software because it’s already been built. However downstream redistributors will not accept a Wheel as the source for a package because it is a binary format. It doesn’t matter if you can unzip it and there is pure python there, it is still a binary
you’re essentially saying that downstream redistributors will never
wrote: possible things setup.py can do. format. So if you release only Wheels package your software (or any
software that depends on it).
A few issues that Wheel only has:
* If your project has a C extension, downstream redistributes need access to the source code not the compiled code (as does anyone wanting to use the project on platform you didn’t release for).
* If your project has tests that don’t get installed, they should get shipped as part of the sdist so that downstream can run them as part of their packaging activities to ensure they didn’t break anything in the test suite. However if you’re installing from Wheel you can’t do that.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 3/30/15 11:56 AM, Donald Stufft wrote:
On Mar 30, 2015, at 11:52 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 March 2015 at 16:45, Donald Stufft <donald@stufft.io> wrote:
Well, parts of it are turing complete, since it pulls the version number out of the module itself and that’s just Python too.
Sorry, I wasn't specifically looking at flit there. But I'm in the camp that says just put the version in your ini file and in your module, and don't worry that you have it in 2 places. If managing version numbers is the biggest showstopper in moving to declarative metadata, then we've won :-)
Paul
Honestly, I don’t think that setup.py as a development interface is that bad. It gets really bad when we start sticking it inside of a sdist and using that as part of the installation metadata.
It’s not unusual for me to want (or need) to do something a little bit different in a project, or something that the original authors didn’t quite intend to do. This is perfectly valid and fine inside of a file that only ever gets executed on a developer machine. However it *needs* to be “compiled” down to a static file when creating a sdist.
Right, that is my understanding: setup.py is fine except when it is executed on installation. But I think there is a slight cognitive advantage to setup.ini vs. setup.py. You can never execute an ini file, even in development. So the same file can (somehow) be used in development and production without "compiling down" first. In other words: maybe switching to ini is the right thing to do long term. However, the practicality of doing so may be so small (due to disutils/setuptools baggage and/or inability to overcome setup.py momentum) that "compiling down" (setup.py) becomes a more attractive first step, at least.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Alex Clark · http://about.me/alex.clark
On Mar 30, 2015, at 8:33 PM, Alex Clark <aclark@aclark.net> wrote:
On 3/30/15 11:56 AM, Donald Stufft wrote:
On Mar 30, 2015, at 11:52 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 March 2015 at 16:45, Donald Stufft <donald@stufft.io> wrote:
Well, parts of it are turing complete, since it pulls the version number out of the module itself and that’s just Python too.
Sorry, I wasn't specifically looking at flit there. But I'm in the camp that says just put the version in your ini file and in your module, and don't worry that you have it in 2 places. If managing version numbers is the biggest showstopper in moving to declarative metadata, then we've won :-)
Paul
Honestly, I don’t think that setup.py as a development interface is that bad. It gets really bad when we start sticking it inside of a sdist and using that as part of the installation metadata.
It’s not unusual for me to want (or need) to do something a little bit different in a project, or something that the original authors didn’t quite intend to do. This is perfectly valid and fine inside of a file that only ever gets executed on a developer machine. However it *needs* to be “compiled” down to a static file when creating a sdist.
Right, that is my understanding: setup.py is fine except when it is executed on installation.
But I think there is a slight cognitive advantage to setup.ini vs. setup.py. You can never execute an ini file, even in development. So the same file can (somehow) be used in development and production without "compiling down" first. In other words: maybe switching to ini is the right thing to do long term.
However, the practicality of doing so may be so small (due to disutils/setuptools baggage and/or inability to overcome setup.py momentum) that "compiling down" (setup.py) becomes a more attractive first step, at least.
I think trying to use the same file is an attractive nuisance. Separating them lets you do a lot more for developer convenience without hurting the ability of the installation side of things. For instance, it’s somewhat common to want to import the module in order to pull the version out of it. That’s not something we’d want to do outside of the developer side of things, but it’s something that’s completely reasonable to do in the developer toolchain (either implicitly as in flit, or explicitly as in setup.py). IOW, splitting these into the “developer side” tooling and the “format that automated installers consume” lets you define both of them to be the best at what they are doing instead of trying to put something together that satisfies two competing use cases. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mar 31, 2015, at 09:14 AM, Nick Coghlan wrote:
We need to build from source not just to ensure our binaries match the published source code, but also because our build systems are designed to let us *patch* the packages before we build them. This is what lets us backport security updates, bug fixes, and sometimes even entire features without needing to rebase a package on a new upstream release of a project.
For Debian, all this, plus it's required by the Debian Social Contract and Debian Free Software Guidelines.
(AFAIK Debian still favours invoking setup.py directly, hopefully Barry will correct me if I'm wrong).
Correct. For any setup.py/distutils/setuptools-based project, we have one preferred, recommended, and non-deprecated "build system", and though it's fairly flexible, it mostly relies on a working setup.py. As Debian Jessie is frozen right now, I don't expect that to change for now. We have sort of a love-hate relationship with pip, but it would be interesting to sit down at Pycon and discuss how we might like to see downstream distro support improve or change as it relates to common upstream packaging standards. Cheers, -Barry
On 30 March 2015 at 16:56, Donald Stufft <donald@stufft.io> wrote:
Honestly, I don’t think that setup.py as a development interface is that bad. It gets really bad when we start sticking it inside of a sdist and using that as part of the installation metadata.
It’s not unusual for me to want (or need) to do something a little bit different in a project, or something that the original authors didn’t quite intend to do. This is perfectly valid and fine inside of a file that only ever gets executed on a developer machine. However it *needs* to be “compiled” down to a static file when creating a sdist.
Hmm, I don't think I'd ever really understood the distinction between "development setup" and "sdist" that clearly. I take your point, it's the sdist level that we want to avoid executable metadata formats in. Paul
On 31 March 2015 at 08:04, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 March 2015 at 16:56, Donald Stufft <donald@stufft.io> wrote:
Honestly, I don’t think that setup.py as a development interface is that bad. It gets really bad when we start sticking it inside of a sdist and using that as part of the installation metadata.
It’s not unusual for me to want (or need) to do something a little bit different in a project, or something that the original authors didn’t quite intend to do. This is perfectly valid and fine inside of a file that only ever gets executed on a developer machine. However it *needs* to be “compiled” down to a static file when creating a sdist.
Hmm, I don't think I'd ever really understood the distinction between "development setup" and "sdist" that clearly. I take your point, it's the sdist level that we want to avoid executable metadata formats in.
Thinking some more about that, my confusion is probably in part because pip doesn't distinguish between a "development directory" and a sdist at the moment. For both, it runs "setup.py bdist_wheel/install". So I guess work on a new sdist format would have to include pip learning to distinguish between a sdist and a working directory, and installing (or building wheels from) the two things differently. Paul
On 31 March 2015 at 17:10, Paul Moore <p.f.moore@gmail.com> wrote:
On 31 March 2015 at 08:04, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 March 2015 at 16:56, Donald Stufft <donald@stufft.io> wrote:
Honestly, I don’t think that setup.py as a development interface is that bad. It gets really bad when we start sticking it inside of a sdist and using that as part of the installation metadata.
It’s not unusual for me to want (or need) to do something a little bit different in a project, or something that the original authors didn’t quite intend to do. This is perfectly valid and fine inside of a file that only ever gets executed on a developer machine. However it *needs* to be “compiled” down to a static file when creating a sdist.
Hmm, I don't think I'd ever really understood the distinction between "development setup" and "sdist" that clearly. I take your point, it's the sdist level that we want to avoid executable metadata formats in.
Thinking some more about that, my confusion is probably in part because pip doesn't distinguish between a "development directory" and a sdist at the moment. For both, it runs "setup.py bdist_wheel/install". So I guess work on a new sdist format would have to include pip learning to distinguish between a sdist and a working directory, and installing (or building wheels from) the two things differently.
Yep, the current PEP 426 draft suggests that sdists should grow a "dist-info" directory (akin to wheel files and installed packages), while development directories would continue to lack any of the generated metadata. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Most sdists already have static metadata in the form of a PKG-INFO file or an .egg-info directory. If it has dependencies it's almost guaranteed to have .egg-info because distutils does not support dependencies. The only problem is that the static metadata is not trustworthy because the dependencies often change based on the target environment as detected in setup.py. To work reliably pip has to execute setup.py before it can download a package's install_requires. If we want to improve the dependency resolver we would need to put install_requires in a static file and promise that it was actually static. On Tue, Mar 31, 2015 at 8:06 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 31 March 2015 at 17:10, Paul Moore <p.f.moore@gmail.com> wrote:
On 31 March 2015 at 08:04, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 March 2015 at 16:56, Donald Stufft <donald@stufft.io> wrote:
Honestly, I don’t think that setup.py as a development interface is that bad. It gets really bad when we start sticking it inside of a sdist and using that as part of the installation metadata.
It’s not unusual for me to want (or need) to do something a little bit different in a project, or something that the original authors didn’t quite intend to do. This is perfectly valid and fine inside of a file that only ever gets executed on a developer machine. However it *needs* to be “compiled” down to a static file when creating a sdist.
Hmm, I don't think I'd ever really understood the distinction between "development setup" and "sdist" that clearly. I take your point, it's the sdist level that we want to avoid executable metadata formats in.
Thinking some more about that, my confusion is probably in part because pip doesn't distinguish between a "development directory" and a sdist at the moment. For both, it runs "setup.py bdist_wheel/install". So I guess work on a new sdist format would have to include pip learning to distinguish between a sdist and a working directory, and installing (or building wheels from) the two things differently.
Yep, the current PEP 426 draft suggests that sdists should grow a "dist-info" directory (akin to wheel files and installed packages), while development directories would continue to lack any of the generated metadata.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 31 March 2015 at 13:06, Nick Coghlan <ncoghlan@gmail.com> wrote:
Thinking some more about that, my confusion is probably in part because pip doesn't distinguish between a "development directory" and a sdist at the moment. For both, it runs "setup.py bdist_wheel/install". So I guess work on a new sdist format would have to include pip learning to distinguish between a sdist and a working directory, and installing (or building wheels from) the two things differently.
Yep, the current PEP 426 draft suggests that sdists should grow a "dist-info" directory (akin to wheel files and installed packages), while development directories would continue to lack any of the generated metadata.
The thing is, because sdists include setup.py and the whole build layout, there's no point (at the moment) in pip looking at any sdist-specific data. We have to be able to do "pip install /my/dev/directory", and so reusing the same code for a sdist is the obvious thing to do. If the sdist metadata meant that we could somehow do things *better* for a sdist than for a dev directory, then maybe it would be worth doing that. But at the moment, it basically gains us nothing to use the metadata from a sdist. It's not quite that simple, I know. But until we work out how to do something useful with a sdist that we can't do with a dev checkout, it's hard to justify treating sdists specially. Paul
On 1 Apr 2015 00:53, "Paul Moore" <p.f.moore@gmail.com> wrote:
It's not quite that simple, I know. But until we work out how to do something useful with a sdist that we can't do with a dev checkout, it's hard to justify treating sdists specially.
I see it as more a matter of eventually migrating to a "devdir -> sdist -> wheel -> installed" build & deployment pipeline, where the tools used at each stage are only required to support the transition to the next stage rather than having to support the whole pipeline as setuptools does. (setup.py support would necessarily remain as a backwards compatibility requirement) The 3 transitions (devdir -> sdist, sdist -> wheel, wheel -> installed) may then not only be executed separately, but can also reduce the combinatorial complexity of what needs to be tested from a tooling perspective. (Whether there's a standard "devdir -> local dev build" command can be considered independently of the design of the distribution & deployment pipeline itself) Cheers, Nick.
Paul
Nick Coghlan <ncoghlan@gmail.com> writes:
I see it as more a matter of eventually migrating to a "devdir -> sdist -> wheel -> installed" build & deployment pipeline
Yes. Increasingly often these days, that first distinction (dev working tree is not the same as source for distribution) gets ignored, and actively obliterated. Getting many GitHub or Bitbucket projects to distribute a source release, that doesn't have a mess of build artifacts in it, is an exercise in great frustration – largely because those platforms provide no convenient means to distribute tarballs that are *not* an undifferentiated bundling of everything in the repository.
where the tools used at each stage are only required to support the transition to the next stage
For the “dev tree → sdist” transition, part of the battle is going to be educating developers that such a distinction actually exists. The common bad practice of including build artifacts in VCS obliterates the separate “package the source for distribution” step, which means that step is rarely reproducible or reliable. -- \ “Simplicity and elegance are unpopular because they require | `\ hard work and discipline to achieve and education to be | _o__) appreciated.” —Edsger W. Dijkstra | Ben Finney
On 1 April 2015 at 00:03, Nick Coghlan <ncoghlan@gmail.com> wrote:
It's not quite that simple, I know. But until we work out how to do something useful with a sdist that we can't do with a dev checkout, it's hard to justify treating sdists specially.
I see it as more a matter of eventually migrating to a "devdir -> sdist -> wheel -> installed" build & deployment pipeline, where the tools used at each stage are only required to support the transition to the next stage rather than having to support the whole pipeline as setuptools does. (setup.py support would necessarily remain as a backwards compatibility requirement)
I wasn't particularly clear - my apologies. By "we" I meant "pip" in this context. Users expect to be able to do "pip install /dev/directory" and have it "just work". Internally, pip can dor the dev->sdist->wheel->install dance, certainly, and that's the ultimate goal, I agree. What I was trying to say that there's no obvious benefit to pup from splitting out the dev->sdist and sdist->wheel step. We can just as easily run setup.py bdist_wheel in a dev directory directly. I know we can *also* just run "setup.py install" from a dev directory (we do now), but there are benefits in not doing so - splitting out the step of building a wheel lets us cache the wheels and speed up the process, because installing a wheel is quicker than building from source. So what I'm saying is that we need similar motivating benefits to justify splitting out the "build a sdist" step, and it's not yet clear to me what those would be. Paul
participants (11)
-
Alex Clark -
Barry Warsaw -
Ben Finney -
Daniel Holth -
Donald Stufft -
Ian Cordasco -
Ian Foote -
Ionel Cristian Mărieș -
Nick Coghlan -
Paul Moore -
Xavier Fernandez