Tarek, Speaking of packaging patches. Would there be any chance that you could get the 'bdist_deb' patches integrated in Distutils prior to PyCon? Regards, Gerry
2009/3/11 Gerry Reno <greno@verizon.net>:
Tarek, Speaking of packaging patches. Would there be any chance that you could get the 'bdist_deb' patches integrated in Distutils prior to PyCon?
The work to be done for bdist_deb is quite important and requires some Debian expertise I don't have (yet). My plan was to be helped by some Debian people amongst the people that have interacted on that issue and in this list, and work on the patches. But that won't be possible before Pycon, because I am commited on other issues before this one. But if someone is willing to lead this work, and if a fully tested patch is provided before Pycon, I am willing to review it and commit it. Although I doubt this will be possible in such a short time. Regards Tarek
Regards, Gerry
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé - Directeur Technique INGENIWEB (TM) - SAS 50000 Euros - RC B 438 725 632 Bureaux de la Colline - 1 rue Royale - Bâtiment D - 9ème étage 92210 Saint Cloud - France Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04 http://www.ingeniweb.com - une société du groupe Alter Way
Tarek: Will the bdist_deb code be based on or copied from the existing stdeb tool? Should people who are interested in bdist_deb be using stdeb right now in order to find bugs in it and learn what they like/ dislike about it? Regards, Zooko
Zooko, While we're waiting for the 'official' answer from Tarek... I don't think the code is being taken from stdeb. There were 'bdist_deb' patches that were developed/tested/contributed a while ago and just never got into distutils. To the best of my knowledge it is that patch set. Of course let's see what Tarek says but I'm pretty sure that's the set. Regards, Gerry
Gerry Reno wrote:
Tarek, Speaking of packaging patches. Would there be any chance that you could get the 'bdist_deb' patches integrated in Distutils prior to PyCon?
Hi Tarek, Has there been any progress on getting the 'bdist_deb' target (akin to 'bdist_rpm') implemented in Distutils? We are looking for behavior that would be similar to bdist_rpm driven by setup.cfg with fully automated generation of packages. There is in existence a set of patches that supposedly does this. Has any progress been made to integrate those patches? Regards, Gerry
Gerry Reno wrote:
Tarek, Speaking of packaging patches. Would there be any chance that you could get the 'bdist_deb' patches integrated in Distutils prior to PyCon?
Hi Tarek, Has there been any progress on getting the 'bdist_deb' target (akin to 'bdist_rpm') implemented in Distutils? We are looking for behavior that would be similar to bdist_rpm driven by setup.cfg with fully automated generation of packages. There is in existence a set of patches that supposedly does this. Has any progress been made to integrate those patches? Regards, Gerry
Hi Gerry, During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) : - it's better for them to have their own release cycles - it's hard for me, as distutils maintainer, to maintain and make evolve os-specific tools. People that are specialists on those OS will do a better job. Now the problems to reach that goal are: - some people are reluctant not to have everything included in the stdlib (I am not). I don't think we can all agree on this, and since Guido have encouraged this strategy during the summit, I guess I'm more inclined to follow it nevertheless. - we need to detect for each existing command (rpm, etc) a project that can take over, prior to remove it from Distutils when appliable . Now, for debian, since it's not included in distutils, the question is : what is the most used/advanced project ? sdteb ? ++ Tarek
On Mon, 8 Jun 2009 14:28:52 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) :
- it's better for them to have their own release cycles - it's hard for me, as distutils maintainer, to maintain and make evolve os-specific tools. People that are specialists on those OS will do a better job.
Let's think about this.... and what it means..... It seems that this direction comes down to "we don't want to maintain it" rather than anything else. The whole point of distutils is to be able to make up a package in such a way that it can be distributed to every platform.... right? So removing the code that is specific to specific platforms is just collapsing the whole project. Rather than "removing" platform specific code... and rendering distutils crippled... why not work on developing the code so that it operates more like a "cross-compiler"... That is... runs on any platform... builds for any platform.... David
On Jun 8, 2009, at 8:50 PM, David Lyon wrote:
On Mon, 8 Jun 2009 14:28:52 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) :
- it's better for them to have their own release cycles - it's hard for me, as distutils maintainer, to maintain and make evolve os-specific tools. People that are specialists on those OS will do a better job.
Let's think about this.... and what it means.....
It seems that this direction comes down to "we don't want to maintain it" rather than anything else.
The whole point of distutils is to be able to make up a package in such a way that it can be distributed to every platform.... right?
So removing the code that is specific to specific platforms is just collapsing the whole project.
Rather than "removing" platform specific code... and rendering distutils crippled... why not work on developing the code so that it operates more like a "cross-compiler"...
That is...
runs on any platform...
builds for any platform....
Because this is almost impossible... for example building a win_inst package on linux (let alone other unixes, etc) is very very hard (specially if it involves extensions), the same for rpm, deb and mpkg. Distutils should provide all tools to make this happen, but the package creation itself should be let to each system. This is the same problem with svn and setuptools where you end up with non working code for long periods of time. Now that python and probably a ton of other projects are going to use mercurial this can even become worse. -- Leonardo Santagada santagada at gmail.com
On Mon, 8 Jun 2009 23:37:42 -0300, Leonardo Santagada <santagada@gmail.com> wrote:
builds for any platform....
Because this is almost impossible...
Not true... Don't forget, cygwin runs under windows, and wine runs under linux So it's not so difficult to build windows targets under linux and linux targets under windows. Cross compiling has been common practice for the last 30 years.
for example building a win_inst package on linux (let alone other unixes, etc) is very very hard (specially if it involves extensions).
It should work under wine, or should be made to work under wine. This is an example of something that can be adapted to work under linux with not too many problems.
the same for rpm, deb and mpkg.
Granted. They're are a bit more tricky. But essentially they are all just zip files with some entry and exit scripts.
Distutils should provide all tools to make this happen, but the package creation itself should be let to each system.
At a minimum, distutils should be able to create an .egg that will work on every system. I still believe in cross-compilers... even python itself is a sort of cross compiler.... killing the cross compilation side of distutils seems way worse than enhancing it.... David
David Lyon wrote:
Not true...
Don't forget, cygwin runs under windows
And how does that help to build a linux binary ?
, and wine runs under linux
Yes, but Visual Studio does not. So you are out of luck if you need full compatibility with pre-built python binaries on python.org. It works almost ok with mingw, but not always - and it is far from working well on windows 64 bits BTW. Anyway, as long as distutils works under windows, distutils will work under wine - the problem is the toolchain, not distutils.
Cross compiling has been common practice for the last 30 years.
And still hard to get right. That should certainly not be built in distutils - a sane build tool needs very little knowledge about cross compilation, really. Python itself cannot be cross-compiled out of the box anyway.
At a minimum, distutils should be able to create an .egg that will work on every system.
Before implementing things that no other system has ever done, we should focus on basic things that distutils cannot do today and are expected from a build/distribution tool. Those are hard problems to get right, saying that it is not hard because it is just moving files and scripts does not really help. That's like saying programming is easy because it is just inputting characters in a text file. cheers, David
On Mon, 8 Jun 2009 23:37:42 -0300, Leonardo Santagada <santagada@gmail.com> wrote:
That is...
runs on any platform...
builds for any platform....
Because this is almost impossible... for example building a win_inst package on linux (let alone other unixes, etc) is very very hard (specially if it involves extensions), the same for rpm, deb and mpkg.
quoting... http://docs.python.org/distutils/builtdist.html#cross-compiling-on-windows "A built distribution is how you make life as easy as possible for installers of your module distribution: for users of RPM-based Linux systems, it’s a binary RPM; for Windows users, it’s an executable installer; for Debian-based Linux users, it’s a Debian package; and so forth. Obviously, no one person will be able to create built distributions for every platform under the sun, so the Distutils are designed to enable module developers to concentrate on their specialty—writing code and creating source distributions"
On Tue, 09 Jun 2009 12:20:17 +0900, David Cournapeau
Don't forget, cygwin runs under windows
And how does that help to build a linux binary ?
I wasn't thinking of it for building binaries.. rather using various packaging tools that could be adapted or compiled from source to run. Not all distributions need to be binary executables.
, and wine runs under linux
Yes, but Visual Studio does not. So you are out of luck if you need full compatibility with pre-built python binaries on python.org. It works almost ok with mingw, but not always - and it is far from working well on windows 64 bits BTW.
Yes. Point taken. But not everybody is building C extensions for python. A lot of python libraries are built with absolutely no C code within them at all.
Anyway, as long as distutils works under windows, distutils will work under wine - the problem is the toolchain, not distutils.
Yes. That's correct.
Cross compiling has been common practice for the last 30 years.
And still hard to get right. That should certainly not be built in distutils -
Well distutils already has partial support for it. And the alternative being disccused here is removing the platform specific code. quoting http://docs.python.org/distutils/builtdist.html#cross-compiling-on-windows "A built distribution is how you make life as easy as possible for installers of your module distribution: for users of RPM-based Linux systems, it’s a binary RPM; for Windows users, it’s an executable installer; for Debian-based Linux users, it’s a Debian package; and so forth. Obviously, no one person will be able to create built distributions for every platform under the sun, so the Distutils are designed to enable module developers to concentrate on their specialty—writing code and creating source distributions"
At a minimum, distutils should be able to create an .egg that will work on every system.
Before implementing things that no other system has ever done, we should focus on basic things that distutils cannot do today and are expected from a build/distribution tool.
I'm just reading the distutils documentation..... The documentation is setting an expectation... Yes - you can be expected to work towards the goals that are documented within your project. Change the documentation and don't promise it if it is not going to be delivered.....
Those are hard problems to get right, saying that it is not hard because it is just moving files and scripts does not really help. That's like saying programming is easy because it is just inputting characters in a text file.
yeah - but there are clever people working on this stuff.... They should be encouraged to work towards the solution as documented. David
On Tue, 09 Jun 2009 12:20:17 +0900, David Cournapeau
Before implementing things that no other system has ever done, we should focus on basic things that distutils cannot do today and are expected from a build/distribution tool.
"If you have a pure module distribution (only containing pure Python modules and packages), the resulting installer will be version independent and have a name like foo-1.0.win32.exe. These installers can even be created on Unix platforms or Mac OS X." http://docs.python.org/distutils/builtdist.html#creating-windows-installers David
On Jun 9, 2009, at 12:52 AM, David Lyon wrote:
Anyway, as long as distutils works under windows, distutils will work under wine - the problem is the toolchain, not distutils.
Yes. That's correct.
Cross compiling has been common practice for the last 30 years.
And still hard to get right. That should certainly not be built in distutils -
Well distutils already has partial support for it.
And the alternative being disccused here is removing the platform specific code.
quoting http://docs.python.org/distutils/builtdist.html#cross-compiling-on-windows
"A built distribution is how you make life as easy as possible for installers of your module distribution: for users of RPM-based Linux systems, it’s a binary RPM; for Windows users, it’s an executable installer; for Debian-based Linux users, it’s a Debian package; and so forth. Obviously, no one person will be able to create built distributions for every platform under the sun, so the Distutils are designed to enable module developers to concentrate on their specialty—writing code and creating source distributions"
At a minimum, distutils should be able to create an .egg that will work on every system.
Before implementing things that no other system has ever done, we should focus on basic things that distutils cannot do today and are expected from a build/distribution tool.
I'm just reading the distutils documentation.....
The documentation is setting an expectation...
Yes - you can be expected to work towards the goals that are documented within your project.
Change the documentation and don't promise it if it is not going to be delivered.....
The point is, getting past the trivial stuff, building packages for different systems is hard, maintaining the code to build packages to other systems is way way harder. For example, Tarek may be using OS/2 warp (Its just an example people) and for him to maintain the rpm/deb/mpkg/win32_inst stuff is actually impossible, so he wants that people on those platforms to take responsability for a little part of distutils so when rpm-ng 6.0 comes either red hat people would say that building python package is not supported or they are going to update the rpm code which is external to disutils. The point is not exactly cross compilation, but cross maintaining of distribution specific code. The only way this actually works now, I would guess, is because rpm format hasn't changed much, and win32_inst has a huge user base. -- Leonardo Santagada santagada at gmail.com
Tarek Ziadé wrote:
Hi Gerry,
During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) :
- it's better for them to have their own release cycles - it's hard for me, as distutils maintainer, to maintain and make evolve os-specific tools. People that are specialists on those OS will do a better job.
Now the problems to reach that goal are:
- some people are reluctant not to have everything included in the stdlib (I am not). I don't think we can all agree on this, and since Guido have encouraged this strategy during the summit, I guess I'm more inclined to follow it nevertheless.
- we need to detect for each existing command (rpm, etc) a project that can take over, prior to remove it from Distutils when appliable .
Now, for debian, since it's not included in distutils, the question is : what is the most used/advanced project ? sdteb ?
++ Tarek
Tarek, I understand your position on this. However, here is what needs considered if you want to split out this functionality. 'bdist_rpm' operates rather well and you can automate the generation of packages using this target. What I would hope not to see is that we end up with an entire fragmentation of different approaches to developing similar targets for other platforms such as targets like 'bdist_deb', 'bdist_mac', etc. Whatever group would be selected to take over this functionality should take over responsibility for all these targets and they be developed in a coordinated manner so that it would be easily understood by the user how to generated packages for multiple platforms using the targets and similar settings in setup.cfg and any auxiliary files similar to rpm_install_sh.txt. If a group can be selected to handle these all these targets with the above goals in mind then I think it would be possible to split this functionality out of the core. But, if the cannot be done, then I think this functionality should remain with the core in order to avoid fragmentation to the approaches in developing these 'bdist_xxx' targets. Regards, Gerry
2009/6/8 Tarek Ziadé <ziade.tarek@gmail.com>:
During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) :
- it's better for them to have their own release cycles
I don't see why. The one I'm interested in (bdist_wininst) doesn't change often, and doesn't need a different release cycle (IMHO).
- it's hard for me, as distutils maintainer, to maintain and make evolve os-specific tools. People that are specialists on those OS will do a better job.
Sorry, but while I appreciate your difficulty, that's not a good enough reason. The rest of Python core is maintained by people who don't have access to all OSes. Packages like subprocess and threading manage. Just take patches from interested community members - you act as a manager rather than a developer in those areas, but that's OK.
Now the problems to reach that goal are:
- some people are reluctant not to have everything included in the stdlib (I am not). I don't think we can all agree on this, and since Guido have encouraged this strategy during the summit, I guess I'm more inclined to follow it nevertheless.
While "Guido says so" beats any other argument, I'd like to hear some good reasons - at least for bdist_wininst. For the Linux package formats, where there's a lot of interest in packaging and strong views on what's "right", I can see the point. For Windows, where nobody gets excited about it, moving bdist_wininst out of the core risks nobody caring enough to keep it going, and thus no packaging solution at all on Windows.
- we need to detect for each existing command (rpm, etc) a project that can take over, prior to remove it from Distutils when appliable .
And a policy on what to do if no such project can be found. Paul.
On Jun 9, 2009, at 1:45 PM, Gerry Reno wrote:
Tarek, I understand your position on this. However, here is what needs considered if you want to split out this functionality. 'bdist_rpm' operates rather well and you can automate the generation of packages using this target. What I would hope not to see is that we end up with an entire fragmentation of different approaches to developing similar targets for other platforms such as targets like 'bdist_deb', 'bdist_mac', etc. Whatever group would be selected to take over this functionality should take over responsibility for all these targets and they be developed in a coordinated manner so that it would be easily understood by the user how to generated packages for multiple platforms using the targets and similar settings in setup.cfg and any auxiliary files similar to rpm_install_sh.txt. If a group can be selected to handle these all these targets with the above goals in mind then I think it would be possible to split this functionality out of the core. But, if the cannot be done, then I think this functionality should remain with the core in order to avoid fragmentation to the approaches in developing these 'bdist_xxx' targets.
What exactly you mean by all these? If I'm not mistaken only rpm, windows and a generic unixy binary packages exists in distutils today, and rpm packages are hardly close to a windows installer package (in terms of install scripts etc) so I don't see what you mean by they being the same now and your fear of they growing different. Also I don't see how having a bit of code inside the stdlib is easier to maintain than code outside it. Seeing how long a patch takes to be applied in the stdlib and how no one can receive commit rights to work on just a submodule of it I think that working as an outside project is actually beneficial to all those projects in case something change on their package tool, and if nothing changes they can keep up pretty easily with distutils. ps: Tarek is one of the exceptions right? He works only on distutils. -- Leonardo Santagada santagada at gmail.com
On 8 Jun, 2009, at 20:15, David Lyon wrote:
the same for rpm, deb and mpkg.
Granted. They're are a bit more tricky. But essentially they are all just zip files with some entry and exit scripts.
To pick some nits: the mpkg one cannot be created on any other system than an OSX one because one of the files that's included in an OSX package has an undocumented format and is created using a tool that's only available on OSX. Ronald
On 8 Jun, 2009, at 22:22, Leonardo Santagada wrote:
The point is not exactly cross compilation, but cross maintaining of distribution specific code. The only way this actually works now, I would guess, is because rpm format hasn't changed much, and win32_inst has a huge user base.
Even for bdist_rpm it is questionable if the implementation is correct w.r.t. the policies of various distributions. Given the discussions about this in the past we wouldn't need a "bdist_rpm" or "bdist_deb" but "bdist_redhat", "bdist_suse", "bdist_debian", ... Bdist_wininst and bdist_msi are somewhat special because the OS vendor does not have a specific policy on how to install python software. Ronald
2009/6/8 Tarek Ziadé <ziade.tarek@gmail.com>:
< plans and reasons to drop some commands from distutils, which I agree with >
- we need to detect for each existing command (rpm, etc) a project that can take over, prior to remove it from Distutils when appliable .
And a policy on what to do if no such project can be found.
I don't think we need a policy at all. If no one wants to support it, it gets dropped. You can't tell volunteers what they have to do. Eric.
On Tue, 09 Jun 2009 14:01:35 -0700, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Even for bdist_rpm it is questionable if the implementation is correct w.r.t. the policies of various distributions. Given the discussions about this in the past we wouldn't need a "bdist_rpm" or "bdist_deb" but "bdist_redhat", "bdist_suse", "bdist_debian", ...
Makes perfect sense to me... I'd include ubuntu in the list of primaries..
Bdist_wininst and bdist_msi are somewhat special because the OS vendor does not have a specific policy on how to install python software.
I'd suggest that they merge into bdist_win. Then be designated with "bdist_win /wininst" and "bdist_win /msi" respectively. One could be made the default option but I don't have a preference. David
On Jun 5, 2009, at 12:22 PM, Gerry Reno wrote:
Has there been any progress on getting the 'bdist_deb' target (akin to 'bdist_rpm') implemented in Distutils? We are looking for behavior that would be similar to bdist_rpm driven by setup.cfg with fully automated generation of packages. There is in existence a set of patches that supposedly does this. Has any progress been made to integrate those patches?
I use stdeb for this purpose: http://pypi.python.org/pypi/stdeb For example, here is a buildbot which automatically builds a new .deb from the Python source whenever a new patch is committed to the revision control repository: http://allmydata.org/buildbot-pycryptopp/builders/linux-amd64-ubuntu- hardy-yukyuk/builds/82/steps/stdeb/logs/stdio Regards, Zooko
On Wed, 10 Jun 2009 08:30:26 -0600, Zooko Wilcox-O'Hearn <zooko@zooko.com> wrote:
I use stdeb for this purpose:
http://pypi.python.org/pypi/stdeb
For example, here is a buildbot which automatically builds a new .deb from the Python source whenever a new patch is committed to the revision control repository:
http://allmydata.org/buildbot-pycryptopp/builders/linux-amd64-ubuntu- hardy-yukyuk/builds/82/steps/stdeb/logs/stdio
That sounds excellent.... To my mind, that's what bdist_* and pypi need.... Not talking about anything contriversial or new or big.... But just better integration so that we can get a build of our projects done more smoothly and uploaded to pypi on several different python versions and platforms... with the built in qa facilities that might already exist to us..... Maybe it is logical to discuss splitting bdist_* from distutils. Given that there is so much platform specific stuff in bdist_*. But then what does it leave distutils as? "packageutils" ? I think there is some logical division between platform independant package definition issues (which I know certain people like working on - and that's good) and the package delivery side. (bdist) The two seem fairly seperate to me. Yet they both make up distutils. Just following on and trying to clarify what other people have put in the discussion pot... David
participants (11)
-
David Cournapeau -
David Lyon -
Eric Smith -
Gerry Reno -
Leonardo Santagada -
Paul Moore -
Ronald Oussoren -
Tarek Ziade -
Tarek Ziadé -
zooko -
Zooko Wilcox-O'Hearn