Tarek:

Will the bdist_deb code be based on or copied from the existing stdeb tool? Should people who are interested in bdist_deb be using stdeb right now in order to find bugs in it and learn what they like/ dislike about it?

Regards,

Zooko

On Wed, 10 Jun 2009 08:30:26 -0600, Zooko Wilcox-O'Hearn zooko@zooko.com wrote:

I use stdeb for this purpose:

http://pypi.python.org/pypi/stdeb

For example, here is a buildbot which automatically builds a new .deb from the Python source whenever a new patch is committed to the revision control repository:

http://allmydata.org/buildbot-pycryptopp/builders/linux-amd64-ubuntu- hardy-yukyuk/builds/82/steps/stdeb/logs/stdio

That sounds excellent....

To my mind, that's what bdist_* and pypi need....

Not talking about anything contriversial or new or big....

But just better integration so that we can get a build of our projects done more smoothly and uploaded to pypi on several different python versions and platforms...

with the built in qa facilities that might already exist to us.....

Maybe it is logical to discuss splitting bdist_* from distutils. Given that there is so much platform specific stuff in bdist_*. But then what does it leave distutils as? "packageutils" ?

I think there is some logical division between platform independant package definition issues (which I know certain people like working on - and that's good) and the package delivery side. (bdist)

The two seem fairly seperate to me. Yet they both make up distutils.

Just following on and trying to clarify what other people have put in the discussion pot...

David

Hi Gerry,

During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) :

- it's better for them to have their own release cycles - it's hard for me, as distutils maintainer, to maintain and make evolve os-specific tools. People that are specialists on those OS will do a better job.

Now the problems to reach that goal are:

- some people are reluctant not to have everything included in the stdlib (I am not). I don't think we can all agree on this, and since Guido have encouraged this strategy during the summit, I guess I'm more inclined to follow it nevertheless.

- we need to detect for each existing command (rpm, etc) a project that can take over, prior to remove it from Distutils when appliable .

Now, for debian, since it's not included in distutils, the question is : what is the most used/advanced project ? sdteb ?

++ Tarek

On Jun 8, 2009, at 8:50 PM, David Lyon wrote:

On Mon, 8 Jun 2009 14:28:52 +0200, Tarek Ziadé ziade.tarek@gmail.com wrote:

...

During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) :

• it's better for them to have their own release cycles
• it's hard for me, as distutils maintainer, to maintain and make

evolve os-specific tools. People that are specialists on those OS will do a better job.

Let's think about this.... and what it means.....

It seems that this direction comes down to "we don't want to maintain it" rather than anything else.

The whole point of distutils is to be able to make up a package in such a way that it can be distributed to every platform.... right?

So removing the code that is specific to specific platforms is just collapsing the whole project.

Rather than "removing" platform specific code... and rendering distutils crippled... why not work on developing the code so that it operates more like a "cross-compiler"...

That is...

runs on any platform...

builds for any platform....

Because this is almost impossible... for example building a win_inst package on linux (let alone other unixes, etc) is very very hard (specially if it involves extensions), the same for rpm, deb and mpkg.

Distutils should provide all tools to make this happen, but the package creation itself should be let to each system. This is the same problem with svn and setuptools where you end up with non working code for long periods of time. Now that python and probably a ton of other projects are going to use mercurial this can even become worse.

-- Leonardo Santagada santagada at gmail.com

David Lyon wrote:

Not true...

Don't forget, cygwin runs under windows

And how does that help to build a linux binary ?

, and wine runs under linux

Yes, but Visual Studio does not. So you are out of luck if you need full compatibility with pre-built python binaries on python.org. It works almost ok with mingw, but not always - and it is far from working well on windows 64 bits BTW.

Anyway, as long as distutils works under windows, distutils will work under wine - the problem is the toolchain, not distutils.

Cross compiling has been common practice for the last 30 years.

And still hard to get right. That should certainly not be built in distutils - a sane build tool needs very little knowledge about cross compilation, really. Python itself cannot be cross-compiled out of the box anyway.

At a minimum, distutils should be able to create an .egg that will work on every system.

Before implementing things that no other system has ever done, we should focus on basic things that distutils cannot do today and are expected from a build/distribution tool.

Those are hard problems to get right, saying that it is not hard because it is just moving files and scripts does not really help. That's like saying programming is easy because it is just inputting characters in a text file.

cheers,

David

On Jun 9, 2009, at 12:52 AM, David Lyon wrote:

...

Anyway, as long as distutils works under windows, distutils will work under wine - the problem is the toolchain, not distutils.

Yes. That's correct.

...

...

Cross compiling has been common practice for the last 30 years.

And still hard to get right. That should certainly not be built in distutils -

Well distutils already has partial support for it.

And the alternative being disccused here is removing the platform specific code.

quoting http://docs.python.org/distutils/builtdist.html#cross-compiling-on-windows

"A built distribution is how you make life as easy as possible for installers of your module distribution: for users of RPM-based Linux systems, it’s a binary RPM; for Windows users, it’s an executable installer; for Debian-based Linux users, it’s a Debian package; and so forth. Obviously, no one person will be able to create built distributions for every platform under the sun, so the Distutils are designed to enable module developers to concentrate on their specialty—writing code and creating source distributions"

...

...

At a minimum, distutils should be able to create an .egg that will work on every system.

Before implementing things that no other system has ever done, we should focus on basic things that distutils cannot do today and are expected from a build/distribution tool.

I'm just reading the distutils documentation.....

The documentation is setting an expectation...

Yes - you can be expected to work towards the goals that are documented within your project.

Change the documentation and don't promise it if it is not going to be delivered.....

The point is, getting past the trivial stuff, building packages for different systems is hard, maintaining the code to build packages to other systems is way way harder.

For example, Tarek may be using OS/2 warp (Its just an example people) and for him to maintain the rpm/deb/mpkg/win32_inst stuff is actually impossible, so he wants that people on those platforms to take responsability for a little part of distutils so when rpm-ng 6.0 comes either red hat people would say that building python package is not supported or they are going to update the rpm code which is external to disutils.

The point is not exactly cross compilation, but cross maintaining of distribution specific code. The only way this actually works now, I would guess, is because rpm format hasn't changed much, and win32_inst has a huge user base.

-- Leonardo Santagada santagada at gmail.com

On Tue, 09 Jun 2009 14:01:35 -0700, Ronald Oussoren ronaldoussoren@mac.com wrote:

Even for bdist_rpm it is questionable if the implementation is correct w.r.t. the policies of various distributions. Given the discussions about this in the past we wouldn't need a "bdist_rpm" or "bdist_deb" but "bdist_redhat", "bdist_suse", "bdist_debian", ...

Makes perfect sense to me... I'd include ubuntu in the list of primaries..

Bdist_wininst and bdist_msi are somewhat special because the OS vendor does not have a specific policy on how to install python software.

I'd suggest that they merge into bdist_win. Then be designated with "bdist_win /wininst" and "bdist_win /msi" respectively. One could be made the default option but I don't have a preference.

David

On 8 Jun, 2009, at 20:15, David Lyon wrote:

...

the same for rpm, deb and mpkg.

Granted. They're are a bit more tricky. But essentially they are all just zip files with some entry and exit scripts.

To pick some nits: the mpkg one cannot be created on any other system than an OSX one because one of the files that's included in an OSX package has an undocumented format and is created using a tool that's only available on OSX.

Ronald

Tarek Ziadé wrote:

Hi Gerry,

During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) :

• it's better for them to have their own release cycles
• it's hard for me, as distutils maintainer, to maintain and make

evolve os-specific tools. People that are specialists on those OS will do a better job.

Now the problems to reach that goal are:

• some people are reluctant not to have everything included in the

stdlib (I am not). I don't think we can all agree on this, and since Guido have encouraged this strategy during the summit, I guess I'm more inclined to follow it nevertheless.

• we need to detect for each existing command (rpm, etc) a project

that can take over, prior to remove it from Distutils when appliable .

Now, for debian, since it's not included in distutils, the question is : what is the most used/advanced project ? sdteb ?

++ Tarek

Tarek, I understand your position on this. However, here is what needs considered if you want to split out this functionality. 'bdist_rpm' operates rather well and you can automate the generation of packages using this target. What I would hope not to see is that we end up with an entire fragmentation of different approaches to developing similar targets for other platforms such as targets like 'bdist_deb', 'bdist_mac', etc. Whatever group would be selected to take over this functionality should take over responsibility for all these targets and they be developed in a coordinated manner so that it would be easily understood by the user how to generated packages for multiple platforms using the targets and similar settings in setup.cfg and any auxiliary files similar to rpm_install_sh.txt. If a group can be selected to handle these all these targets with the above goals in mind then I think it would be possible to split this functionality out of the core. But, if the cannot be done, then I think this functionality should remain with the core in order to avoid fragmentation to the approaches in developing these 'bdist_xxx' targets.

Regards, Gerry

2009/6/8 Tarek Ziadé ziade.tarek@gmail.com:

During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) :

• it's better for them to have their own release cycles

I don't see why. The one I'm interested in (bdist_wininst) doesn't change often, and doesn't need a different release cycle (IMHO).

• it's hard for me, as distutils maintainer, to maintain and make

evolve os-specific tools. People that are specialists on those OS  will do a better job.

Sorry, but while I appreciate your difficulty, that's not a good enough reason. The rest of Python core is maintained by people who don't have access to all OSes. Packages like subprocess and threading manage. Just take patches from interested community members - you act as a manager rather than a developer in those areas, but that's OK.

Now the problems to reach that goal are:

• some people are reluctant not to have everything included in the

stdlib (I am not). I don't think we can all agree on this,  and since Guido have encouraged this strategy during the summit, I guess I'm more inclined to follow it nevertheless.

While "Guido says so" beats any other argument, I'd like to hear some good reasons - at least for bdist_wininst. For the Linux package formats, where there's a lot of interest in packaging and strong views on what's "right", I can see the point. For Windows, where nobody gets excited about it, moving bdist_wininst out of the core risks nobody caring enough to keep it going, and thus no packaging solution at all on Windows.

• we need to detect for each existing command (rpm, etc) a project

that can take over, prior to remove it from Distutils when appliable .

And a policy on what to do if no such project can be found.

Paul.