On Fri, May 29, 2009 at 08:35:16AM -0700, Andrew Straw wrote:

Brian Sutherland wrote:

...

Hi,

I've just published a very small library that does three things so far:

* Provides a mapping between python project names and Debian binary/source package names * Converts setuptools versions to Debian versions while maintaining sort order * Can introspect an .egg-info directory to figure out the Debian dependencies for use in the debian/control file. It can also handle/understand extras (I Hope!)

It's basically an attempt to find a common ground between all the projects doing automated Python->Debian packaging. I have a feeling everyone is re-implementing this code all the time.

The stdeb package certainly needs most of these things, so it would be nice to consolidate the implementation. It looks like there's some very useful stuff in van.pydeb along with good tests. I'll attempt to convert stdeb to use it. Do you have a public source code repository for van.pydeb in case I want to start making patches?

I added a link to the svn repository to the web page, and am willing to accept patches!

...

And an example of direct use in a packaging situation with complex dependencies is here:

http://svn.debian.org/viewsvn/pkg-zope/zope.component/trunk/debian/rules?vie...

As an aside, now that I've been looking at debian/rules files made for debhelper 7 and python-support, that file looks bloated to my eye. (Of course, if you're targeting older Debians without dh7, there's not much you can do.) FWIW stdeb is growing dh7 support in the "dh7" branch. For example, stdeb just generated this debian/rules file, which is completely functional:

#!/usr/bin/make -f

# This file was automatically generated by stdeb 0.3+git+dh7 at # Thu, 28 May 2009 15:38:44 -0700

%: dh $@

Yes we were thinking of using something similar; i.e. an includable makefile. But this does look very interesting, I will definitely investigate. But just to point it out, it's way beyond the scope of van.pydeb to define the rules file, that's for things like stdeb to do. -- Brian Sutherland

On Sat, May 30, 2009 at 07:27:06PM -0400, David Lyon wrote:

On Fri, 29 May 2009 08:35:16 -0700, Andrew Straw <strawman@astraw.com> wrote:

...

Brian Sutherland wrote:

...

I've just published a very small library that does three things so far:

* Provides a mapping between python project names and Debian binary/source package names * Converts setuptools versions to Debian versions while maintaining sort order * Can introspect an .egg-info directory to figure out the Debian dependencies for use in the debian/control file. It can also handle/understand extras (I Hope!)

It's basically an attempt to find a common ground between all the projects doing automated Python->Debian packaging. I have a feeling everyone is re-implementing this code all the time.

Maybe not everybody - but I am sure there are a few...

at least there are many who would like to do it..

Recently I posted a suggestion to Andrea Gavana about py2exe on the wxPython list about his excellent gui interface for py2exe.

My use case is this... I'm writing a package manager for python.. I want it to be deployed to every python platform... I dont have all those platforms....

I want p2exe or distutils (Distribution Utils?? is that what it stands for?) to take my program and make it for every platform...

- windows - mac - linux (ubuntu,debian,suse,redhat)

It's actually not so many..

No, I am not talking jibberish or pie in the sky here..

Here's specifically what needs doing.....

1) create a three page "wizard" style app in wxpython

2) the app has checkboxes to allow the user to select the operating systems that they want to deploy to

3) p2app for mac, py2exe for windows.. whichever one for linux is driven.. much like gui2exe

4) Cheetah templates are used to create the necessary output scripts from the meta information.

(building scripts in code is so 1970s.. we don't want to do it that way again)

5) The scripts get run and the user ends up with .debs .rpm... windows installer

basic structure...

for plat in user_selected_platforms: build_for_platform(plat)

def build_for_platform(platform): if platform == "mac": .. elif ....

I think distutils has been in a "frozen" state for the last few years. With no notable improvements apart from bug fixes and internal work.

Given that the general consensus is largely to leave package/application management to the operating system, what I describe above is a more modern way to accomplish what people are trying to do with less effort on the part of all...

If a user makes a python app... why shouldn't they be able to run a distutils script to "distribute" to a platform they don't have....

Building ubuntu or debian packages is no longer a black art....

To be honest, it still is;)

It's 2009 now nearly 2010...

When was the last time distutils ever got a major upgrade....?

Is this not what "disutils" should be doing?

I would personally be very happy if distutils kept out of the business of making packages for anything but python to consume. Making OS level packages is, in my view, something that should be built on top of, not inside distutils. So the current distutils development is going in the right direction for me: Specifying common metadata formats and making APIs to access that.

in this day and age?

David

-- Brian Sutherland

Hi Chris, I started prototyping of this a month or so ago but haven't had a chance to do much more on it too now. Maybe in the next I'll put more effort into this multi-platform package builder and register it as a project somewhere. If you want to send me your scripts offline, I'd be interested seeing them. Unfortunately, I've been busy writing documentation for the Package Manager and trying to figure out how to use distutils myself. It's a steep learning curve. I get the point about having some web based app (pypi? haha) where you type in a package name, click the platforms.. and out come all the different packages for all the different python versions and operating systems. That's all I ever asked for.... :-) but I only know backend stuff and gui programming. I kindof know how to seperate the two so doing a web based one is totally impossible. Hopefully I'll find enough hacking time in the next few months.. David On Sat, 11 Jul 2009 00:32:49 -0700 (PDT), Chris Dew <cmsdew@googlemail.com> wrote:

On May 31, 12:27 am, David Lyon <david.l...@preisshare.net> wrote:

...

 1) create a three page "wizard" style app in wxpython

After spending a three hours trying to figure out how to correctly build a debian package from Python sources, I agree it needs to be a lot easier.

But scripts allow for continuous integration, which is a very good thing, as servers build packages and run tests nearly for free.

GUIs require an expensive person to click buttons, which is also not an interesting task for the person.

If anyone's tempted to go down this route and create a GUI system for Python distro-packaging, please build in a scriptable/headless option.

On Wed, Jul 22, 2009 at 08:41:38AM -0600, Zooko Wilcox-O'Hearn wrote:

...

* Provides a mapping between python project names and Debian binary/source package names * Converts setuptools versions to Debian versions while maintaining sort order * Can introspect an .egg-info directory to figure out the Debian dependencies for use in the debian/control file. It can also handle/understand extras (I Hope!)

I looked briefly at this code, and it appears that it is doing purely syntactic mapping between Debian package names and Python distribution names, for example:

def py_to_bin(setuptools_project): """Convert a setuptools project name to a debian binary package name""" return _PY_TO_BIN.get(setuptools_project, 'python-%s' % setuptools_project.lower())

This works most of the time, but it isn't reliable. For example, the module name is "OpenSSL", the distribution name is "pyOpenSSL", and the Debian package name is "python-openssl":

http://packages.debian.org/sid/amd64/python-openssl/filelist

That's why I contributed a patch to stdeb which uses the Debian database of which files are included in which packages (the same database that generates the web page linked above):

http://github.com/astraw/stdeb/blob/647dd441a1712f8df37b5f7f5ba22ab6aeb2c3e7...

The way stdeb does it looks in the database for files named "$DISTRIBUTION-$VERSION-py$PYTHONVERSION.egg-info". The Debian package that includes such a file is the Debian package that you need to install in order to satisfy a dependency on $DISTRIBUTION, $VERSION. This works regardless of whether the Python package is built with distutils or setuptools, and indeed it works for all packages that I know of. (There is actually one exception: the Debian package for setuptools itself doesn't include a version number in its .egg-info filename:

http://packages.debian.org/sid/amd64/python-setuptools/filelist

It has a file named:

/usr/share/pyshared/setuptools.egg-info/

I guess we should add a fall-back-with-warning behavior to stdeb that if it can't find "$DISTRIBUTION-$VERSION.egg-info", but it can find "$DISTRIBUTION.egg-info", then it should (optionally) assume that the Debian package that has that file will satisfy the requirement, regardless of the version number in the requirement. That, or someone should open a ticket asking Debian to add a version number to that filename.

The exact regexp is currently:

egginfore=("(/(%s)(?:-[^/]+)?(?:-py[0-9]\.[0-9.]+)?\.egg-info)" % '|'.join(req.project_name for req in requirements))

If you would be interested in including this mechanism to query the database in van.pydeb, I would be happy to advise you.

Hi Zooko, van.pydeb is designed to be run at package build time, rather than at the time you create the source package. I assume that's when stdeb's code runs? Using apt-file during package build time on one of the Debian project's auto-builders will not be acceptable (I assume). Also, "isn't reliable" has very different meanings :) van.pydeb is reliable in that it produces the same answer independently of the machine it's run on. But it's not reliable in that often the answer is just plain wrong;) The results of stdeb's mechanism depend on the configuration of the machine where you run it (or even if you havn't run apt-file update recently). So, I don't think van.pydeb can use stdeb's mechanism as-is, but I'd love to see a patch (with tests!) for a variation of it. I was thinking of writing a script/function that could use apt-file to generate a list of python->debian package mappings that don't fit the heuristic. So py_to_bin could be re-written as: def py_to_bin(setuptools_project): """Convert a setuptools project name to a debian binary package name""" result = _HANDWRITTEN_PY_TO_BIN.get(setuptools_project) if result is None: result = _APT_FILE_GENERATED_PY_TO_BIN.get(setuptools_project) if result is None: result = 'python-%s' % setuptools_project.lower()) return result The _APT_FILE_GENERATED_PY_TO_BIN could be re-generated periodically. I'm hoping that'll be enough given that: * packages don't change names that frequently * there should be few that don't fit the heuristic (*cough* policy *cough*)

Regards,

Zooko

-- Brian Sutherland

On Wednesday,2009-07-22, at 9:52 , Brian Sutherland wrote:

van.pydeb is designed to be run at package build time, rather than at the time you create the source package. I assume that's when stdeb's code runs? Using apt-file during package build time on one of the Debian project's auto-builders will not be acceptable (I assume).

So if I understand correctly there is a Python package with its accompanying metadata, in a setup.py and possibly a setup.cfg file, and then there is some step which produces a "Debian source package", which is a thing that includes a .dsc file, a .orig.tar.gz file, and a .diff.gz file, right? And then you transport those files to a build machine and run a package builder to turn those files into a .deb. So, stdeb is a tool to accomplish that first step, of taking in a setup.py and writing out a .dsc file and related files. Now the problem I'm having is that I want stdeb to *correctly* (i.e., without guessing) determine which Debian package provides which Python distribution, and the answer to that question depends on which Debian (/Ubuntu) distribution you are building for. There is no way for that question to be automatically, correctly answered in a way that is generic for all Debian distributions. That means that you can't create a .dsc-and-related-files one time, generically, and then copy them to the autobuilders and have a bunch of .deb's -- one for each specific distribution -- be built from the same .dsc. So they way I currently do it [1, 2] is that I run stdeb and then run dpkg-buildpackage on many separate buildslaves, each of which is running a different Debian distribution. This works fine! The result is a .deb for that specific Debian distribution which has the exact right Dependencies for that distribution. Regards, Zooko [1] http://allmydata.org/buildbot-pycryptopp/waterfall [2] http://allmydata.org/buildbot-zfec/waterfall