[Distutils] Low Level API for translating distutils/setuptools metatdata to Debian metadata

[Brian Sutherland]

Hi Zooko,

I just implemented what I was talking about below on the van.pydeb
trunk. Running the script against unstable I get a list of 565 packages
that don't match the default mapping, I've added them to van.pydeb's
database.

The checkin is here:
    http://mail.zope.org/pipermail/checkins/2009-July/036397.html

If you have the time, please have a look over the list to see that it
contains the packages you need correctly. At least pyOpenSSL you mention
below is handled correctly.

I probably will make a new release of van.pydeb in a week or so.

On Wed, Jul 22, 2009 at 05:52:33PM +0200, Brian Sutherland wrote:
> On Wed, Jul 22, 2009 at 08:41:38AM -0600, Zooko Wilcox-O'Hearn wrote:
> >>     * Provides a mapping between python project names and Debian
> >>       binary/source package names
> >>     * Converts setuptools versions to Debian versions while  
> >> maintaining
> >>       sort order
> >>     * Can introspect an .egg-info directory to figure out the Debian
> >>       dependencies for use in the debian/control file. It can also
> >>       handle/understand extras (I Hope!)
> >
> > I looked briefly at this code, and it appears that it is doing purely  
> > syntactic mapping between Debian package names and Python distribution 
> > names, for example:
> >
> > def py_to_bin(setuptools_project):
> >     """Convert a setuptools project name to a debian binary package  
> > name"""
> >     return _PY_TO_BIN.get(setuptools_project, 'python-%s' %  
> > setuptools_project.lower())
> >
> > This works most of the time, but it isn't reliable.  For example, the  
> > module name is "OpenSSL", the distribution name is "pyOpenSSL", and the 
> > Debian package name is "python-openssl":
> >
> > http://packages.debian.org/sid/amd64/python-openssl/filelist
> >
> > That's why I contributed a patch to stdeb which uses the Debian database 
> > of which files are included in which packages (the same database that 
> > generates the web page linked above):
> >
> > http://github.com/astraw/stdeb/blob/647dd441a1712f8df37b5f7f5ba22ab6aeb2c3e7/stdeb/util.py#L135
> >
> > The way stdeb does it looks in the database for files named  
> > "$DISTRIBUTION-$VERSION-py$PYTHONVERSION.egg-info".  The Debian package 
> > that includes such a file is the Debian package that you need to install 
> > in order to satisfy a dependency on $DISTRIBUTION, $VERSION.  This works 
> > regardless of whether the Python package is built with distutils or 
> > setuptools, and indeed it works for all packages that I know of.  (There 
> > is actually one exception: the Debian package for setuptools itself 
> > doesn't include a version number in its .egg-info filename:
> >
> > http://packages.debian.org/sid/amd64/python-setuptools/filelist
> >
> > It has a file named:
> >
> > /usr/share/pyshared/setuptools.egg-info/
> >
> > I guess we should add a fall-back-with-warning behavior to stdeb that if 
> > it can't find "$DISTRIBUTION-$VERSION.egg-info", but it can find  
> > "$DISTRIBUTION.egg-info", then it should (optionally) assume that the  
> > Debian package that has that file will satisfy the requirement,  
> > regardless of the version number in the requirement.   That, or someone 
> > should open a ticket asking Debian to add a version number to that 
> > filename.
> >
> > The exact regexp is currently:
> >
> > egginfore=("(/(%s)(?:-[^/]+)?(?:-py[0-9]\.[0-9.]+)?\.egg-info)"
> >                % '|'.join(req.project_name for req in requirements))
> >
> > If you would be interested in including this mechanism to query the  
> > database in van.pydeb, I would be happy to advise you.
> 
> Hi Zooko,
> 
> van.pydeb is designed to be run at package build time, rather than at
> the time you create the source package. I assume that's when stdeb's
> code runs? Using apt-file during package build time on one of the Debian
> project's auto-builders will not be acceptable (I assume).
> 
> Also, "isn't reliable" has very different meanings :) van.pydeb is
> reliable in that it produces the same answer independently of the
> machine it's run on. But it's not reliable in that often the answer is
> just plain wrong;)
> 
> The results of stdeb's mechanism depend on the configuration of the
> machine where you run it (or even if you havn't run apt-file update
> recently).
> 
> So, I don't think van.pydeb can use stdeb's mechanism as-is, but I'd
> love to see a patch (with tests!) for a variation of it. I was thinking
> of writing a script/function that could use apt-file to generate a list
> of python->debian package mappings that don't fit the heuristic.
> 
> So py_to_bin could be re-written as:
> 
>     def py_to_bin(setuptools_project):
>         """Convert a setuptools project name to a debian binary package name"""
>         result = _HANDWRITTEN_PY_TO_BIN.get(setuptools_project)
>         if result is None:
>             result = _APT_FILE_GENERATED_PY_TO_BIN.get(setuptools_project)
>         if result is None:
>             result = 'python-%s' % setuptools_project.lower())
>         return result
> 
> The _APT_FILE_GENERATED_PY_TO_BIN could be re-generated periodically.
> I'm hoping that'll be enough given that:
> 
>     * packages don't change names that frequently
>     * there should be few that don't fit the heuristic (*cough* policy *cough*)
> 
> > Regards,
> >
> > Zooko
> 
> -- 
> Brian Sutherland

-- 
Brian Sutherland