[Catalog-sig] Standardized way of getting to a package download URL in Cheeseshop

Grig Gheorghiu grig at gheorghiu.net
Mon Nov 14 05:22:32 CET 2005

--- "Phillip J. Eby" <pje at telecommunity.com> wrote:

> At 01:33 PM 11/9/2005 -0800, Grig Gheorghiu wrote:
> >I'm working on a project I called "Cheesecake" which aims to
> quantify
> >the "goodness" of Python packages based on various indicators such
> as
> >presence of special files (README, LICENSE, etc.), special
> directories
> >(tests, doc, etc.), percentage of modules/classes/methods/functions
> >with docstrings, etc.
> >
> >It would be nice if there was a standardized way to get to a
> download
> >URL for a given package listed at the Cheeseshop.
> setuptools' "package_index" module has facilities to scan Cheeseshop
> pages 
> and the pages they link to for URLs that are downloadable 
> distributions.  Also, the "easy_install" command-line tool can be
> used to 
> fetch and extract editable source (or check out from subversion) the
> source 
> code of a project.  For example:
>      $ easy_install -eb~/projects SQLObject
>      Searching for SQLObject
>      Reading http://www.python.org/pypi/SQLObject/
>      Reading http://sqlobject.org
>      Best match: SQLObject 0.7.0
>      Downloading 
>      Processing SQLObject-0.7.0.tar.gz
>      Extracted editable version of SQLObject to ~/projects/sqlobject
> Notice that this works even with projects not hosted at the
> Cheeseshop, 
> even if they are on Sourceforge mirrors:
>      $ easy_install -eb~/projects ctypes
>      Searching for ctypes
>      Reading http://www.python.org/pypi/ctypes/
>      Reading http://starship.python.net/crew/theller/ctypes/
>      Reading
> http://sourceforge.net/project/showfiles.php?group_id=71702
>      Best match: ctypes 0.9.6
>      Downloading 
> http://prdownloads.sourceforge.net/ctypes/ctypes-0.9.6.zip?download
>      Requesting redirect to (randomly selected) 'surfnet' mirror
>      Downloading 
> http://surfnet.dl.sourceforge.net/sourceforge/ctypes/ctypes-0.9.6.zip
>      Processing ctypes-0.9.6.zip
>      Extracted editable version of ctypes to ~/projects/ctypes
> As long as a download link is available from one of these three
> places, it 
> can be found:
>   1. The cheeseshop page for the package
>   2. The page referenced by the cheeseshop "download url" link
>   3. The page referenced by the cheeseshop "home page" link
> I would say that any project whose download URL(s) can't be found by 
> easy_install should get a bad score on your metrics, since it's going
> to be 
> pretty tough for the *user* to find the package!
> EasyInstall is not limited to source downloads, however.  It's just
> that 
> the '-e' option tells it you want an "editable" distribution (e.g., a
> Subversion checkout preferably, with a source distribution being
> second 
> best), and the '-b' option tells it the base directory under which
> the 
> project directory should be created.  (Ordinarily, EasyInstall just 
> attempts to get binaries or build and install them.)

My immediate goal is to download a package in a "sandbox" directory,
unpack it and examine its files and directories.

I found a way to download a package using setuptools by looking it up
on PyPI by its "short" name (e.g. "funkload"). I do this in my code:

from setuptools.package_index import PackageIndex
pkgindex = PackageIndex()
output = pkgindex.download(name, sandbox)

This works just fine, but PackageIndex downloads the first package it
finds -- which in many cases is an egg file. The problem with egg
packages is that they don't contain the source distribution, so my
Cheesecake module doesn't find docs, tests, special files, etc. inside
an egg.

Is there a way I can tell setuptools what type of package I want it to
download (e.g. "tar.gz" or "zip")? I guess I could find the answer to
this question myself by poring some more over the setuptools source
code, but if you know the answer already, I'd appreciate it!

My intention is to try to always download the source distribution for a
package, and add extra points if I also find an egg for that package.



More information about the Catalog-sig mailing list