[Catalog-sig] simple package index has links back into the human interface

[Jim Fulton]

A while ago, I created an experimental PyPI mirror:

   http://download.zope.org/ppix/

Recently, I've been working on a mirror of the new simple index:

   http://download.zope.org/simple/

This mirrors the pages at:

   http://cheeshop.python.org/simple/

In experimenting with this, I found that buildouts were taking much  
longer (e.g. 70 second vs 40 seconds) using the simpler mirror than  
using the ppix mirror.  I added some additional logging and found  
that when using the simple index, buildout was getting a lot of non- 
simple pages.

A common practice is to use the package index page for a project as  
the project home page.  There's no point in a simple page including a  
link to the non-simple page as it contains the same or less  
information.  I filter these pages out in the ppix index.  The simple  
index doesn't. For example, the simple page for zc.buildout:

   http://cheeseshop.python.org/simple/zc.buildout

has home page links to http://www.python.org/pypi/zc.buildout.

Martin, can you filter links like this out of the simple output?  (If  
not, I'll filter them out when I mirror.)

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org