[Catalog-sig] start on static generation, and caching - apache config.

[Phillip J. Eby]

At 08:09 AM 7/12/2007 +1000, Richard Jones wrote:
>On Thu, 12 Jul 2007, you wrote:
> > Yup.  Absolutely.  That's why it we should change the index or
> > setuptools, or both.  IMO, it makes the most sense to change the
> > index to have setuptools specific pages, in addition to the ones for
> > humans, that allow:
>
>... you know about the XML-RPC interface, yes?
>
>http://wiki.python.org/moin/CheeseShopXmlRpc
>
>I never fully understood why setuptools went with HTML scraping instead of
>XML-RPC.

Fundamentally, it was because the XML-RPC API did not then (and does 
not now) provide everything that's needed.  (As I mentioned a few of 
the other times you asked this.)  The API has improved and added some 
of the missing bits, but not all of them.

There are two pieces still missing:

1. Access to "hidden" packages' release info

2. Links in the long_description that are rendered by PyPI's web interface

Without #2, we can't pick up author-provided Subversion links; see:

   http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall

for details.

With this information, easy_install could be changed to use the 
XML-RPC API....  *but* it would make even *more* round-trips to PyPI 
than it does now, unless those APIs were also designed differently 
than the ones that exist now, because you would need at least one 
search to find the correct package and its PKG-INFO, and another 
search to get the download files.  Currently, it can at least get 
both of these in one trip, if the package name is an exact match.

To answer Martin's question of why setuptools doesn't "trust" the 
PyPI specification even more, it's because having chosen to use the 
web interface to get the information, I thought it prudent to use 
only that subset of the web interface that could be easily duplicated 
using simple Apache directory indexes, since that meant someone could 
create their own index or mirror a portion of PyPI without having to 
implement its entire feature set.  This later proved prudent when Jim 
wanted to have tests of his buildout framework that did not rely on 
PyPI being up, as it made it easier to create a mock PyPI for unit 
testing purposes.

To be honest, the one thing I did *not* anticipate in this design was 
that Jim would be making 20 releases of the same package available in 
"unhidden" form.  :)