At 04:04 AM 7/5/2006 -0400, Jim Fulton wrote:
>On Jun 23, 2006, at 4:51 PM, Jim Fulton wrote:
>>That's a lot of screen scraping. :)
>>It would be good to capture this as part of the documentation IMO
>>>I'm considering adding XML-RPC support to easy_install in 0.7,
>>>though. PyPI now has a nice XML-RPC API that is more responsive
>>>than the web UI, and it supports case-insensitive partial match
>>>searches, making it suitable for easy_install to query when a typed-
>>>in name doesn't exactly match the spelling of a PyPI entry.
>>I think that would be much better.
>I just wanted to emphasize that I think this would be a good
Patches welcome. :) Note that there should still be a fallback to the
screen scraping code in case of a problem with the XML-RPC, to allow people
to continue using static mirrors of PyPI or imitation PyPIs without needing
to support XML-RPC.
> I was just talking to Richard, and he pointed out that the
>current approach is a problem for him, because it means he can't
>evolve the pypi UI without risking breaking setuptools.
What I would suggest is creating a "microformat" for marking up web pages
with sniffable information. For example, adding rel="homepage" and
rel="download" to the links that go to those URLs.
In other words, invisible hints on the page to supplement the visible
information. Then, I could change easy_install to start using the
invisible hints, and drop the visible ones, freeing PyPI to evolve the UI
While the XML-RPC API would be great, I still want easy_install to be able
to use a package index that's made from static files, and that requires
some kind of screen scraping. So, let's make it invisible scraping of a
documented format, so that anybody can use it, with whatever visual formats
Currently, easy_install gets most of its information from URLs; the only
actual scraping of visible data is of the title, the download MD5's, and
the table cells that identify links as being to the home page or download
URL (since it needs to specifically identify these in order to spider them).
The MD5 information dependency could be removed if PyPI included "#md5=..."
at the end of the download URLs; easy_install can see that information and
use it. The table cell checking could be removed by adding
'rel="easy_install"' or something like that to the spiderable links.
The title checking is used to distinguish pages that list multiple packages
from pages that list single packages. I don't have any ready ideas as to
how that could or should be represented in a semantic (as opposed to
visual) way. Your thoughts?