[Catalog-sig] Recent PyPI changes

[Sridhar Ratnakumar]

On 2010-07-27, at 2:55 AM, M.-A. Lemburg wrote:

> Alexis Metaireau wrote:
>> On Tue, 2010-07-27 at 09:52 +0100, Chris Withers wrote:
>>> there is now a way to request release information in JSON,
>>>>  see http://tinyurl.com/38lefsp 
>> That's indeed cool, and useful, but we can't rely on this while
>> crawling, too bad this JSON is not replicated on the mirrors.
>> 
>> It could help a lot, since there is currently no way to request the
>> metadatas statically in others way that downloading the distribution
>> archives and extracting them. (we also could use xmlrpc, but that's not
>> static).
>> 
>> What's the process I have to follow in order to get this mirrored ? Does
>> that sounds good for you ? IOW, whats needed to have this as a
>> requirements for mirrors? 
> 
> Easiest would be to dump the complete release information
> (PKG-INFO) to a text file using the name format <version>_pkg_info
> in the simple/ index.

What we ended up doing for our internal comprehensive Python package mirror is this:

- for pkg in changed_since_yesterday(pypi): download_source_using_easy_install(pkg) 
- extract PKG-INFO out of source
- extract 'requires.txt' (if it exists) out of source 

If you want to find the dependencies of a package, it can only be found in requires.txt (not PKG-INFO).

But then even if PKG-INFO/requires.txt is provided by /simple, keep in mind that it won't be comprehensive. Not all package authors use PyPI for serving their source distributions. (This is why we also had to use setuptools.package_index).

-srid