[Catalog-sig] Recent PyPI changes

[Sridhar Ratnakumar]

On 2010-07-27, at 9:31 AM, Sridhar Ratnakumar wrote:

> 
> On 2010-07-27, at 2:55 AM, M.-A. Lemburg wrote:
> 
>> Alexis Metaireau wrote:
>>> On Tue, 2010-07-27 at 09:52 +0100, Chris Withers wrote:
>>>> there is now a way to request release information in JSON,
>>>>> see http://tinyurl.com/38lefsp 
>>> That's indeed cool, and useful, but we can't rely on this while
>>> crawling, too bad this JSON is not replicated on the mirrors.
>>> 
>>> It could help a lot, since there is currently no way to request the
>>> metadatas statically in others way that downloading the distribution
>>> archives and extracting them. (we also could use xmlrpc, but that's not
>>> static).
>>> 
>>> What's the process I have to follow in order to get this mirrored ? Does
>>> that sounds good for you ? IOW, whats needed to have this as a
>>> requirements for mirrors? 
>> 
>> Easiest would be to dump the complete release information
>> (PKG-INFO) to a text file using the name format <version>_pkg_info
>> in the simple/ index.
> 
> What we ended up doing for our internal comprehensive Python package mirror is this:
> 
> - for pkg in changed_since_yesterday(pypi): download_source_using_easy_install(pkg) 
> - extract PKG-INFO out of source
> - extract 'requires.txt' (if it exists) out of source 
> 
> If you want to find the dependencies of a package, it can only be found in requires.txt (not PKG-INFO).
> 
> But then even if PKG-INFO/requires.txt is provided by /simple, keep in mind that it won't be comprehensive. Not all package authors use PyPI for serving their source distributions. (This is why we also had to use setuptools.package_index).

And, of course, not all source distributions include a PKG-INFO file, in which case it becomes mandatory to run "python setup.py egg_info" (after patching setup.py to do 'import setuptools') to generate it.  Twisted-10.1.0.tar.bz2 is one example of this.

There are still several other minor issues with metadata mirroring that I don't recall at the moment.

-srid