[Catalog-sig] PEP 381 Sample/Reference Implementation, Mirroring PyPI Meta Data

[ssteinerX@gmail.com]

I've taken this off of the "PyPI down" thread (sorry, Chris).

I was poking at a little project that came out of this discussion on distutils around December 27th-30th.  http://mail.python.org/pipermail/distutils-sig/2009-December/015178.html
http://mail.python.org/pipermail/distutils-sig/2009-December/015177.html

Essentially, the goal is/was to put a pkl of all of the meta data up so that someone wanting to mirror just the metadata (for whatever reason) could start with an up-to-date set, then just pull deltas from pypi.

I thought it'd be smart to just use and possibly improve the "sample code" for PEP 381 rather than start from scratch.

Now I have:

On Jan 12, 2010, at 5:20 PM, Martin v. Löwis wrote:

>> If z3c.pypimirror is not the starting point then is there a different
>> implementation that will be used as a 'jumpstart'?
> 
> I'll write it from scratch, using urllib (or perhaps Twisted). We
> evalutated z3c.pypimirror at the last PyCon, and I came to the
> conclusion that it is not appropriate as a starting point: it does
> too little (not mirroring everything that PEP 381 tells it to), and
> it does too much (also grabbing packages/distributions/parcels
> from other places - i.e. not from PyPI), to support offline operation
> (which is not the objective of PEP 381).

And:

On Jan 11, 2010, at 4:22 PM, Tarek Ziadé wrote:

> If we have time at Pycon, and if some folks are interested, I would
> like to finish the implementation of PEP 381,

On Jan 12, 2010, at 5:22 PM, Tarek Ziadé wrote:

>> If z3c.pypimirror is not the starting point then is there a different implementation that will be used as a 'jumpstart'?  The mirroring protocol is "XXX Need to describe the protocol here." and z3c.pypimirror is mentioned in the only other sentence within that section.
> 
> The "XXX Need to describe the protocol here." I've added is just because I didn't take the time yet to write down the implicit protocol that was defined some years ago and that z3c.pypimirror uses. The objective here is to describe the most efficient way to scan and look for things tomirror (via the changelog) etc.

So...it seems that Tarek thinks the z3c.pypimirror implements the eventual protocol and Martin is just planning on writing "it" from scratch himself.

If this is to be a group effort, perhaps this would be a good time to whack up a repository with whatever specification (beyond what's in the PEP) or starting-point code is available and I'll do my little project within that framework.  Or not.

Tarek?  Martin?

S

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20100112/7905385e/attachment.htm>