[Catalog-sig] pypi mirrors in bad shape

Christian Theune ct at gocept.com
Wed Jan 23 22:52:32 CET 2013

On 2013-01-16 15:41:00 +0000, Donald Stufft said:

> On Wednesday, January 16, 2013 at 4:43 AM, Jannis Leidel wrote:
> The mirror script was hanging on d, too. Since I've had to fix the 
> mirror every other month now I wonder if the pep381client is up to the 
> task. Does anyone have recommendations for fixing the situation? Is 
> there an easy way to monitor the client and kill it if it runs longer 
> than x?
> Should be able to use the timeout command.
> timeout -k HARD_LIMIT SOFT_LIMIT /path/pep381client/pep381run -q /var/pypi

I used to run this with "timeout 5m pep381run" but it seems that the 
client really didn't like getting a SIGTERM.

Looking at the code there seems to be something special going on 
regarding handling KeyboardInterrupt so I switched to using "timeout -2 
5m …".

Still, I saw a weird problem and I'm suspicious that the client really 
doesn't always manage to write a consistent status file when being 
For some reason I'm reinitializing my whole mirror after only a day.

I think I'm happy running Tarek's branch on the f mirror for now, but 
I'm not super sure about that …

My proposal would be to sit down at PyCon to dig a bit more into the 
code to make it more robust. My feeling is that the current client 
tries way to hard on a very low-level API (httplib) for a lot of 
mechanics. I'd be happy to refactor and provide tests, I think.

Also, a script to determine internal consistency and consistency 
compared to PyPI would be nice. Then again, the rsync idea might not be 
that far off regarding  the amount of work and the problems we're 
dealing with …


More information about the Catalog-SIG mailing list