[Catalog-sig] pypi mirrors in bad shape

[Christian Theune]

On 2013-01-16 15:41:00 +0000, Donald Stufft said:

> On Wednesday, January 16, 2013 at 4:43 AM, Jannis Leidel wrote:
> The mirror script was hanging on d, too. Since I've had to fix the 
> mirror every other month now I wonder if the pep381client is up to the 
> task. Does anyone have recommendations for fixing the situation? Is 
> there an easy way to monitor the client and kill it if it runs longer 
> than x?
> Should be able to use the timeout command.
> 
> timeout -k HARD_LIMIT SOFT_LIMIT /path/pep381client/pep381run -q /var/pypi

I used to run this with "timeout 5m pep381run" but it seems that the 
client really didn't like getting a SIGTERM.

Looking at the code there seems to be something special going on 
regarding handling KeyboardInterrupt so I switched to using "timeout -2 
5m …".

Still, I saw a weird problem and I'm suspicious that the client really 
doesn't always manage to write a consistent status file when being 
terminated.
For some reason I'm reinitializing my whole mirror after only a day.

I think I'm happy running Tarek's branch on the f mirror for now, but 
I'm not super sure about that …

My proposal would be to sit down at PyCon to dig a bit more into the 
code to make it more robust. My feeling is that the current client 
tries way to hard on a very low-level API (httplib) for a lot of 
mechanics. I'd be happy to refactor and provide tests, I think.

Also, a script to determine internal consistency and consistency 
compared to PyPI would be nice. Then again, the rsync idea might not be 
that far off regarding  the amount of work and the problems we're 
dealing with …

Christian