[Catalog-sig] pypi mirrors in bad shape

[Aron Xu]

On Wed, Jan 16, 2013 at 5:43 PM, Jannis Leidel <jannis at leidel.info> wrote:
>
> On 15.01.2013, at 14:42, ken cochrane <kencochrane at gmail.com> wrote:
>
>> According to http://www.pypi-mirrors.org the pypi mirrors are in bad shape right now. We only have one mirror (f.pypi.python.org) that isn't more then 8 hours out of date.
>>
>> We have one server e.pypi.python.org I can't even connect to anymore, and I'm not sure if is even online. Anyone know why?
>>
>> c.pypi.python.org and d.pypi.python.org are both over 1 day old
>>
>> g.pypi.python.org is over 11 days old
>>
>> b.pypi.python.org is about 8 hours old at this point, I looked at the logs on this server and it looks like it is getting stuck trying to download packages for QSTK. Hopefully it resolves itself.
>>
>> Restarting download of source/Q/QSTK/QSTK-0.2.1.tar.gz
>> Restarting download of source/Q/QSTK/QSTK-0.2.0.tar.gz
>>
>>
>> Even pypi.crate.io is over 4 hours out of date, and that one is usually pretty reliable.
>>
>> I'm guessing there was something that was recently added to pypi, that broke the mirror replication, and each of these servers needs to be looked at to make sure they can get back in sync.
>>
>> If I don't see any improvements in these mirrors by the end of the day, I'll start emailing the owners individually.
>
> Sigh.
>
> The mirror script was hanging on d, too. Since I've had to fix the mirror every other month now I wonder if the pep381client is up to the task. Does anyone have recommendations for fixing the situation? Is there an easy way to monitor the client and kill it if it runs longer than x?
>
> I've now started a manual sync and hope to see d being up-to-date soonish.
>

Why not just use something based on rsync? I see no problem syncing
Debian mirrors for years, which have lots of small files and update
frequently. The script of Debian ftpsync cannot be used directly for
pypi, but it's an example that worth considering:

http://ftp.debian.org/debian/project/ftpsync/ftpsync-current.tar.gz



-- 
Regards,
Aron Xu