[Catalog-sig] Mirror problem f.pypi.python.org

[Christian Theune]

Hi,

my mirror has a problem, once again.

Yesterday it started trying to sync lxml which seems to take longer 
than 25 minutes which is my timeout for a single sync.

Unfortunately, the resume is only on a per-package granularity and 
seems to restart with lxml all the time.

After the Nth re-try, the client failed to properly pause the sync with 
this error:

Feb 10 20:25:01 services02 pep381run: Mirroring paused. Resume by 
restarting pep381run.
Feb 10 20:25:01 services02 pep381run: Exception in thread Thread-1 
(most likely raised during interpreter shutdown):
Feb 10 20:25:01 services02 pep381run: Traceback (most recent call last):
Feb 10 20:25:01 services02 pep381run:   File 
"/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
Feb 10 20:25:01 services02 pep381run:   File 
"/opt/pep381client/dev/pep381client/scripts/../pep381client/__init__.py", 
line 75, in run
Feb 10 20:25:01 services02 pep381run:   File 
"/opt/pep381client/dev/pep381client/scripts/../pep381client/__init__.py", 
line 270, in _synchronize_project
Feb 10 20:25:01 services02 pep381run:   File 
"/opt/pep381client/dev/pep381client/scripts/../pep381client/__init__.py", 
line 390, in maybe_copy_file
Feb 10 20:25:01 services02 pep381run:   File 
"/usr/lib/python2.7/httplib.py", line 548, in read
Feb 10 20:25:01 services02 pep381run:   File 
"/usr/lib/python2.7/httplib.py", line 647, in _safe_read
Feb 10 20:25:01 services02 pep381run:   File 
"/usr/lib/python2.7/socket.py", line 387, in read
Feb 10 20:25:01 services02 pep381run: <type 'exceptions.TypeError'>: 
'NoneType' object is not callable

The client did continue to try syncing lxml for a couple more hours. 
The last time was 21:25 CET when it paused supposedly correctly.

At 21:30 the next try determined that the index was broken and seems to 
re-sync everything now:

Feb 10 21:30:11 services02 pep381run: File exists but seem broken, 
rebuilding it
Feb 10 21:30:11 services02 pep381run: Building a status - this may be long...
Feb 10 21:30:11 services02 pep381run: Starting the mirror sync.
Feb 10 21:30:11 services02 pep381run: Creating 10 workers
Feb 10 21:30:11 services02 pep381run: Synchronizing "Coversation With Your Car"
Feb 10 21:30:11 services02 pep381run: Synchronizing 0x10c-asm
Feb 10 21:30:11 services02 pep381run: Synchronizing 18-e
Feb 10 21:30:11 services02 pep381run: Synchronizing 1ee
Feb 10 21:30:11 services02 pep381run: Synchronizing 2C.py
Feb 10 21:30:11 services02 pep381run: Synchronizing 2gis
Feb 10 21:30:11 services02 pep381run: Synchronizing 2mp4
Feb 10 21:30:11 services02 pep381run: Synchronizing 3-1
Feb 10 21:30:11 services02 pep381run: Synchronizing 3to2

Which of course also takes ages due to the 25min limit and the coarse 
granularity for pausing.

Looking at the mirror page we're already back to 50% aging or old mirrors. :/

As an operator, this is really harder than promised from the packaging. 
I'm really looking forward to do something to make this more robust. 
Happy to help at PyCon.

Christian