[Catalog-sig] PyPi down again

Richard Jones richardjones at optushome.com.au
Thu Nov 16 23:11:35 CET 2006

On Friday 17 November 2006 09:04, Jim Fulton wrote:
> Richard Jones wrote:
> > On Friday 17 November 2006 08:44, Jim Fulton wrote:
> >> Any idea why now?
> >
> > Just logged in:
> >
> >  23:00:48 up 30 days,  7:59,  2 users,  load average: 34.41, 38.20, 39.90
> >
> > There's a flood of apache processes, so I assume something's hitting the
> > web server hard.
> >
> > I don't have access to the logs, so I can't say anythin beyond that.
> Thanks for looking.
> Is there someone who can help? Can I help somehow?

We need someone with root access to look at the logs.

I'm posting to pydotorg to see if we can get someone to look.

Without seeing the logs I'm going to guess we've got a badly-behaved web 
spider hitting the browse interface.

> What can we do to protect ourselves from this in the future?

Assuming my guess about it being a spider hitting the browser interface, we 
already attempt to block spiders off from the browse interface both through 
the robots.txt file and by detecting their user-agent in the pypi code. 
Spiders don't *have* to read robots.txt, and my RE string for user-agent 
detection might be missing someone.

We won't know until we can get someone to look at the logs.


More information about the Catalog-sig mailing list