I'll take a look at this. We might need to increase the number of allowed failures on the health checks.

On May 14, 2014, at 1:51 PM, Wichert Akkerman wrote:

...

On 14 May 2014, at 19:49, Paul Moore wrote:

...

On 14 May 2014 18:37, Wichert Akkerman wrote: PyPI is down again for me (from .nl): when I go to https://pypi.python.org/pypi I am greeted with a "You've reached the static mirror of https://pypi.python.org” message. http://status.python.org/ does not show any problems though.

Works for me, FWIW.

It’s working again for me now as well. PyPI does seem to be a bit flaky recently though; I had the exact same outage message earlier today as well, and a varnish error earlier this week.

Wichert. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Can you perhaps reproduce this and check the headers? Look for a X-Served-By header. I want to make sure that we don't have a cache with a bad config loaded. I'm not entirely great at sorting out graphite, but it appears to be like we don't have any elevated error rates except for one spike at ~9:15. I'm not sure what timezone that is in, probably UTC? On May 19, 2014, at 3:39 AM, Wichert Akkerman wrote:

On 14 May 2014, at 20:51, Donald Stufft wrote:

...

I'll take a look at this. We might need to increase the number of allowed failures on the health checks.

FWIW I just got another varnish error (on https://pypi.python.org/pypi/FormAlchemy/1.4.3 ). A page reload worked correctly.

Is there a monitoring system in place that detects these errors? I’m seeing them reasonably regularly, but I don’t know how useful it is to report them every time.

Wichert.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On May 19, 2014, at 7:22 AM, Wichert Akkerman wrote:

On 19 May 2014, at 13:20, Donald Stufft wrote:

...

Can you perhaps reproduce this and check the headers? Look for a X-Served-By header. I want to make sure that we don't have a cache with a bad config loaded.

I can’t reproduce it - it was a single failing request as far as I could see. I’ll look for that header the next time I run into this.

...

I'm not entirely great at sorting out graphite, but it appears to be like we don't have any elevated error rates except for one spike at ~9:15. I'm not sure what timezone that is in, probably UTC?

carbon just stores unix timestamps. graphite-web can be configured with a timezone; it defaults to America/Chicago, which is apparently a default it gets from Django.

Wichert.

Here’s the data from Fastly’s Stats Panel: http://cl.ly/image/1N3a3s3Y3J0I Yellow == 2xx responses, Red = 5xx responses. That’s the last 8 hours and appears to be in Eastern (my timezone). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA