[Catalog-sig] Cheeseshop performance improved

[Sean Reifschneider]

On Mon, Jun 25, 2007 at 11:44:07PM +0200, Laura Creighton wrote:
>Jodok Batlogg of Lovely Systems has already proposed donating a machine

The problem is that nobody has the 8-ish hours of time required to migrate
the services to a new machine.

The quick fix would be to engage XS4ALL to upgrade the RAM in that box,
leaving the box otherwise untouched.  The system has only 1GB of RAM in it.
It's got a 2.8GHz Xeon CPU in it, so I would expect it can take at least
4GB of RAM, if not 8 or 16GB.

Thomas: If the PSF threw a grand or two at XS4ALL, could we get the memory
in ximinez upgraded?  Preferably to 4 or 8GB of RAM?

>From what I've seen, the problem really boils down to 50 instances of
Apache at 20MB each consumes all RAM.  20MB each for Apache processes is
not unusual.  Our web-site, which is fairly light-weight apache+mod_python
uses around 20MB per Apache process resident, for example.

The Apache processes consuming memory cause other problems.  As memory is
consumed, disc caching is reduced to nearly nothing, causing more data to
have to be pulled from disc, causing fewer requests able to be processed
per second.  For example, right now I'm looking at the system and it's
spending half it's time waiting for the disc.  This is after a restart of
Apache.

I think it's a little premature to call 20 to 40MB Apache processes
"leaking".  As I said, our web site is running 20MB Apache processes and
it's not doing that much.  One of our clients runs a fairly high traffic
web site that is fairly dynamic, using mod_perl, and their Apache processes
stabilize around 30MB each.  Their problem is that they want to handle a
concurrency of 200 or more, which adds up to 6GB right there, not counting
the database.

Once system load starts going up above 1, it is probably just going to
result in a death spiral because memory accesses are now taking
milliseconds instead of tens of nanoseconds.

I've taken the fairly drastic measure of pushing down the requests that
Apache will handle before restarting from 1,000 to 10.  Whether the problem
is that something is leaking memory in the Apache processes, or that we
just don't have enough RAM for the Apache processes to stabilize at
whatever their natural limit is once they have everything they need loaded,
this should help.

We still have 60 concurrent clients allowed, so we're less vulnerable to
running out of hooks due to slow clients or spiders with high concurrency.

So far, things are looking good.  I don't have much real-world time with it
running this way, but I'm optimistic that this will help prevent the Apache
restart process from being required much if any.

However, I'd really like to see this box get 4 to 8GB of RAM.  This is
absolutely a place where the PSF can throw money at a problem to make it go
away.

The donation of another machine to host pypi is very generous, but it's not
something that we really have the manpower to be able to take advantage of
it seems.

Anyway, the current config seems to be working well based on the baseline I
took and some other changes I played around with.  By now the baseline
would have been consuming 20MB per Apache instance.

Thomas: Can you contact someone at XS4ALL about getting us a quote for
upgrading the RAM in ximinez to 4 or 8GB?  This will require the least
amount of time from the community.

Sorry I haven't been able to look at this in more depth sooner.  Busy busy.

Sean
-- 
 "I was on IRC once and got mistaken for Dan Bernstein. I still have
 nightmares."  -- Donnie Barnes
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability