[Catalog-sig] Cheeseshop performance improved

Jodok Batlogg jodok at lovelysystems.com
Tue Jun 26 14:10:17 CEST 2007


On 26.06.2007, at 12:52, Sean Reifschneider wrote:

> On Mon, Jun 25, 2007 at 11:44:07PM +0200, Laura Creighton wrote:
>> Jodok Batlogg of Lovely Systems has already proposed donating a  
>> machine
>
> The problem is that nobody has the 8-ish hours of time required to  
> migrate
> the services to a new machine.
>
> The quick fix would be to engage XS4ALL to upgrade the RAM in that  
> box,
> leaving the box otherwise untouched.  The system has only 1GB of  
> RAM in it.
> It's got a 2.8GHz Xeon CPU in it, so I would expect it can take at  
> least
> 4GB of RAM, if not 8 or 16GB.
>
> Thomas: If the PSF threw a grand or two at XS4ALL, could we get the  
> memory
> in ximinez upgraded?  Preferably to 4 or 8GB of RAM?

very good idea. memory is cheap. that should solve the issues without  
a lot of work.

regarding the cheeseshop-software:
it seems hard to find qualified persons that can improve the pypi  
software (for money).
me and parts of my team will be at europython and have some time for  
sprinting on code improvements (baking static files,...) probably we  
can organize a cheeseshop sprint there?

jodok


>> From what I've seen, the problem really boils down to 50 instances of
> Apache at 20MB each consumes all RAM.  20MB each for Apache  
> processes is
> not unusual.  Our web-site, which is fairly light-weight apache 
> +mod_python
> uses around 20MB per Apache process resident, for example.
>
> The Apache processes consuming memory cause other problems.  As  
> memory is
> consumed, disc caching is reduced to nearly nothing, causing more  
> data to
> have to be pulled from disc, causing fewer requests able to be  
> processed
> per second.  For example, right now I'm looking at the system and it's
> spending half it's time waiting for the disc.  This is after a  
> restart of
> Apache.
>
> I think it's a little premature to call 20 to 40MB Apache processes
> "leaking".  As I said, our web site is running 20MB Apache  
> processes and
> it's not doing that much.  One of our clients runs a fairly high  
> traffic
> web site that is fairly dynamic, using mod_perl, and their Apache  
> processes
> stabilize around 30MB each.  Their problem is that they want to  
> handle a
> concurrency of 200 or more, which adds up to 6GB right there, not  
> counting
> the database.
>
> Once system load starts going up above 1, it is probably just going to
> result in a death spiral because memory accesses are now taking
> milliseconds instead of tens of nanoseconds.
>
> I've taken the fairly drastic measure of pushing down the requests  
> that
> Apache will handle before restarting from 1,000 to 10.  Whether the  
> problem
> is that something is leaking memory in the Apache processes, or  
> that we
> just don't have enough RAM for the Apache processes to stabilize at
> whatever their natural limit is once they have everything they need  
> loaded,
> this should help.
>
> We still have 60 concurrent clients allowed, so we're less  
> vulnerable to
> running out of hooks due to slow clients or spiders with high  
> concurrency.
>
> So far, things are looking good.  I don't have much real-world time  
> with it
> running this way, but I'm optimistic that this will help prevent  
> the Apache
> restart process from being required much if any.
>
> However, I'd really like to see this box get 4 to 8GB of RAM.  This is
> absolutely a place where the PSF can throw money at a problem to  
> make it go
> away.
>
> The donation of another machine to host pypi is very generous, but  
> it's not
> something that we really have the manpower to be able to take  
> advantage of
> it seems.
>
> Anyway, the current config seems to be working well based on the  
> baseline I
> took and some other changes I played around with.  By now the baseline
> would have been consuming 20MB per Apache instance.
>
> Thomas: Can you contact someone at XS4ALL about getting us a quote for
> upgrading the RAM in ximinez to 4 or 8GB?  This will require the least
> amount of time from the community.
>
> Sorry I haven't been able to look at this in more depth sooner.   
> Busy busy.
>
> Sean
> -- 
>  "I was on IRC once and got mistaken for Dan Bernstein. I still have
>  nightmares."  -- Donnie Barnes
> Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
> tummy.com, ltd. - Linux Consulting since 1995: Ask me about High  
> Availability
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig

--
"Flat is better than nested."
   -- The Zen of Python, by Tim Peters

Jodok Batlogg, Lovely Systems
Schmelzhütterstraße 26a, 6850 Dornbirn, Austria
phone: +43 5572 908060, fax: +43 5572 908060-77


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2454 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20070626/ece2a40a/attachment.bin 


More information about the Catalog-SIG mailing list