[Chicago] Out of Memory: Killed Process: on CentOS

[Brian Ray]

On Apr 27, 2009, at 11:43 AM, Garrett Smith wrote:

>
>
>> The server has several Gig's of memory, but the requests it tries to
>> process are really huge.  It run's fine if the requests are small.
>
> What do you mean by big/small? Size of HTTP request bodies?
>

That is correct, the size of the HTTP request bodies.


>> What is the best ways to manage this scenario?  Specifically, what
>> should I try?
>
> I suggest first confirming that it's memory. Get the server into the
> non responsive state and use 'top' to survey the problematic  
> processes.
> By default, top will list the top-CPU consumers first. Use SHIFT + >
> (greater than sign) to change the sort order to list the top memory
> consumers.
>

I see from the "var/log/messages" it is killing httpd when it can.


> top also shows uptime stats on the top line. Big numbers probably mean
> lots of processes running at full tilt. This would cause server slow
> down.
>
> For memory, you're interested in RSS (resident) - that's the memory  
> the
> process is actually using (as opposed to what's reserved for it).
>
> You can install sysstat and run pidstat to track a particular process,
> but I suspect top is all you'll need.

I wonder if I can run these from the process itself to help  monitor  
the situation.


>
>
>> Should I be looking at my apache configuration?  I tried setting
>> StartServers, MinSpareServers, MaxSpareServers, and MaxClients to low
>> numbers, like 3.  Overall it seems to help but, I am not sure it will
>> if I get a really huge request.
>
> I'd recommend running your Python app as a standalone, multithreaded
> server (e.g. use CherryPy). This is going to eliminate a huge number
> of variables and help you isolate problems more quickly.
>
> Take some time and get into the details of apache + mod_python before
> running apps in production. I'd offer some specific suggestions, but
> it's been a while since I've used that mod -- and it's got a lot of
> configuration options.
>
> You might also look into modwsgi -- this has really matured over the
> last couple years. If I use apache, I generally use that mod for  
> Python
> apps.


I am ok with running differently; although, generally Apache is  
working well for us on situations where we have sites that get a lot  
of traffic. I would want to use two things at once, perhaps or simply  
run a separate process of python for each request.

>
>
>> Is there some other way to clear memory when running large requests?
>
> Verify that you understand what's causing your server to become
> unresponsive before getting into application level tweaks. If it is
> indeed memory, first check for leaks (in Python, best case is global
> state including static/class variables, caches that aren't being
> attended to, etc.) You'll see this right away if you run as a  
> standalone
> app -- you're process will keep gobbling memory (use top to watch) and
> the server will eventually start swapping.
>

What methods work best for finding leaks. Of course the app consumed  
more and more memory while it is parsing.  Are there any tools out  
there you would recommend to check for leaks.

> If it's simply that cElementTree is keeping huge structures in memory
> and you're getting too many concurrent requests for your RAM, move to
> a SAX style API (state machine).


Do you have any good SAX style API you would recommend?


Thanks,   Brian Ray