Python hits the spot

[Bengt Richter]

On Sun, 23 Jun 2002 11:05:02 +0200, Siegfried Gonzi <siegfried.gonzi at kfunigraz.ac.at> wrote:

>Bengt Richter wrote:
>
>> Unless you are doing a lot of i/o from the program, sounds like maybe that
>> 13% (100-87) may be waiting for swapping to get more virtual space? Or it could be
>> coincidental background stuff? (Do you trigger any other processes at some point?)
>
>To cut it here: I tested the Mie code again and it works! The images 
>what I get are exactly the same as the one in a specific peer reviewed 
>paper. After that calculation I wait 20 minutes in order to watch
        ^^^^^
Not sure what you mean: Do you mean that the program completes, and then
you wait a further 20 minutes after that, expecting a possible "freeze?"
Is that something you have observed before? I.e., "freezing" _after_
successfully completing calculations, with something else working before
the "freeze"? BTW, what will Suse with your configuration do if you press
Alt-F2 etc? Will it give you a console login screen? Can you boot without
X and keep two logged-in sessions, with maybe one as root to see if you
can Alt-Fn to that when things "freeze" running your app from the other?
BTW2: you're not running your app as root, right? Does your app need X
for display or could you run it from a plain terminal console? The latter
would have a lot less baggage when trying to switch between sessions.

BTW. I heard there was one problematic version of SuSe in the past. You don't
happen to have that version I hope. Would someone set that FUD straight?
  
>whether the system will freeze or not: it freezes not. THE calculation 
>does not use any swap! I have got 256MB RAM but the calculation consumes 
>about 20 MB on Linux; there is no need for swapping (side note: my swap 
>is 756MB).
Hm, ok. 20MB and not growing. I missed that point. BTW, are you getting
your 87% number from top? If you toggle with "i" (no-idle) what else is left?
Anything besides top?

I wonder how/whether the percentage calculated is affected by power management
that slows down the CPU clock to keep it cool? Wonder if that could be happening.
I've never looked into power management stuff. I guess there is a temperature
where it just shuts down too, to protect itself?
>
>In turn I tried the following combinations:
>
>1. The complete calculations without the Mie code (note: the Mie code 
>works) and dummy values instead.
>Result: The first 10 minutes the system consumes 99.8% of the processor 
>(memory consumption is up to 5%) but after 10 minutes it sucks down to 
>87% and after 15 minutes the system freezes (I mean there is only the 
>power off option then!).
How is your system configured to respond to Ctl-Alt-Del? Does it respond
to that? If so, it's not _absolutely_ "frozen." Likewise, if you run without
a GUI with plain virtual consoles, you can often switch to another even if
if a current one appears "frozen." If you are on a local network and can
log into another machine, you could also try to telnet in to the "frozen"
machine, depending on how it's configured. You could enable telnet just for
a local test. Plain old Ctrl-C or Ctrl-Break don't do anything when it's
"frozen," right?

>
>2. I did the same without the Mie and Fortran code. Same situation as in 
>point 1.
>
>3. I did the same without Fortran/C/Satellite binary reading -> same 
>situation as in point 1 or 2.
Well, seems like it must be something about your Python code?

>
>4. I turned off every power management for my laptop, but that won't 
>cure the situation.
I don't know how that works, but I would think a system shouldn't shut
itself down if it has work to do, unless it's an emergency like running
out of battery power or overheating. So I don't understand this shutdown.
>
>
>Summarize: The system frezzes after 15 minutes and sucks down to 87% 
                       ^^^^^^^                  ^^^
In that order? How can you get a display of "87%" on a "frozen" system??

>system performance, without any notice why (I cannot see why under "top" 
>; sometimes I notice that many "kdeinits" appear right after the python 
>process)? Exactly the same situation appears on Windows: system goes 
>down from 99% to 87% after some time, but on Windows I do not get a 
>frozen system.
I guess I need a clearer definition of "frozen" ;-)
>
>I then wrote a small Python script in order to test the swap. The Python 
>script does constantly claim memory and my swap memory consumption goes 
>up to 500MB (system perfromance is only some 5%). After 30 minutes the 
>calculation is completed and I can work on without any problems.
>
Ok. Sounds normal.
>
>That means, if I have got a calculation of the form: processor 
>consumption 99% and memory consumption 5% the system fails; if I have
What exactly do you see or try that makes you conclude the system has "failed?"
 
>got a calculation in the reverse order: 90% memory and 5% processor the 
>system remains stable.
Does your program accumulate a lot of state to keep referring to as it
progresses? E.g., if information were saved in lists that grew and had
to be traversed a lot, you'd expect one kind of behavior. If you were
simulating some population of things with behavior that was either easy
or hard to calculate, and the progress of simulation forced changes to
a more complex mode of calculation (e.g., adaptive adjustments to the
resolution of a grid etc.), then you could expect increasingly bogged
down behavior, which might be proper. But if there's a bug so that
new iterations decide to shift gears using leftover parameters, you could
get degraded performance (e.g., because of unneeded high resolution calculations),
yet finally come out with valid numbers (and a false impression that Python sucked ;-)

I have no idea what your app is doing, so I am just speculating, in the hope
that something useful will trigger a helpful thought.
>
>
>Maybe this all can be explained that the ActivePython version has a bug 
>or something like that. Or that the combination Linux/Python/laptop is 
>not optimal. But I am not willing to install the normal Python version 
>again (I did it until I noticed that idle was not installed; I  invested 
>2 days in order to install idle but was not successfull; ActiveState was 
>my isle in the following).
Why not run from a plain text mode virtual terminal?

>
>The offer remains valid: if someone is interested in my source code he 
>can get it.
Ok.
>
>I am not any longer willing to try the calculation on my Linux box and 
>Python 2.2 (20 Linux power off  shut downs are enough in a day; should I 
Do you know why it's shutting down? What are the manufacturer's criteria
for shutting down? Or can some user software be calling for shutdown for
some reason? Is a screen saver involved? It must be maddening ;-)

>destroy my laptop?). In the afternoon I will decide whether I will wait 
>6 hours for the calculation on Windows or not. And I will also decide
Are there parameters you can input to the problem to control how accurately
you want things to be calculated? Can you make a low accuracy/resolution/whatever
run in a short time? It might let you do debug experiments more quickly.
Or maybe there's a bug in how that is controlled? Especially if it is adaptive.
Maybe config parameters are getting modified so all 8 (IIRC) major loops
don't see the same values?

Have you any profiling information at all to know where all the time is being
spent, and how the proportions change as things bog down?

>whether it will be a good idea to rewrite it in Fortran 90. Python can 
>then be used in combination with DISLIN for the plots.
>
Still no real diagnosis, though ;-/

Regards,
Bengt Richter