python slowdown after a longish time of running (garbage collector)
pyth at devel.trillke.net
Fri Aug 30 14:55:52 CEST 2002
Michal Vitecek wrote:
> okay, thank you. it'll be pretty hard to describe it so much in detail,
> but here it goes:
> - the server waits on named socket and accepts clients' requests via
> via stream where each command is a cPickled tuple (command,
> - the server is used to generate certain machine states => it uses
> a state generator and returns the generated state back to clients
> in format of cPickled tuple (status, machineState,)
> - the state generator uses sets of rules (defined as tuples of class
> instances) which tell it how to generate a machine state
> - the result is a tuple of class instances
> this is the core of how it works.
sounds reasonable. Are you using threads, asyncore or some hand made
> the problem is IMHO in the state generator that initializes lots of
> objects (both builtin (list, tuples, dictionaries) and user made) and
> relies on the garbage collector to clean them up when they are unused.
You might try to disable garbage collection to see if the problem persists.
> the memory usage is not increasing (even after a day of running), but
> the server (state generator) generates the states more and more slowly.
I'd try to run your program through the profiler
and look at the results.
> i tested the generator for generating the same state (=> the executed
> code is still the same) and from the same initial state, so i am almost
> sure the problem is not in the algorithm. however i measured that the
> number of generated states right after the start is ~70/sec whereas
> after a day or so it goes down to ~35/sec.
sorry, I can't quite understand this. Can you reproduce this behaviour without
involving any socket-IO? That would help in analyzing a lot.
More information about the Python-list