python slowdown after a longish time of running (garbage collector)

[holger krekel]

Michal Vitecek wrote:
>  okay, thank you. it'll be pretty hard to describe it so much in detail,
>  but here it goes:
> 
>     - the server waits on named socket and accepts clients' requests via
>       via stream where each command is a cPickled tuple (command,
>       commandData,)
>  
>     - the server is used to generate certain machine states => it uses
>       a state generator and returns the generated state back to clients
>       in format of cPickled tuple (status, machineState,)
> 
>     - the state generator uses sets of rules (defined as tuples of class
>       instances) which tell it how to generate a machine state
> 
>     - the result is a tuple of class instances
> 
>  this is the core of how it works.

sounds reasonable. Are you using threads, asyncore or some hand made
loop?

>  the problem is IMHO in the state generator that initializes lots of
>  objects (both builtin (list, tuples, dictionaries) and user made) and
>  relies on the garbage collector to clean them up when they are unused.

You might try to disable garbage collection to see if the problem persists.

    import gc
    gc.disable()

>  the memory usage is not increasing (even after a day of running), but
>  the server (state generator) generates the states more and more slowly.

I'd try to run your program through the profiler

    import profile
    profile.run(codeobject/string/file)

and look at the results.

>  i tested the generator for generating the same state (=> the executed
>  code is still the same) and from the same initial state, so i am almost
>  sure the problem is not in the algorithm. however i measured that the
>  number of generated states right after the start is ~70/sec whereas
>  after a day or so it goes down to ~35/sec.

sorry, I can't quite understand this.  Can you reproduce this behaviour without
involving any socket-IO?  That would help in analyzing a lot.

regards,

    holger