python slowdown after a longish time of running (garbage collector)

[Michal Vitecek]

 okay, thank you. it'll be pretty hard to describe it so much in detail,
 but here it goes:

    - the server waits on named socket and accepts clients' requests via
      via stream where each command is a cPickled tuple (command,
      commandData,)
 
    - the server is used to generate certain machine states => it uses
      a state generator and returns the generated state back to clients
      in format of cPickled tuple (status, machineState,)

    - the state generator uses sets of rules (defined as tuples of class
      instances) which tell it how to generate a machine state

    - the result is a tuple of class instances

 this is the core of how it works.
 
 the problem is IMHO in the state generator that initializes lots of
 objects (both builtin (list, tuples, dictionaries) and user made) and
 relies on the garbage collector to clean them up when they are unused.

 the memory usage is not increasing (even after a day of running), but
 the server (state generator) generates the states more and more slowly.
 
 i tested the generator for generating the same state (=> the executed
 code is still the same) and from the same initial state, so i am almost
 sure the problem is not in the algorithm. however i measured that the
 number of generated states right after the start is ~70/sec whereas
 after a day or so it goes down to ~35/sec.

 i suspect the problem must be somewhere in the garbage collector - i
 don't know if it uses only reference counting, but if i recall
 correctly garbage collector was a source of speed problems in java as
 well and it now uses two algorithms to collect the garbage (mark&sweep
 and copying).
    
holger krekel wrote:
>Michal Vitecek wrote:
>>  hello all,
>> 
>>  i plan to use python for servers that should have pretty good
>>  throughput. however, after my test server runs for a longer time
>>  (constantly processing commands issued by clients) my measurements show
>>  that the speed of commands that it processes slowly decreases to a
>>  crawl. i suspect the problem is with the garbage collector.
>> 
>>  is it normal behaviour of python garbage collector? can i help it by
>>  deleting local variables in methods? is it better if i disable the
>>  garbage collector and run it only after a longish time?
>
>You have do give *a lot* more detail for a meaningful answer. E.g.
>
>What kind of server application is running?
>
>How is the design, what are the involved network protocols?
>
>how long do you run your applications? Is memory usage increasing?
>
>etc.pp.
>
>The more specific your questions are the more likely you get
>a good answer.
>
>regards,
>
>    holger

-- 
		fuf		(fuf at mageo.cz)