python slowdown after a longish time of running (garbage collector)
fuf at mageo.cz
Fri Aug 30 14:27:05 CEST 2002
okay, thank you. it'll be pretty hard to describe it so much in detail,
but here it goes:
- the server waits on named socket and accepts clients' requests via
via stream where each command is a cPickled tuple (command,
- the server is used to generate certain machine states => it uses
a state generator and returns the generated state back to clients
in format of cPickled tuple (status, machineState,)
- the state generator uses sets of rules (defined as tuples of class
instances) which tell it how to generate a machine state
- the result is a tuple of class instances
this is the core of how it works.
the problem is IMHO in the state generator that initializes lots of
objects (both builtin (list, tuples, dictionaries) and user made) and
relies on the garbage collector to clean them up when they are unused.
the memory usage is not increasing (even after a day of running), but
the server (state generator) generates the states more and more slowly.
i tested the generator for generating the same state (=> the executed
code is still the same) and from the same initial state, so i am almost
sure the problem is not in the algorithm. however i measured that the
number of generated states right after the start is ~70/sec whereas
after a day or so it goes down to ~35/sec.
i suspect the problem must be somewhere in the garbage collector - i
don't know if it uses only reference counting, but if i recall
correctly garbage collector was a source of speed problems in java as
well and it now uses two algorithms to collect the garbage (mark&sweep
holger krekel wrote:
>Michal Vitecek wrote:
>> hello all,
>> i plan to use python for servers that should have pretty good
>> throughput. however, after my test server runs for a longer time
>> (constantly processing commands issued by clients) my measurements show
>> that the speed of commands that it processes slowly decreases to a
>> crawl. i suspect the problem is with the garbage collector.
>> is it normal behaviour of python garbage collector? can i help it by
>> deleting local variables in methods? is it better if i disable the
>> garbage collector and run it only after a longish time?
>You have do give *a lot* more detail for a meaningful answer. E.g.
>What kind of server application is running?
>How is the design, what are the involved network protocols?
>how long do you run your applications? Is memory usage increasing?
>The more specific your questions are the more likely you get
>a good answer.
fuf (fuf at mageo.cz)
More information about the Python-list