[issue12775] immense performance problems related to the garbage collector

[Daniel Svensson]

New submission from Daniel Svensson <dsvensson at gmail.com>:

I've noticed problems with the GC in two applications. In one case the application did not respond to SOAP-requests for 9 seconds, every couple of minutes, when it really shouldn't have taken more than 20ms. In another case we had one operation executed by 20 threads that ought to have taken 0.5 seconds, but instead took about 90 seconds.

Both cases were solved by disabling the garbage collector and hunting down possible circular references. Once this was done, the first example went down to its expected 20ms max latency, and the second one to its 0.5s processing time.


Here is a short python script that demonstrates the issue, the JSON file in this case is 1.2GB large:

> import cjson, time, gc

> def read_json_blob():
>   t0 = time.time()
>   fd = file("mytestfile")
>   data = fd.read()
>   fd.close()
>   t1 = time.time()
>   parsed = cjson.decode(data)
>   t2 = time.time()
>   print "read file in %.2fs, parsed json in %.2fs, total of %.2fs" % \
>                                                   (t1-t0, t2-t1, t2-t0)

> read_json_blob()
read file in 10.57s, parsed json in 531.10s, total of 541.67s

> gc.disable()
> read_json_blob()
read file in 0.59s, parsed json in 15.13s, total of 15.72s

> gc.collect()
0

I don't understand how Python can work like this default, at least not without a warning, to me it looks like a joke gone too far. All documentation ought to recommend people to disable the garbage collector at the first sign of performance problems, or the garbage collector problem should be fixed, this the "Documentation" or "Interpreter Core" Components in the ticket classification.

----------
assignee: docs at python
components: Documentation, Interpreter Core
messages: 142305
nosy: docs at python, dsvensson
priority: normal
severity: normal
status: open
title: immense performance problems related to the garbage collector
type: performance
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12775>
_______________________________________