[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
mal at egenix.com
Mon Dec 22 13:20:59 CET 2008
On 2008-12-20 23:16, Martin v. Löwis wrote:
>>> I will try next week to see if I can come up with a smaller,
>>> submittable example. Thanks.
>> These long exit times are usually caused by the garbage collection
>> of objects. This can be a very time consuming task.
> I doubt that. The long exit times are usually caused by a bad
> malloc implementation.
With "garbage collection" I meant the process of Py_DECREF'ing the
objects in large containers or deeply nested structures, not the GC
mechanism for breaking circular references in Python.
This will usually also involve free() calls, so the malloc
implementation affects this as well. However, I've seen such long
exit times on Linux and Windows, which both have rather good
I don't think there's anything much we can do about it at the
interpreter level. Deleting millions of objects takes time and that's
not really surprising at all. It takes even longer if you have
instances with .__del__() methods written in Python.
Applications can choose other mechanisms for speeding up the
exit process in various (less clean) ways, if they have a need for
BTW: Rather than using a huge in-memory dict, I'd suggest to either
use an on-disk dictionary such as the ones found in mxBeeBase or
Professional Python Services directly from the Source (#1, Dec 22 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2008-12-02: Released mxODBC.Connect 1.0.0 http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
More information about the Python-Dev