Mailman 3 December 2005 - Python-Dev

suggestion for smarter garbage collection in function of size (gc.set_collect_mem_growth(2))
by Andrea Arcangeli 29 Dec '05

29 Dec '05

Hello, I run into a problem recently with a reconnectingclientfactory with twisted while write some spare time software, that turned out to be a gc inefficiency. In short the protocol memory wasn't released after the reconnect and the protocol had about 50M attached to it. So with frequent reconnects the rss of the task grown to >1G and it pushed the system into heavy swap. In reality only 50M were allocated, so 950M were wasted by python. A gc.collect() explicit invocation at every retry of the factory fixed it. However this means there is significant room for improvement in the default behaviour of the python gc. In short the real bug is that the python gc isn't working in function of size in any way. It doesn't matter the number of objects, it matters their size! The whole story can be found in this thread: http://twistedmatrix.com/pipermail/twisted-python/2005-December/012233.html My suggestion to fix this problem in autopilot mode (without requiring explicit gc.collect()) is to invoke a gc.collect() (or anyway to go deep down freeing everything possible) at least once every time the amount of anonymous memory allocated by the interpreter doubles. The tunable should be a float >= 1. When the tunable is 1 the feature is disabled (so it works like current python today). Default should be 2 (which means to invoke gc.collect() after a 100% increase of the anonymous memory allocated by the interpreter). We could also have yet another threshold that sets a minimum of ram after which this heuristic in function of size kicks in, but it's not very important and it may not be worth it (whem little memory is involved gc.collect() should be fast anyway). To implement this we need two hooks, in the malloc and free that allocate python objects. Then we have to store the minimum of this value (i.e. the last minimum of memory allocated by the interpreter). The algorithm I'd suggest is like this (supposedly readable pseudocode): gc.set_collect_mem_growth(v): assert float(v) >= 1 gc.collect_mem_growth = v python_malloc(size): ram_size += size if ram_size > min_ram_size * gc.collect_mem_growth: gc.collect() # python_free runs inside it min_ram_size = ram_size # ram size after gc.collect() python_free(size): ram_size -= size min_ram_size = min(min_ram_size, size) The overhead of this should be zero, and it'll fix my testcase just fine. I believe the default should be 2 (equivalent to 100% growth of rss to trigger a full collect) even though it alters the behaviour of the gc, I think it's a bug that so much memory can be leaked when it could be reclaimed istantly. I wouldn't change other parameters, this heuristic in function of size would be completely orthogonal and disconnected by the current heuristics in function of the number of elements. It has taken me a day and precious help from the twisted folks to realize it wasn't a memleak in my twisted spare time application (but well, it was good since I learnt about the fact I created an heisenbug by using __del__ to debug the apparent memleak ;). Thanks.

6 9

Re: [Python-Dev] suggestion for smarter garbage collection in function of size (gc.set_collect_mem_growth(2))
by Andrea Arcangeli 28 Dec '05

28 Dec '05

On Wed, Dec 28, 2005 at 07:14:32PM -0700, Neil Schemenauer wrote: > [This message has also been posted.] > Martin v. Löwis <martin(a)v.loewis.de> wrote: > > One challenge is that PyObject_GC_Del doesn't know how large the memory > > block is that is being released. So it is difficult to find out how > > much memory is being released in the collection. > > Another idea would be to add accounting to the PyMem_* interfaces. > It could be that most memory is used by objects that are not tracked > by the GC (e.g. strings). I guess you still have the same problem > in that PyMem_Free may not know how large the memory block is. In ram_size (per my pseudocode) we have to account everything that can be possibly released by the "gc" by an inovcation of a deep gc.collect(). So if strings can't be freed by the gc (as a side effect of releasing other objects), then we don't necessairly need to account for them in the algorithm, otherwise we have to. I guess some strings can be freed by the gc too so it sounds like PyMem_ may be a more correct hooking point. We definitely must know the size of the free operation. The sad thing is that glibc knows it. size_t free_size(void * ptr) /* free and return size of freed object */ An API like the above would be able to answer our question at very little cost, but it requires changing glibc, and we'd need to make sure it's really the more efficient way of doing it before considering it, because I've some doubt at the moment (otherwise I wonder why something like the above doesn't already exist in glibc?!?). OTOH I guess not many apps are doing their own garbage collection, and the ones that do it, may be using their own allocators instead of the glibc ones. This reminds me about the pymalloc thing I heard about over time. That should be able to provide a pymalloc_free_size kind of thing returning the size of the object freed, we could start with that assuming it's more efficient than doing the accounting in the upper layer. PS. your mail client looks broken the way it handles CC ;)

1 0

set.copy documentation string
by Noam Raphael 28 Dec '05

28 Dec '05

is currently "Return a shallow copy of a set." Perhaps "shallow" should be removed, since set members are supposed to be immutable so there's no point in a deep copy? Noam

2 4

Automated Python testing (was Re: status of development documentation)
by Tim Peters 28 Dec '05

28 Dec '05

Take a look at: http://buildbot.zope.org/ That runs code from: http://buildbot.sourceforge.net/ Someone sets up a "buildbot master" (that's what the Zope URL points at), and then any number of people can volunteer to set up their boxes as "buildbot slaves". From time to time the buildbot master asks the slaves to do the checkout/build/test dance (or any other code dance you like), the slaves report results back to the master, and the master displays the slaves' results. If you look at the 2nd-leftmost column, you can see that the master knows when checkins have been done. Checkins can trigger asking the slaves to run tests, and if the tests fail on some slave the master can send email saying so, including the list of checkins ("the blamelist") done since the last time that slave ran tests: The guilty developer can be identified and harassed without human intervention. :-) This really helps at Zope Corp. One downside is that we seem unable to get an in-house Windows buildbot slave to work reliably, and so far don't even know whether that's because of Windows, the buildbot code, or flakiness in our internal network. It seems quite reliable on Linux, though.

7 11

Small any/all enhancement
by Valentino Volonghi aka Dialtone 28 Dec '05

28 Dec '05

I see that Python 2.5 will include a simple implementation of any/all. What I would like to ask, before it's too late, is if it's possible to add an optional test argument. any(iterable, test=bool) and all(iterable, test=bool) These would be the new proposed APIs. The usecase is to be able to extract attributes from objects or simply to have arbitrary checks like: import operator any(some_objects, test=operator.attrgetter('some_attribute')) or def zerop(x): return x==0 all(some_objects, zerop) instead of preprocessing the generator with a generator expression before passing it to any/all. Thanks. -- Valentino Volonghi aka Dialtone Now Running MacOSX 10.4 Blog: http://vvolonghi.blogspot.com http://weever.berlios.de

7 10

A few questions about setobject
by Noam Raphael 28 Dec '05

28 Dec '05

Hello, I'm going over setobject.c/setobject.h, while trying to change them to support cheap frozen-copying. I have a few questions. 1) This is a part of setobject.h: typedef struct { long hash; PyObject *key; } setentry; typedef struct _setobject PySetObject; struct _setobject { ... setentry *table; setentry *(*lookup)(PySetObject *so, PyObject *key, long hash); setentry smalltable[PySet_MINSIZE]; ... }; It seems to me that setentry and _setobject are defined for every file that includes Python.h. In the Python C API, in the section about include files, it is written that: "All user visible names defined by Python.h (except those defined by the included standard headers) have one of the prefixes "Py" or "_Py". Names beginning with "_Py" are for internal use by the Python implementation and should not be used by extension writers. Structure member names do not have a reserved prefix." Is this desirable? Even if it is, it seems that the second sentence contradicts the first sentence. Perhaps the header file should stick with writing "struct { long hash; PyObject *key; }" three times (or define it in a macro and then undefine it), and the typedef be left to the .c file? 2) The hash table used by sets uses a dummy element for deleted entries. The implementation goes into the trouble of allocating it, managing its reference count, and deallocating it at the end. What is the reason for that? It seems to me that the only requirement of the dummy element is that it shouldn't be a pointer to a valid PyObject, and as such I would think that defining it like int dummy_int; PyObject *dummy = (PyObject *)(&dummy_int); would be enough, and that it shouldn't be INCREFed or DECREFed every time it is used. I think it should be ok because it's never used really as a PyObject. Am I missing something? (Ok, I now thought that maybe it's because some parts don't treat dummy elements specially. But it seems to me that most parts do treat them specially, so perhaps it would be better to make a few changes so that all parts will treat them specially?) 3) The type of the result of a binary operator applied on a set and a frozenset is the type of the left set. You are welcomed to ignore this, but I just wanted to say that it seems to me better to make the operator symmetric, and to return a frozenset only if both sets are frozen. If you think that these questions belong to c.l.py, then please say so and I will go away. Have a good day, Noam

3 5

Re: [Python-Dev] deque alternative (was: Linked lists)
by Phillip J. Eby 28 Dec '05

28 Dec '05

At 09:10 PM 12/25/2005 -0500, Martin Blais wrote: >I still haven't had time to cook a proper reply to Guido, but one >thing I see is that many ppl on the list seem to think that because >there aren't many use cases (that they can see), therefore there isn't >much use for a list collection type. I would instead argue that >because the list type has never been available, people have gotten >used not to use them, and therefore settle with arrays and do not see >a need for them. Since I routinely use 2-item tuples (twoples?) as cons cells when I actually need a linked list, I think I'm in a good position to say that this isn't true. Certainly it's not true for me. Since Python effectively has single-character operations for both consing and car/cdr (a ',' on the right or left sides of an assignment statement respectively), as well as trivial truth testing for () as a "nil", suggests to me that anybody who wants a lispy list can easily have one. For any application where such a structure excels (like shared sublists/trees and recursive traversals), it's so darn easy to use them in Python that I don't think anything special is needed. Honestly, ISTM that the One Obvious Way to do lispy lists in Python is to just use two-tuples, with no special library functions. Sequence packing and unpacking is extremely fast in Python, too, probably faster than any method calls to a more heavyweight builtin type could be. Thus, I'm -1 on creating another cons type. Why do this: foo = cons(bar, None) car_foo = foo.car cdr_foo = foo.cdr when you can just do this: foo = bar, () car_foo, cdr_foo = foo How much simpler can you get?

4 3

Keep default comparisons - or add a second set?
by Jim Jewett 28 Dec '05

28 Dec '05

Today, x < y will return a consistent result, even if the two objects are not comparable in any meaningful manner.[*] The current plan for Python 3 is to instead raise TypeError. If this really needs to be changed (and I don't think it does), then there should still be a standard way to get a canonical ordering. Jim Fulton asked how this canonical ordering should be defined. The obvious answer is "Just like the __lt__ operator today". But it doesn't actually *need* to be that sensible, if there are strong reasons to prefer a simpler ordering. The more I think about it, though, __eq__ really does need to be honored; I don't want an "Are these two containers equivalent?" test to depend on how well the memory subsystem happened to be working. Given that, also honoring an explicit __lt__ isn't much of an extra burden, and will make the ordering much more useful for debugging and output. The only real question left is what to do when the two objects' classes do not define a comparison between them. Current CPython eventually falls back to the objects' locations in memory. Unfortunately, this isn't stable across different sessions, and can mask some wrong-type bugs. There is no general answer to the lack of stability; if objects from multiple sessions might be compared (Jim Fulton's BTree example), then whatever communication channel mixes the objects will have to provide its own wrapper for fallback comparisons. Taking away the default fallback will (at best) change where bugs are caught, and it won't always be earlier. There is some value in saying "unlike objects should not be compared by accident", so I would understand a decision to create two types of sorting. "Comparison" would raise a TypeError if the pair of objects were not comparable; "Ordering" would continue on to "something consistent", just like sorts do today. The catch is that both operations are needed, and the obvious syntax (<) should go with the operation whose semantics are not backwards- compatible. Given that, I don't think the separation of operations would clarify code in practice. [*] Paul Moore pointed out that comparing Numeric arrays to other objects does raise a TypeError. But this was an explicit choice, and the Numeric authors had to jump through enough hoops to get this behavior that it isn't likely to be a common problem. -jJ

5 7

PyCon TX 2006: Early-bird registration ends Dec. 31!
by David Goodger 28 Dec '05

28 Dec '05

Early bird registration for PyCon TX 2006 ends on December 31st, so there are only a few days LEFT. To register, please visit: http://us.pycon.org/TX2006/Registration You can still register after Dec. 31st, but the cost will go up by US$65 (US$25 for students). This year PyCon will feature a day of tutorials before the three days of regular presentations. Course outlines for all the tutorials have been posted; see http://wiki.python.org/moin/PyCon2006/Tutorials All of the PyCon tutorials are still open for new registrations, but space is limited, and we suspect they'll all be filled up by the time early-bird registration closes. Don't forget to book your hotel room, too. PyCon TX 2006 is being held at a Dallas/Addison hotel, and we have negotiated a special low rate: http://us.pycon.org/Addison/Hotels We hope to see you in Texas! -- David Goodger (on behalf of A.M. Kuchling, Chair, PyCon 2006)

1 0

status of development documentation
by Fredrik Lundh 28 Dec '05

28 Dec '05

the "Documentation Development" page at http://www.python.org/dev/doc/ contains a link to a "SVN trunk" version which was last updated nearly four months ago. what would it take to automatically update the trunk docs, say, once a day or so ? or is it time to move away from the current "odd-format-with-extensive- semantic-markup-that-nobody-uses-and-only-a-few-people-fully-under- stand" approach to something quicker and dirtier [1] ? </F> 1) http://microformats.org/wiki/microformats

24 88