Mailman 3 January 2009 - Python-Dev

mlockall() in Python?
by Evgeni Golov 29 Jan '09

29 Jan '09

Hi, I'd like to write a small daemon in Python, which should never be swapped out (on Linux, this daemon will be Linux specific, so no need in a platform-independent solution). In C I'd do: #include <sys/mman.h> mlockall(MCL_FUTURE); //do stuff here munlockall(); Is there anything similar in Python? TIA & kind regards Evgeni Golov

3 2

Re: [Python-Dev] [PSF-Board] I've got a surprise for you!
by Steve Holden 28 Jan '09

28 Jan '09

Jim Walker wrote: [Trent's announcement] > > Great stuff Trent! I was wondering how you were doing. > > I really appreciate what it takes to put these open resources > together ;) There's a lot of moving parts :) > > Cheers, > Jim > > BTW. > > We now have zone servers in the OpenSolaris test farm, and > I plan to add guest os servers in the next few weeks using > ldoms (sparc) and xvm (x64). The zone servers provide whole > root zones, which should be a good development environment > for most projects. Check it out: > > http://test.opensolaris.org/testfarm > http://www.opensolaris.org/os/community/testing/testfarm/zones/ > > Let me know if there is interest from the python community to > manage one of the test farm servers for python development. > Besides the general use machines, the php community is already > managing a T2000 server. Jim: Thanks, this is a terrific offer. I am copying it to the Python developers list so they can discuss it - I know that Solaris is one of the platforms we do get quite a few build questions about. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Chairman, PSF http://www.holdenweb.com/

2 1

undesireable unpickle behavior, proposed fix
by Jake McGuire 27 Jan '09

27 Jan '09

Instance attribute names are normally interned - this is done in PyObject_SetAttr (among other places). Unpickling (in pickle and cPickle) directly updates __dict__ on the instance object. This bypasses the interning so you end up with many copies of the strings representing your attribute names, which wastes a lot of space, both in RAM and in pickles of sequences of objects created from pickles. Note that the native python memcached client uses pickle to serialize objects. >>> import pickle >>> class C(object): ... def __init__(self, x): ... self.long_attribute_name = x ... >>> len(pickle.dumps([pickle.loads(pickle.dumps(C(None), pickle.HIGHEST_PROTOCOL)) for i in range(100)], pickle.HIGHEST_PROTOCOL)) 3658 >>> len(pickle.dumps([C(None) for i in range(100)], pickle.HIGHEST_PROTOCOL)) 1441 >>> Interning the strings on unpickling makes the pickles smaller, and at least for cPickle actually makes unpickling sequences of many objects slightly faster. I have included proposed patches to cPickle.c and pickle.py, and would appreciate any feedback. dhcp-172-31-170-32:~ mcguire$ diff -u Downloads/Python-2.4.3/Modules/ cPickle.c cPickle.c --- Downloads/Python-2.4.3/Modules/cPickle.c 2004-07-26 22:22:33.000000000 -0700 +++ cPickle.c 2009-01-26 23:30:31.000000000 -0800 @@ -4258,6 +4258,8 @@ PyObject *state, *inst, *slotstate; PyObject *__setstate__; PyObject *d_key, *d_value; + PyObject *name; + char * key_str; int i; int res = -1; @@ -4319,8 +4321,24 @@ i = 0; while (PyDict_Next(state, &i, &d_key, &d_value)) { - if (PyObject_SetItem(dict, d_key, d_value) < 0) - goto finally; + /* normally the keys for instance attributes are + interned. we should try to do that here. */ + if (PyString_CheckExact(d_key)) { + key_str = PyString_AsString(d_key); + name = PyString_FromString(key_str); + if (! name) + goto finally; + + PyString_InternInPlace(&name); + if (PyObject_SetItem(dict, name, d_value) < 0) { + Py_DECREF(name); + goto finally; + } + Py_DECREF(name); + } else { + if (PyObject_SetItem(dict, d_key, d_value) < 0) + goto finally; + } } Py_DECREF(dict); } dhcp-172-31-170-32:~ mcguire$ diff -u Downloads/Python-2.4.3/Lib/ pickle.py pickle.py --- Downloads/Python-2.4.3/Lib/pickle.py 2009-01-27 01:41:43.000000000 -0800 +++ pickle.py 2009-01-27 01:41:31.000000000 -0800 @@ -1241,7 +1241,15 @@ state, slotstate = state if state: try: - inst.__dict__.update(state) + d = inst.__dict__ + try: + for k,v in state.items(): + d[intern(k)] = v + # keys in state don't have to be strings + # don't blow up, but don't go out of our way + except TypeError: + d.update(state) + except RuntimeError: # XXX In restricted execution, the instance's __dict__ # is not accessible. Use the old way of unpickling

5 11

PyCon 2009 registration is now open!
by David Goodger 27 Jan '09

27 Jan '09

Register here: http://us.pycon.org/2009/register/ Information (rates etc.): http://us.pycon.org/2009/registration/ Hotel information & reservations: http://us.pycon.org/2009/about/hotel/ Early bird registration ends February 21, so don't delay! -- David Goodger, PyCon 2009 Chair

1 0

V8, TraceMonkey, SquirrelFish and Python
by Mart Sõmermaa 27 Jan '09

27 Jan '09

As most of you know there's constant struggle on the JavaScript front to get even faster performance out of interpreters. V8, TraceMonkey and SquirrelFish have brought novel ideas to interpreter design, wouldn't it make sense to reap the best bits and bring them to Python? Has anyone delved into the designs and considered their applicability to Python? Hoping-to-see-some-V8-and-Python-teams-collaboration-in-Mountain-View-ly yours, Mart Sõmermaa

3 3

FormatError() in callproc.c under win32
by Ulrich Eckhardt 27 Jan '09

27 Jan '09

Hi! In callproc.c from trunk is a function called SetException(), which calls FormatError() only to discard the contents. Can anyone enlighten me to the reasons thereof? Is it just to check if the errorcode is registered in the stringtables? The reason I ask is the CE port. The FormatMessage API exists there (or, rather, the FormatMessageW API), but the tables containing the error messages are optional for the OS and for space reasons many vendors chose not to include them. That means that the function there regularly fails to retrieve the requested string. My first approach was to fall back to simply providing a sting with a numeric representation of the errorcode, but that would change the meaning of above function, because then it could never fails. My second approach was to enhance PyErr_SetFromWindowsErr() to handle the additional error codes that are checked in SetException(). However, those require more context than just the error code, they use the EXCEPTION_RECORD passed to SetException() for that. My third approach would be to filter out the special error codes first and delegate all others to PyErr_SetFromWindowsErr(). The latter already handles the lack of a string for the code by formatting it numerically. This would also improve consistency, since the two functions use different ways to format unrecognised errors numerically. This approach would change where and how a completely unrecognised error code is formatted, but would otherwise be pretty equivalent. Suggestions? Uli -- Sator Laser GmbH Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at <http://www.satorlaser.de/> ************************************************************************************** Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht verantwortlich. **************************************************************************************

4 6

Sun resources [Was: I've got a surprise for you!]
by Steve Holden 27 Jan '09

27 Jan '09

So, if anyone wants to run a Sun buildbot or whatever, Jim would be the person to contact. Synchronize on this list to ensure Jim doesn't get multiple approaches, please. regards Steve -------- Original Message -------- Subject: Re: [PSF-Board] I've got a surprise for you! Date: Tue, 27 Jan 2009 10:49:29 -0700 From: Jim Walker <James.Walker(a)Sun.COM> Reply-To: James.Walker(a)Sun.COM Organization: Sun Microsystems, Inc. To: Steve Holden <steve(a)holdenweb.com> References: <20090126233246.GA37662(a)wind.teleri.net> <497E9320.2030404(a)sun.com> <497EF94E.3050302(a)holdenweb.com> Steve Holden wrote: > > Thanks, this is a terrific offer. I am copying it to the Python > developers list so they can discuss it - I know that Solaris is one of > the platforms we do get quite a few build questions about. > Sounds good. Depending on what you want to do, I can assign a system to your group within a week or two. Cheers, Jim -- Jim Walker, http://blogs.sun.com/jwalker Sun Microsystems, Software, Solaris QE x77744, 500 Eldorado Blvd, Broomfield CO 80021 -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/

1 0

enabling a configure option
by Antoine Pitrou 26 Jan '09

26 Jan '09

Hello python-dev, r68924 in py3k introduced a new configure option named --with-computed-gotos. It would be nice if one of the buildbots could exercise this option, so that the code doesn't rot (the buildbot has to use gcc). Whom should I ask for this? Speaking of which, there are only five buildbots remaining in the "stable" bunch... What has happened to the others? Regards Antoine.

3 4

Re: [Python-Dev] subprocess crossplatformness and async communication
by Hrvoje Niksic 26 Jan '09

26 Jan '09

Nick Craig-Wood wrote: > But for the conversational case (eg using it to do a remote login) it > doesn't work at all :- > > run child > send stuff to stdin > child reads stdin and writes stdout Can this really be made safe without an explicit flow control protocol, such as a pseudo-TTY? stdio reads data from pipes such as stdin in 4K or so chunks. I can easily imagine the child blocking while it waits for its stdin buffer to fill, while the parent in turn blocks waiting for the child's output arrive. Shell pipelines (and the subprocess module as it stands) don't have this problem because they're unidirectional: you read input from one process and write output to another, but you typically don't feed data back to the process you've read it from.

5 6

IDLE docs at python.org/idle
by Georg Brandl 26 Jan '09

26 Jan '09

(re #5066) Is that documentation maintained in some way? Shouldn't it be merged into the main docs? Georg

3 2