Python-Dev September 2006

python-dev@python.org

100 participants
120 discussions

by Gustavo Carneiro

We have to resort to timeouts in pygtk in order to catch unix signals in threaded mode. The reason is this. We call gtk_main() (mainloop function) which blocks forever. Suppose there are threads in the program; then any thread can receive a signal (e.g. SIGINT). Python catches the signal, but doesn't do anything; it simply sets a flag in a global structure and calls Py_AddPendingCall(), and I guess it expects someone to call Py_MakePendingCalls(). However, the main thread is blocked calling a C function and has no way of being notified it needs to give control back to python to handle the signal. Hence, we use a 100ms timeout for polling. Unfortunately, timeouts needlessly consume CPU time and drain laptop batteries. According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. Another alternative would be to add a new API like Py_AddPendingCallNotification, which would let python notify extensions that new pending calls exist and need to be processed. But I would really prefer the first alternative, as it could be fixed within python 2.5; no need to wait for 2.6. Please, let's make Python ready for the enterprise! [2] [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 [2] http://perkypants.org/blog/2006/09/02/rfte-python/

13 years, 10 months

Python Doc problems

by xah lee

There are a lot reports on the lousy state of python docs. I'm not much in the python community so i don't know what the developers are doing anything about it. anyway, i've rewrote the Python's RE module documentation, at: http://xahlee.org/perl-python/python_re-write/lib/module-re.html and have recently made the term of user clear. may i ask what the python developers is doing about the python's docs? Are you guys aware, that there are rampant criticisms of python docs and many diverse tries by various individuals to rewrite the doc by starting another wiki or site? Xah xah(a)xahlee.org ∑ http://xahlee.org/ ☄

13 years, 10 months

PEP 351 - do while

by Hans Polak

Hi, Just an opinion, but many uses of the 'while true loop' are instances of a 'do loop'. I appreciate the language layout question, so I'll give you an alternative: do: <body> <setup code> while <condition> Cheers, Hans Polak. This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.

13 years, 10 months

Caching float(0.0)

by Nick Craig-Wood

I just discovered that in a program of mine it was wasting 7MB out of 200MB by storing multiple copies of 0.0. I found this a bit suprising since I'm used to small ints and strings being cached. I added the apparently nonsensical lines + if age == 0.0: + age = 0.0 # return a common object for the common case and got 7MB of memory back! Eg :- Python 2.5c1 (r25c1:51305, Aug 19 2006, 18:23:29) [GCC 4.1.2 20060814 (prerelease) (Debian 4.1.1-11)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> a=0.0 >>> print id(a), id(0.0) 134738828 134738844 >>> Is there any reason why float() shouldn't cache the value of 0.0 since it is by far and away the most common value? A full cache of floats probably doesn't make much sense though since there are so many 'more' of them than integers and defining small isn't obvious. -- Nick Craig-Wood <nick(a)craig-wood.com> -- http://www.craig-wood.com/nick

13 years, 10 months

Re: [Python-Dev] Caching float(0.0)

by Kristján V. Jónsson

Well, a lot of extension code, like ours use PyFloat_FromDouble(foo); This can be from vectors and stuff. Very often these are values from a database. Integral float values are very common in such case and id didn't occur to me that they weren't being reused, at least for small values. Also, a lot of arithmetic involving floats is expected to end in integers, like computing some index from a float value. Integers get promoted to floats when touched by them, as you know. Anyway, I now precreate integral values from -10 to 10 with great effect. The cost is minimal, the benefit great. Cheers, Kristján -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com(a)python.org [mailto:python-dev-bounces+kristjan=ccpgames.com@python.org] On Behalf Of "Martin v. Löwis" Sent: 30. september 2006 08:48 To: Bob Ippolito Cc: python-dev(a)python.org Subject: Re: [Python-Dev] Caching float(0.0) Bob Ippolito schrieb: > My guess is that people do have this problem, they just don't know > where that memory has gone. I know I don't count objects unless I have > a process that's leaking memory or it grows so big that I notice (by > swapping or chance). Right. Although I do wonder what kind of software people write to run into this problem. As Guido points out, the numbers must be the result from some computation, or created by an extension module by different means. If people have many *simultaneous* copies of 0.0, I would expect there is something else really wrong with the data structures or algorithms they use. Regards, Martin _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com

13 years, 10 months

Re: [Python-Dev] Caching float(0.0)

by Nick Maclaren

"Jason Orendorff" <jason.orendorff(a)gmail.com> wrote: > > Anyway, this kind of static analysis is probably more entertaining > than relevant. ... Well, yes. One can tell that by the piffling little counts being bandied about! More seriously, yes, it is Well Known that 0.0 is the Most Common Floating-Point Number is most numerical codes; a lot of older (and perhaps modern) sparse matrix algorithms use that to save space. In the software floating-point that I have started to draft some example code but have had to shelve (no, I haven't forgotten) the values I predefine are Invalid, Missing, True Zero and Approximate Zero. The infinities and infinitesimals (a.k.a. signed zeroes) could also be included, but are less common and more complicated. And so could common integers and fractions. It is generally NOT worth doing a cache lookup for genuinely numerical code, as the common cases that are not the above rarely account for enough of the numbers to be worth it. I did a fair amount of investigation looking for compressibility at one time, and that conclusion jumped out at me. The exact best choice depends entirely on what you are doing. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

13 years, 10 months

difficulty of implementing phase 2 of PEP 302 in Python source

by Brett Cannon

I am at the point with my security work that I need to consider how I am going to restrict importing modules. My current plan is to basically implement phase 2 of PEP 302 and control imports through what importer objects are provided. This work should lead to a meta_path importer for built-ins and then path_hooks importers for .py, .pyc, and extension modules. But it has been suggested here that the import machinery be rewritten in Python. Now I have never touched the import code since it has always had the reputation of being less than friendly to work with. I am asking for opinions from people who have worked with the import machinery before if it is so bad that it is worth trying to re-implement the import semantics in pure Python or if in the name of time to just work with the C code. Basically I will end up breaking up built-in, .py, .pyc, and extension modules into individual importers and then have a chaining class to act as a combined .pyc/.py combination importer (this will also make writing out to .pyc files an optional step of the .py import). Any opinions would be greatly appreciated on this. I need to get back to my supervisor by the end of the day Friday with a decision as to whether I think it is worth the rewrite. If you are interested in helping with the Python rewrite (or in general if the work is done with the C code), please let me know since if enough people want to help with the Python rewrite it might help wash out the extra time needed to make it work. -Brett

13 years, 10 months

Tix not included in 2.5 for Windows

by Christos Georgiou

Does anyone know why this happens? I can't find any information pointing to this being deliberate. I just upgraded to 2.5 on Windows (after making sure I can build extensions with the freeware VC++ Toolkit 2003) and some of my programs stopped operating. I saw in a French forum that someone else had the same problem, and what they did was to copy the relevant files from a 2.4.3 installation. I did the same, and it seems it works, with only a console message appearing as soon as a root window is created: attempt to provide package Tix 8.1 failed: package Tix 8.1.8.4 provided instead Cheers.

13 years, 10 months

Possible semantic changes for PEP 352 in 2.6

by Brett Cannon

I am working on PEP 352 stuff for 2.6 and there are two changes that I think should be made that are not explicitly laid out in the PEP. First, and most dramatic, involves what is legal to list in an 'except' clause. Right now you can listing *anything*. This means ``except 42`` is totally legal even though raising a number is not. Since I am deprecating catching string exceptions, I can go ahead and deprecate catching *any* object that is not a legitimate object to be raised. The second thing is changing PyErr_GivenExceptionMatches() to return 0 on false, 1 on true, and -1 on error. As of right now there is no defined error return value. While it could be suggested to check PyErr_Occurred() after every call, there is a way to have the return value reflect all possible so I think this changed should be made. Anybody have objections with any of the changes I am proposing? -Brett

13 years, 10 months

Re: [Python-Dev] Caching float(0.0)

by Kristján V. Jónsson

Well gentlemen, I did gather some stats on the frequency of PyFloat_FromDouble(). out of the 1000 first different floats allocated, we get this frequency distribution once our server has started up: - stats [1000]({v=0.00000000000000000 c=410612 },{v=1.0000000000000000 c=107838 },{v=0.75000000000000000 c=25487 },{v=5.0000000000000000 c=22557 },...) std::vector<entry,std::allocator<entry> > + [0] {v=0.00000000000000000 c=410612 } entry + [1] {v=1.0000000000000000 c=107838 } entry + [2] {v=0.75000000000000000 c=25487 } entry + [3] {v=5.0000000000000000 c=22557 } entry + [4] {v=10000.000000000000 c=18530 } entry + [5] {v=-1.0000000000000000 c=14950 } entry + [6] {v=2.0000000000000000 c=14460 } entry + [7] {v=1500.0000000000000 c=13470 } entry + [8] {v=100.00000000000000 c=11913 } entry + [9] {v=0.50000000000000000 c=11497 } entry + [10] {v=3.0000000000000000 c=9833 } entry + [11] {v=20.000000000000000 c=9019 } entry + [12] {v=0.90000000000000002 c=8954 } entry + [13] {v=10.000000000000000 c=8377 } entry + [14] {v=4.0000000000000000 c=7890 } entry + [15] {v=0.050000000000000003 c=7732 } entry + [16] {v=1000.0000000000000 c=7456 } entry + [17] {v=0.40000000000000002 c=7427 } entry + [18] {v=-100.00000000000000 c=7071 } entry + [19] {v=5000.0000000000000 c=6851 } entry + [20] {v=1000000.0000000000 c=6503 } entry + [21] {v=0.070000000000000007 c=6071 } entry (here I omit the rest). In addition, my shared 0.0 double has some 200000 references at this point. 0.0 is very, very common. The same can be said about all the integers up to 5.0 as well as -1.0 I think I will add a simple cache for these values for Eve. something like: int i = (int) fval; if ((double)i == fval && i>=-1 && i<6) { Py_INCREF(table[i]); return table[i]; } Cheers, Kristján > -----Original Message----- > From: python-dev-bounces+kristjan=ccpgames.com(a)python.org > [mailto:python-dev-bounces+kristjan=ccpgames.com@python.org] > On Behalf Of Kristján V. Jónsson > Sent: 29. september 2006 15:18 > To: Fredrik Lundh; python-dev(a)python.org > Subject: Re: [Python-Dev] Caching float(0.0) > > Acting on this excellent advice, I have patched in a reuse > for -1.0, 0.0 and 1.0 for EVE Online. We use vectors and > stuff a lot, and 0.0 is very, very common. I'll report on > the refcount of this for you shortly. > > K > > > -----Original Message----- > > From: python-dev-bounces+kristjan=ccpgames.com(a)python.org > > [mailto:python-dev-bounces+kristjan=ccpgames.com@python.org] > > On Behalf Of Fredrik Lundh > > Sent: 29. september 2006 15:11 > > To: python-dev(a)python.org > > Subject: Re: [Python-Dev] Caching float(0.0) > > > > Nick Craig-Wood wrote: > > > > > Is there any reason why float() shouldn't cache the value > > of 0.0 since > > > it is by far and away the most common value? > > > > says who ? > > > > (I just checked the program I'm working on, and my analysis > tells me > > that the most common floating point value in that program > is 121.216, > > which occurs 32 times. from what I can tell, 0.0 isn't > used at all.) > > > > </F> > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev(a)python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/kristjan%40c > cpgames.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/kristjan%40c cpgames.com >

13 years, 10 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev September 2006