Mailman 3 September 2008 - Python-Dev

Python performance through times
by Antoine Pitrou 05 Sep '08

05 Sep '08

Here's an interesting blog post comparing Python performance of various versions from 2.2.3 upto the latest 3.0 beta. http://www.ogre.com/node/147 The fact that only Mandelbrot calculation (a hardly representative benchmark for a high-level dynamic language such as Python) has become significantly slower is rather a good sign IMO. cheers Antoine.

2 1

Bug in SimpleHTTPRequestHandler.send_head?
by Kim Gräsman 05 Sep '08

05 Sep '08

Hi all, I'm new to this group and the Python language as such. I stumbled on it when I joined a project to build a rich network library for C++, which in turn uses Python and its CGI HTTP server implementation as part of its unit test suite. We're having a little trouble when serving a text file containing Windows line endings (CRLF) -- the resulting content contains Unix line endings only (LF). This breaks our tests, because we can't verify that the body, as parsed by our HTTP client, is the same as the source file we're serving through the Python HTTP server. I've isolated it to the SimpleHTTPRequestHandler.send_head method in SimpleHTTPServer.py: -- ctype = self.guess_type(path) if ctype.startswith('text/'): mode = 'r' else: mode = 'rb' try: f = open(path, mode) except IOError: self.send_error(404, "File not found") return None -- The f object is returned from this method, and used with shutil.copyfileobj to copy the contents to the output stream. This is easily fixed by omitting the content-type check entirely, and blindly using mode 'rb', and I think that makes sense, because the server should not be concerned with the contents of the body, so treating it as a binary stream seems right. This also fixes another issue, where the actual body size differs from what's specified in the Content-Length header, because CR characters are stripped when the body is served, but Content-Length contains the source file's binary size. I'm not sure which source control system you're using, so I won't try to provide a patch, but I believe the code should read: -- if os.path.isdir(path): if not self.path.endswith('/'): # redirect browser - doing basically what apache does self.send_response(301) self.send_header("Location", self.path + "/") self.end_headers() return None for index in "index.html", "index.htm": index = os.path.join(path, index) if os.path.exists(index): path = index break else: return self.list_directory(path) #patch: removed content-type check try: f = open(path, 'rb') #patch: always open in binary mode except IOError: self.send_error(404, "File not found") return None self.send_response(200) self.send_header("Content-type", self.guess_type(path)) #patch: content-type check here instead fs = os.fstat(f.fileno()) -- My changes marked with "#patch[...]". Grateful for any comments! Best wishes, - Kim

3 2

Can/should built-in functions get __get__?
by Terry Reedy 05 Sep '08

05 Sep '08

One of the nice features of 3.0 is that differences between classes defined in C and Python (other than speed) are mostly erased or hidden from the view of a Python programmer. However, there are still sometimes surprising and quite visible differences between 'functions' written in C and Python. Can these be better unified also? In particular, built-in functions, in spite of of being labeled 'builtin_function_or_method', are not usable as methods because they lack the __get__ method needed to bind function to instance. [Q. Any reason to not shorten that to 'built-in function'?] So, as a c.l.p poster discovered, within a class statement, __hash__ = lambda x: id(x) # works, while the apparently cleaner __hash__ = id # raises TypeError: id() takes exactly one argument (0 given) [Irony: trivial lambda wrapping, lambda x: f(x) has been described here as useless and diseased, but is seems that casting 'built-in function' to 'function' is not so completely useless. ;-] In this case, __hash__ = object.__hash__ is available, as is object.__repr__, but this is not true in general for C functions. The difference between a C function and a C method wrapper is tantalizingly small: >>> i = set(dir(id)) >>> h = set(dir(object.__hash__)) >>> i-h {'__module__', '__self__'} >>> h-i {'__objclass__', '__get__'} Similarly, for >>> def f(): pass >>> ff = set(dir(f)) >>> ff - i {'__defaults__', '__annotations__', '__kwdefaults__', '__globals__', '__closure__', '__dict__', '__code__', '__get__'} the only un-obvious difference is __get__, So I cannot help but wonder: is this is essential? or could it be removed but has not yet? Could the object wrapper for C functions get a __get__ method so they become methods when accessed via a class just like Python functions do? Terry Jan Reedy

3 3

Re: [Python-Dev] 'warnings' module changes still breaking Twisted, still looking for a fix
by glyph＠divmod.com 04 Sep '08

04 Sep '08

On 10:18 pm, musiccomposition(a)gmail.com wrote: >That's why catch_warning keeps track of the warnings filter too, so >you can call warnings.simplefilter("always") within the context >manager and the filter state will be restored. Thanks for the pointer - this is interesting. I misunderstood the way the warnings filter worked. It looks like this combination of features may in fact be all we need. It would be nice if application code could otherwise "officially" keep track of the warnings filter (my understanding is that 'filters', despite the public-looking name, is private, since it is absent from the documentation?) but that doesn't sound like a release blocker.

2 1

'warnings' module changes still breaking Twisted, still looking for a fix
by glyph＠divmod.com 04 Sep '08

04 Sep '08

With the 2.6 final release impending, the Twisted community buildbot is still red, <http://bit.ly/zFymN>, but there only seems to be one real issue: the warn_explicit change. This seems like it could be a pretty minor bit of maintenance to clear up on our end, if Python provided the appropriate hook. The current solution we have in mind for this problem is to build our own warnings module <http://bit.ly/4F3Qr2>, but that is obviously inelegant. While we do want to experiment with our own features for deprecation, users of Twisted will (rightfully) expect our "assertWarns" to continue covering warnings produced by the Python 2.6 warnings system. The proposed solution to this problem seems to have been "warnings.catch_warnings()" <http://bugs.python.org/issue3602>, but it is insufficient for testing, since subsequent calls to "catch_warnings()" will not catch warnings suppressed by duplication suppression; without even getting into multiple tests testing the same code path, this breaks features like our 'trial' tool's "--until- failure" option, which runs a fixed group of tests repeatedly in an attempt to flush out buggy tests which have race conditions that cause them to fail intermittently. We thought we'd filed a ticket for this before, but searching around in the issue tracker hasn't come up with anything. JP Calderone has filed a new ticket for this problem: <http://bugs.python.org/issue3780> . Any chance this could be made into a release blocker? If this one issue were fixed, Python 2.6 would be the smallest disruption to Twisted since Python 2.0 :-).

3 3

patch for Cookie.py to add support for HttpOnly
by Matt Chisholm 04 Sep '08

04 Sep '08

Eighteen months ago, Arvin Schnell contributed a really straightforward three-line patch to Cookie.py adding support for the HttpOnly flag on cookies: http://bugs.python.org/issue1638033 In the last eighteen months, HttpOnly has become a de-facto extension to the cookie standard. It is now supported by IE 7, Firefox 3, and Opera 9.5 (and there's a bug open against WebKit to support it): http://www.owasp.org/index.php/HTTPOnly Ruby, Perl, and PHP all support creating HttpOnly cookies now too. This article explains why HttpOnly is a good way to make cross-site scripting (XSS) attacks significantly more difficult: http://www.codinghorror.com/blog/archives/001167.htmllop Unfortunately this patch appears to have been ignored for the last year. The last thing I want is a delay in the release of 2.6/3.0, but Antoine Pitrou posted on the bug that it will have to wait for Python 2.7/3.1, because it is a feature request. If I'm not mistaken, that means no support for HttpOnly until sometime in 2010. Do we really have to wait two more years to apply a three-line patch which will bring Python in line with the industry state of the art and improve security for Python web applications? Is there a way that this could go in to 2.6.1/3.0.1? -matt

2 1

3rd-party dbms
by Oleg Broytmann 04 Sep '08

04 Sep '08

Brett Cannon <brett(a)python.org> wrote: > for 3.1 there is nothing saying we can't change shelve and the > dbm package to allow 3rd-party code to register with the dbm package > such that bsddb can be used as needed behind the scenes. Many years ago I wrote toy hashes based on ZODB and MetaKit. Registering them with anydbm was easy: import anydbm anydbm._names.insert(len(anydbm._names)-1, ['ZODBhash', 'MKhash']) # Insert before dumbdbm More complex part was to make whichdb to recognize those hashes. I just monkey-patched whichdb. If I were doing this now I'd do something similar to atexit module - every hash module will register its own test function, and whichdb will call them in turn until it finds which db it is. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd(a)phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

2 1

Re: [Python-Dev] Durus memory
by Binger David 04 Sep '08

04 Sep '08

I'm sorry this post is a bit off-topic, but I think I should correct this. On Sep 4, 2008, at 11:54 AM, Oleg wrote: > Durus (and ZODB) has an index of all objects, the index is stored in > memory AFAIK - a real problem if one has millions of objects. Durus now has an option to store the index in the data file in a form that is usable directly from disk. With this option, the in-memory index is only for objects changed since the last pack. Durus works fine with millions of objects, as long as you don't have millions of rapidly-changing objects. I haven't used the Berkeley DB storage for Durus, but I'm pretty sure that the Durus/Python side of that would not use any in-memory index.

1 0

Re: [Python-Dev] C API for gc.enable() and gc.disable()
by Andrey Zhmoginov 04 Sep '08

04 Sep '08

> > Would anyone mind if I did add a public C API for gc.disable() and > gc.enable()? I would like to use it as an optimization for the pickle > module (I found out that I get a good 2x speedup just by disabling the > GC while loading large pickles). Of course, I could simply import the > gc module and call the functions there, but that seems overkill to me. > I included the patch below for review. > I don't know if the following question is relevant, but it seems that many people here are familiar with Python cyclic garbage collector. I see Python [v2.5.2 (r252:60911, Jul 31 2008, 17:28:52)] crashing with Segment fault when I extend Python with a very simple module. This behavior is observed when I create a thousand of lists (it does not crash with 10-100) in the module with the garbage collector turned on. When I turn it off - everything is perfect. I suspect that it is my module, but if it is a Python bug ( _GC_Malloc? memory leak somewhere?), it may be worth reporting. The gdb "where" reply is the following: #0 0x080d8de9 in PyErr_Occurred () #1 0x080f508f in _PyObject_GC_Malloc () #2 0x080f5155 in _PyObject_GC_New () #3 0x08079c98 in PyList_New () #4 0xb7f53519 in draw_d_simple () from ./rt/rt_core.so #5 0xb7cf7833 in ffi_call_SYSV () from /usr/lib/python2.5/lib-dynload/_ctypes.so #6 0xb7cf766a in ffi_call () from /usr/lib/python2.5/lib-dynload/_ctypes.so #7 0xb7cf2534 in _CallProc () from /usr/lib/python2.5/lib-dynload/_ctypes.so #8 0xb7cec02a in ?? () from /usr/lib/python2.5/lib-dynload/_ctypes.so #9 0x0805cb97 in PyObject_Call () #10 0x080c7aa7 in PyEval_EvalFrameEx () #11 0x080c96e5 in PyEval_EvalFrameEx () #12 0x080cb1f7 in PyEval_EvalCodeEx () #13 0x080cb347 in PyEval_EvalCode () #14 0x080ea818 in PyRun_FileExFlags () #15 0x080eaab9 in PyRun_SimpleFileExFlags () #16 0x08059335 in Py_Main () #17 0x080587f2 in main () The crashing python code is: from ctypes import * import gc core = CDLL( "./tst.so" ) core.a.argtypes = [] core.a.restype = py_object #gc.disable() core.a() #gc.enable() And tst.cpp is: #include <Python.h> extern "C" { PyObject *a(); } PyObject *a() { int n = 1000; PyObject *item; for ( int i = 0; i < n; i++ ) item = PyList_New( 0 ); // Crashes here (somewhere in between). return item; } tst.cpp is compiled with: g++ -shared -Wl,-soname,tst.tmp.so -o tst.so tst.o g++ -I /usr/include/python2.5 -Wall -fPIC -o tst.o -c tst.cpp Thanks for your help! - Andrew.

2 1

Python 3.0b3 documentation typo
by Reed O'Brien 04 Sep '08

04 Sep '08

I was reading through the Dictionary views section: http://docs.python.org/dev/3.0/library/stdtypes.html#dictionary-view-objects Unless I am mistaken; the intersection at the end of the example usage should be: >>> keys & {'eggs', 'bacon', 'salad'} {'bacon'} Cheers, ~ro

2 1