Mailman 3 April 2008 - Python-Dev

Buildbots for trunk are all red
by Facundo Batista 15 May '08

15 May '08

All fail in test_compiler.py. -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

21 51

Problems with the new super()
by Armin Ronacher 12 May '08

12 May '08

Hi all, I blogged about that topic today which turned out to be a very bad idea, so I summarize it for the mailinglist here to hopefully start a discussion about the topic, which I think is rather important. In the last weeks something remarkable happened in the Python3 sources: self kinda became implicit. Not in function definitions, but in super calls. But not only self: also the class passed to `super`. That's interesting because it means that the language shifts into a completely different direction. `super` was rarely used in the past, mainly because it was weird to use. In the most common use case the current class and the current instance where passed to it, and the super typed returned looked up the parent methods on the MRO for you. It was useful for multiple inheritance and mixin classes that don't know their parent but confusing for many. I can see that a replacement is a good idea, but I'm not so sure if the current implementation is the way to go. The main problem with replacing `super(Foo, self).bar()` with something like `super.bar()` is obviously that self is explicit and the class (in that case Foo) can't be determined by the caller. Furthermore the Python principle was always against functions doing stack introspection to find the caller. There are few examples in the stdlib or builtins that do some sort of caller introspection. Those are the special functions `vars`, `locals`, `gloabal`, `vars` and some functions in the inspect module. And all of them do nothing more than getting the current frame and accessing the dict of locals or globals. What super in current Python 3 builds does goes way beyond that. The implementation of the automatic super currently has two ugly details that I think violate the Python Zen: The bytecode generated is differently if the name "super" is used in the function. `__class__` is only added as cell to the code if `super` or `__class__` is referenced. That and the fact that `f_localsplus` is completely unavailable from within python makes the whole process appear magical. This is way more magical than anything we've had in Python in the past and just doesn't fit into the language in my opinion. We do have an explicit self in methods and methods are more or less just functions. Python's methods are functions, just that a descriptor puts a method object around it to pass the self as first arguments. That's an incredible cool thing to have and makes things very simple and non-magical. Breaking that principle by introducing an automatic super seems to harm the concept. Another odd thing is that Python 3 starts keeping information on the C layer we can't access from within Python. Super is one example, another good one are methods. They don't have a descriptor that wraps them if they are accessed via their classes. This as such is not a problem as you can call them the same (just that you can call them with completely different receivers now) but it becomes a problem if some of the functions are marked as staticmethods. Then they look completely the same when looking at them from a classes perspective: | >>> class C: | ... normal = lambda x: None | ... static = staticmethod(lambda x: None) | ... | >>> type(C.normal) is type(C.static) | True | >>> C.normal | <function <lambda> at 0x4da150> As far as I can see a documentation tool has no chance to keep them apart even though they are completely different on an instance: | >>> type(C().normal) is type(C().static) | False | >>> C().normal | <bound method C.<lambda> of <__main__.C object at 0x4dbcf0>> | >>> C().static | <function <lambda> at 0x4da198> While I don't knwo about the method thing, I think an automatic super should at least be implementable from within Python. I could imagine that by adding __class__ and __self__ to scopes automatically a lot of that magic could be removed. Regards, Armin

12 17

Warn about mktemp once again?
by skip＠pobox.com 09 May '08

09 May '08

Back in r29829, Guido commented out the security hole warning for tempfile.mktemp: r29829 | gvanrossum | 2002-11-22 09:56:29 -0600 (Fri, 22 Nov 2002) | 3 lines Comment out the warnings about mktemp(). These are too annoying, and often unavoidable. Any thought about whether this warning should be restored? We're 5+ years later. Hopefully many uses of mktemp have been removed. If we're not going to restore the warning perhaps the commented code should just be deleted. Skip

13 32

Another GSoC Student Introduction
by Heiko Weinen 06 May '08

06 May '08

Hi! My name is Heiko N. Weinen and I was accepted as GSoC Student, too. Hoorray! The project i chose is about Filesystem Virtualisation for Python's Standard Library, hopefully something which will prove quite useful. :) Christian Heimes is my Mentor and I'd like to thank him right now for his help. Feel free to contact me, if you have any further questions or related ideas! I will set up a project site at http://heiko.weinen.org/gsoc08 in a few days. Regards, Heiko -- Lubarsky's Law of Cybernetic Entomology: prov. "There is always one more bug."

2 2

Module Suggestion: ast
by Armin Ronacher 04 May '08

04 May '08

Hi all, I would like to propose a new module for the stdlib for Python 2.6 and higher: "ast". The motivation for this module is the pending deprecation for compiler.ast which is widely used (debugging, template engines, code coverage etc.). _ast is a very solid module and is without a doubt easier to maintain then compiler.ast which was written in Python, it's lacking some features such as pretty printing the AST or traversing it. The idea of "ast" would be adding high level functionality for easier working with the AST. It currently provides these features: - pretty printing AST objects - a parse function as easier alias for compile() + flag - operator-node -> operator symbol mappings (eg: _ast.Add -> '+') - methods to modify lineno / col_offset (incrementing or copying the data over from existing nodes) - getting the fields of nodes as dicts - iterating over all child nodes - a function to get the docstring or an AST node - a node walker that yields all child-nodes recursively - a `NodeVistor` and `NodeTransformer` Additionally there is a `literate_eval` function in that module that can safely evaluate python literals in a string. Module and unittests are located in the Pocoo Sandbox HG repository: http://dev.pocoo.org/hg/sandbox/file/tip/ast/ast.py http://dev.pocoo.org/hg/sandbox/file/tip/ast/test_ast.py A slightly modified version of the ast.py module for Python 2.5 compatibility is currently in use for the Mako template engine to achieve support for Google's AppEngine. An example module for the NodeVisitor is in the repository which converts a Python AST back into Python source code: http://dev.pocoo.org/hg/sandbox/file/tip/ast/codegen.py Regards, Armin

5 7

Re: [Python-Dev] Py3k and asyncore/asynchat
by Josiah Carlson 04 May '08

04 May '08

On Tue, Mar 25, 2008 at 11:26 PM, Neal Norwitz <nnorwitz(a)gmail.com> wrote: > Any reason this was sent just to me and not the list? Because gmail only replies to the sender by default. I need to remember to cc python-dev when I reply (I used the same email client for 8 1/2 years, remembering the quirks of gmail may take some time). > On Tue, Mar 25, 2008 at 10:34 PM, Josiah Carlson > <josiah.carlson(a)gmail.com> wrote: > > > > On Tue, Mar 25, 2008 at 9:00 PM, Neal Norwitz <nnorwitz(a)gmail.com> wrote: > > > On Thu, Feb 14, 2008 at 10:09 AM, Giampaolo Rodola' <gnewsg(a)gmail.com> wrote: > > > > On 14 Feb, 16:36, "Giampaolo Rodola'" <gne...(a)gmail.com> wrote: > > > > > Ok, I'll try to take a look at all asyncore/chat reports and try to > > > > > summarize them by splitting patches which solve bugs and patches which > > > > > add enhancements or functionalities. > > > > > > > > > > > > > > === Patches for existing issues === > > > > > > > > - 1736190 which includes fixes for the following issues among other > > > > improvements: > > > > - 1063924 (asyncore should handle ECONNRESET in send()) > > > > - 1736101 (asyncore should handle ECONNABORTED in recv()) > > > > - 760475 (handle_error() should call handle_close() instead of > > > > close()) > > > > - 1740572 (refill_buffer() should call handle_close() rather than > > > > close()) > > > > - 777588 (wrong "connection refused" behavior on Windows) > > > > - 889153 (wrong connect() behavior) > > > > - 953599 (asyncore misses socket closes when poll is used) > > > > - 1025525 (asyncore.file_dispatcher should not take fd as argument) > > > > > > > > - 1519 (async_chat.__init__() and asyncore.dispatcher.__init__ > > > > parameters inconsistency) > > > > - 1541 (Bad OOB data management when using asyncore with > > > > select.poll()) > > > > - 2073 (asynchat push always sends 512 bytes (ignoring > > > > ac_out_buffer_size)) > > > > > > > > > > > > === Open issues with no patches (need review) === > > > > > > > > - 658749 (asyncore connect() and winsock errors) > > > > - 1161031 (neverending warnings from asyncore) > > > > > > > > > > > > === Enhancements & new features === > > > > > > > > - 1641 (add delayed calls feature) > > > > - 1563 (conversion to py3k and some other changes) > > > > > > That's a lot of patches. My immediate concern is that test_asynchat > > > is very flaky and fails often. Sometimes it causes other tests to > > > fail. Is there a patch that addresses this? If you need examples, > > > just look through the buildbot mails that mention test_asynchat in: > > > http://mail.python.org/pipermail/python-checkins/ > > > > No, it's one patch. All of the fixes were performed mostly by myself > > last spring, tested and verified in personal servers, then re-used by > > Giampaolo in his async ftp server (which pointed out a few small bugs, > > which have been fixed). > > > > > > > Some platforms have more problems than others, but almost all > > > platforms have failed test_asynchat at one point or another. > > > > Certainly that is the case. And according to my reading of a few > > buildbot failures, aside from someone breaking asyncore itself, the > > failures seem to stem from the test being unable to create a port to > > listen on in order to test the server/client functionality. This is a > > buildbot configuration issue (per host), not an asyncore issue. > > That was the case a long time (~3? months) ago, but hasn't been the > case recently. See my recent message about the release. I'll look for it tomorrow. For reference, searches of 'site:mail.python.org test_asynchat failure buildbot' only seem to produce the socket listen error. If there is a better incantation to get google to produce the proper errors (and/or a link), I would appreciate the help. > > > Please break up the patches into 2 sets and prioritize the patches > > > with the set. > > > > > > Set #1: Patches that have a test and doc updates if required > > > Set #2: Patches that don't have a test or doc updates as required > > > > > > For the patches that fall into Set #1, list them in priority order. > > > Top priority would be a problem that fixes failures seen in the > > > buildbots. Next priority would go to the patches that solve more > > > serious problems. Post the results here. I will try to look at them. > > > > > > For every patch you list, make sure that it conforms to the proper > > > style (e.g, PEP 8) and is essentially perfect and ready for inclusion. > > > This means that there is a single file to download that contains all > > > the modifications. The changes are appropriately commented, lines are > > > less than 80 characters, etc. One of the modifications should be an > > > entry in Misc/NEWS. > > > > I lied earlier; really there are two patches. The first is a patch to > > asyncore.py and asynchat.py . It addresses those bugs that Giampaolo > > has listed, it is tested, and works. The second patch is to update > > the documentation to mention the sample methods in asynchat for use as > > examples, as well as any other changes to the language used in the > > documentation that I had made last spring, but which are out of date > > from my posting of the original patch. I can update the documentation > > in the next week. > > Can you provide a link to the patches? Do the patches include changes > to test_asyncore and test_asynchat? The next release is April 2. I > would like to commit any patches before Monday to ensure they are > stable. Can you get me the patches by this Saturday? See http://bugs.python.org/issue1736190 for an updated patch for the modules. The current test cases pass without issue, though we may want to add tests, which I need to look at the original patch and the original file from which it was created against, then compare it with the most recent changes to the tests from Facundo last June or July. I should have the time to get patches for tests and documentation by Monday. - Josiah

2 3

socket.try_reuse_address()
by Trent Nelson 01 May '08

01 May '08

Since the recent changes to networking-oriented tests (issue 2550, r62234 and r62237), I think it's safe to say stability of the test suite on all the non-Windows platforms has improved significantly in this area (I don't recall seeing any socket errors in *nix buildbot logs since those commits). However, Windows buildbots are still periodically failing. More specifically, my Windows buildbots are still failing. One thing that's different about my buildbots is that two are being run at the same time for both trunk and py3k -- one doing an x86 build, the other doing x64 build. Since the changes in the aforementioned revisions, the behaviour of my buildbots has definitely improved -- they no longer completely wedge on test_async(chat|core), mainly due to abolishing all use of SO_REUSEADDR as a socket option in any network-oriented tests. However, periodically, they're still dying/failing in a variety of ways -- see relevant log snippets at the end of this e-mail for examples. I attribute this to the fact that SO_REUSEADDR is still set as a socket option in asyncore.py and SocketServer.py. Basically, SO_REUSEADDR should *never* be set on Windows for TCP/IP sockets. Using asyncore.py as an example, here are two ways we could handle this: 1. Hard code the Windows opt-out: --- asyncore.py (revision 62509) +++ asyncore.py (working copy) @@ -267,6 +267,8 @@ def set_reuse_addr(self): # try to re-use a server port if possible + if os.name == 'nt' and self.socket.socket_type != socket.SOCK_DGRAM: + return try: self.socket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 2. Introduce socket.try_reuse_address(): --- asyncore.py (revision 62509) +++ asyncore.py (working copy) @@ -266,15 +266,7 @@ self.add_channel(map) def set_reuse_addr(self): - # try to re-use a server port if possible - try: - self.socket.setsockopt( - socket.SOL_SOCKET, socket.SO_REUSEADDR, - self.socket.getsockopt(socket.SOL_SOCKET, - socket.SO_REUSEADDR) | 1 - ) - except socket.error: - pass + self.socket.try_reuse_address() With try_use_address implemented as follows: --- socket.py (revision 62509) +++ socket.py (working copy) @@ -197,6 +197,10 @@ Return a new socket object connected to the same system resource.""" return _socketobject(_sock=self._sock) + def try_reuse_address(self): + if not (os.name == 'nt' and self._sock.type != SOCK_DGRAM): + self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) + def makefile(self, mode='r', bufsize=-1): """makefile([mode[, bufsize]]) -> file object I prefer the latter as it's cleaner, easier to document and encapsulates what we're trying to do relatively well. The affected modules would be asyncore.py, SocketServer.py and idlelib/rpc.py. Thoughts? Regards, Trent. <eg1> test_ftplib remoteFailed: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion. ] </eg1> <eg2> test_asynchat test test_asynchat failed -- errors occurred; run in verbose mode for details [snip to bottom of log where test_asynchat is re-run] 1 test failed: test_asynchat 33 tests skipped: test__locale test_aepack test_applesingle test_cProfile test_commands test_crypt test_curses test_dbm test_epoll test_fcntl test_fork1 test_gdbm test_grp test_ioctl test_kqueue test_macostools test_mhlib test_nis test_openpty test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd test_resource test_scriptpackages test_signal test_syslog test_threadsignals test_wait3 test_wait4 test_zipfile64 Those skips are all expected on win32. Re-running failed tests in verbose mode Re-running test 'test_asynchat' in verbose mode test_close_when_done (test.test_asynchat.TestAsynchat) ... ok test_empty_line (test.test_asynchat.TestAsynchat) ... ok test_line_terminator1 (test.test_asynchat.TestAsynchat) ... ok test_line_terminator2 (test.test_asynchat.TestAsynchat) ... ok test_line_terminator3 (test.test_asynchat.TestAsynchat) ... ok test_none_terminator (test.test_asynchat.TestAsynchat) ... ok test_numeric_terminator1 (test.test_asynchat.TestAsynchat) ... ok test_numeric_terminator2 (test.test_asynchat.TestAsynchat) ... ok test_simple_producer (test.test_asynchat.TestAsynchat) ... ok test_string_producer (test.test_asynchat.TestAsynchat) ... ok test_close_when_done (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_empty_line (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator1 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator2 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator3 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_none_terminator (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_numeric_terminator1 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_numeric_terminator2 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_simple_producer (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_string_producer (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_find_prefix_at_end (test.test_asynchat.TestHelperFunctions) ... ok test_basic (test.test_asynchat.TestFifo) ... ok test_given_list (test.test_asynchat.TestFifo) ... ok ---------------------------------------------------------------------- Ran 23 tests in 11.812s OK </eg2> (Note that re-running the test here didn't result in the test failing again.) <eg3> 1 test failed: test_smtplib Traceback (most recent call last): File "S:\buildbots\python\3.0.nelson-windows\build\lib\threading.py", line 493, in _bootstrap_inner self.run() File "S:\buildbots\python\3.0.nelson-windows\build\lib\threading.py", line 449, in run self._target(*self._args, **self._kwargs) File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_smtplib.py", line 106, in debugging_server poll_fun(0.01, asyncore.socket_map) File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 132, in poll read(obj) File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 72, in read obj.handle_error() File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 68, in read obj.handle_read_event() File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 390, in handle_read_event self.handle_read() File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_ssl.py", line 524, in handle_read data = self.recv(1024) File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 342, in recv data = self.socket.recv(buffer_size) File "S:\buildbots\python\3.0.nelson-windows\build\lib\ssl.py", line 247, in recv return self.read(buflen) File "S:\buildbots\python\3.0.nelson-windows\build\lib\ssl.py", line 162, in read v = self._sslobj.read(len or 1024) socket.error: [Errno 10053] An established connection was aborted by the software in your host machine </eg3>

5 9

Module properties for C modules
by Christian Heimes 01 May '08

01 May '08

Hello Python Dev! As you all know modules don't support properties. However several places and modules could use properties on module level. For example the sys.py3k_warnings flag could be implemented with a property. Other flags in the sys module could benefit from read only properties, too. How do you like the general idea of properties for builtin modules. That is modules written in C like the sys module. Christian

3 9

shal we redefine "module" and "package"?
by zooko 01 May '08

01 May '08

Folks: Here's an experiment you can perform. Round up a Python programmer and ask him the following three questions: Q1. You type "import foo" and it works. What kind of thing is foo? Q2. You go to the Python package index and download something named "bar-1.0.0.tar.gz". What kind of thing is bar? Q3. What is a "distribution"? I'm willing to bet that you will get the following answers: A1. foo is a module. A2. bar is a package. A3. A distribution is a version of Linux that comes with a lot of Free Software. Unfortunately these answers aren't quite right. A "package" is actually a directory containing an __init__.py file, and a distribution is actually what you think of when you say "package" -- a reusable package of Python code that you can, for example, get from the Python package index. Educational efforts such as the Python tutorial and the distutils docs have not succeeded in training Python programmers to understand the terminology for these things as used by the Python implementors, so perhaps instead the implementors should start using the terminology understood by the programmers: 1. A "module" shall henceforth be the name for either a foo.py file (a single-file module), or a directory with an __init__.py in it (a directory module). 2. A "package" shall henceforth be the name of the thing that is currently called a "distribution". Regards, Zooko who doesn't mind stirring up trouble on occasion...

7 7

Debian/alpha test_math failures
by Mark Dickinson 30 Apr '08

30 Apr '08

Hello all, test_math is currently failing on the Debian/alpha buildbots (trunk and py3k). I've been trying, unsuccessfully, to figure out what's going wrong, and I'm starting to run out of ideas, so I thought I'd ask the list for help to see if anyone has any useful suggestions. Details of the failure: running the following code: from math import log ar = 9.88e-324 x = log(ar) produces: ValueError: math domain error Somehow, it looks like errno is getting set to something nonzero in math_1_from_whatever in mathmodule.c, but I really can't figure out how. I've tried adding '-mieee' to the gcc compile flags, and I've added a bunch of autoconf tests to verify that log(9.88e-324) succeeds, produces roughly the right result, and doesn't set errno to anything nonzero. All the autoconf tests that should pass do. So I can't find anything wrong with the libm implementation of log. test_math is fine on Tru64/alpha. Does anyone have access to a Linux/alpha machine, and a few minutes to figure out exactly what's failing? Or any suggestions about what might be failing. I'm open to wild ideas at this point... :-) Mark

1 0

Python-Dev April 2008