Mailman 3 May 2008 - Python-Dev

Next PyPy sprint in Berlin, 17-22 May
by Armin Rigo May 5, 2008

May 5, 2008

Hi all, Our next PyPy sprint will take place in Berlin, 17-22nd May 2008. More precisely, the sprint will be in the crashed c-base space station, Berlin, Germany, Earth, Solar System. This is a fully public sprint: newcomers (from all planets) and topics other than those proposed in the announcement are welcome. For more information: http://codespeak.net/pypy/extradoc/sprintinfo/berlin-2008/announcement.html A bientot, Armin.

2 1

Re: [Python-Dev] Py3k and asyncore/asynchat
by Josiah Carlson May 5, 2008

May 5, 2008

On Tue, Mar 25, 2008 at 11:26 PM, Neal Norwitz <nnorwitz(a)gmail.com> wrote: > Any reason this was sent just to me and not the list? Because gmail only replies to the sender by default. I need to remember to cc python-dev when I reply (I used the same email client for 8 1/2 years, remembering the quirks of gmail may take some time). > On Tue, Mar 25, 2008 at 10:34 PM, Josiah Carlson > <josiah.carlson(a)gmail.com> wrote: > > > > On Tue, Mar 25, 2008 at 9:… [View More]00 PM, Neal Norwitz <nnorwitz(a)gmail.com> wrote: > > > On Thu, Feb 14, 2008 at 10:09 AM, Giampaolo Rodola' <gnewsg(a)gmail.com> wrote: > > > > On 14 Feb, 16:36, "Giampaolo Rodola'" <gne...(a)gmail.com> wrote: > > > > > Ok, I'll try to take a look at all asyncore/chat reports and try to > > > > > summarize them by splitting patches which solve bugs and patches which > > > > > add enhancements or functionalities. > > > > > > > > > > > > > > === Patches for existing issues === > > > > > > > > - 1736190 which includes fixes for the following issues among other > > > > improvements: > > > > - 1063924 (asyncore should handle ECONNRESET in send()) > > > > - 1736101 (asyncore should handle ECONNABORTED in recv()) > > > > - 760475 (handle_error() should call handle_close() instead of > > > > close()) > > > > - 1740572 (refill_buffer() should call handle_close() rather than > > > > close()) > > > > - 777588 (wrong "connection refused" behavior on Windows) > > > > - 889153 (wrong connect() behavior) > > > > - 953599 (asyncore misses socket closes when poll is used) > > > > - 1025525 (asyncore.file_dispatcher should not take fd as argument) > > > > > > > > - 1519 (async_chat.__init__() and asyncore.dispatcher.__init__ > > > > parameters inconsistency) > > > > - 1541 (Bad OOB data management when using asyncore with > > > > select.poll()) > > > > - 2073 (asynchat push always sends 512 bytes (ignoring > > > > ac_out_buffer_size)) > > > > > > > > > > > > === Open issues with no patches (need review) === > > > > > > > > - 658749 (asyncore connect() and winsock errors) > > > > - 1161031 (neverending warnings from asyncore) > > > > > > > > > > > > === Enhancements & new features === > > > > > > > > - 1641 (add delayed calls feature) > > > > - 1563 (conversion to py3k and some other changes) > > > > > > That's a lot of patches. My immediate concern is that test_asynchat > > > is very flaky and fails often. Sometimes it causes other tests to > > > fail. Is there a patch that addresses this? If you need examples, > > > just look through the buildbot mails that mention test_asynchat in: > > > http://mail.python.org/pipermail/python-checkins/ > > > > No, it's one patch. All of the fixes were performed mostly by myself > > last spring, tested and verified in personal servers, then re-used by > > Giampaolo in his async ftp server (which pointed out a few small bugs, > > which have been fixed). > > > > > > > Some platforms have more problems than others, but almost all > > > platforms have failed test_asynchat at one point or another. > > > > Certainly that is the case. And according to my reading of a few > > buildbot failures, aside from someone breaking asyncore itself, the > > failures seem to stem from the test being unable to create a port to > > listen on in order to test the server/client functionality. This is a > > buildbot configuration issue (per host), not an asyncore issue. > > That was the case a long time (~3? months) ago, but hasn't been the > case recently. See my recent message about the release. I'll look for it tomorrow. For reference, searches of 'site:mail.python.org test_asynchat failure buildbot' only seem to produce the socket listen error. If there is a better incantation to get google to produce the proper errors (and/or a link), I would appreciate the help. > > > Please break up the patches into 2 sets and prioritize the patches > > > with the set. > > > > > > Set #1: Patches that have a test and doc updates if required > > > Set #2: Patches that don't have a test or doc updates as required > > > > > > For the patches that fall into Set #1, list them in priority order. > > > Top priority would be a problem that fixes failures seen in the > > > buildbots. Next priority would go to the patches that solve more > > > serious problems. Post the results here. I will try to look at them. > > > > > > For every patch you list, make sure that it conforms to the proper > > > style (e.g, PEP 8) and is essentially perfect and ready for inclusion. > > > This means that there is a single file to download that contains all > > > the modifications. The changes are appropriately commented, lines are > > > less than 80 characters, etc. One of the modifications should be an > > > entry in Misc/NEWS. > > > > I lied earlier; really there are two patches. The first is a patch to > > asyncore.py and asynchat.py . It addresses those bugs that Giampaolo > > has listed, it is tested, and works. The second patch is to update > > the documentation to mention the sample methods in asynchat for use as > > examples, as well as any other changes to the language used in the > > documentation that I had made last spring, but which are out of date > > from my posting of the original patch. I can update the documentation > > in the next week. > > Can you provide a link to the patches? Do the patches include changes > to test_asyncore and test_asynchat? The next release is April 2. I > would like to commit any patches before Monday to ensure they are > stable. Can you get me the patches by this Saturday? See http://bugs.python.org/issue1736190 for an updated patch for the modules. The current test cases pass without issue, though we may want to add tests, which I need to look at the original patch and the original file from which it was created against, then compare it with the most recent changes to the tests from Facundo last June or July. I should have the time to get patches for tests and documentation by Monday. - Josiah [View Less]

2 3

Duplicate tests
by Benjamin Peterson May 4, 2008

May 4, 2008

test_builtin tests most of the builtin types some how or another. All of these also have complete suites elsewhere in the tests. Shouldn't the tests in test_builtin be moved to the type specific tests? -- Cheers, Benjamin Peterson

2 1

AST Optimization: Branch Elimination in Generator Functions
by Thomas Lee May 3, 2008

May 3, 2008

The next problem that cropped up during the implementation of the AST code optimizer is related to branch elimination and the elimination of any code after a return. Within a FunctionDef node, we would (ideally) like to blow away If nodes with a constant - but false - test expression. e.g.: def foo(): if False: # ... stuff ... For most functions, this will cause no problems and the code will behave as expected. However, if the eliminated branch contains a "yield" expression, the … [View More]

1 0

Special offer! Ten code reviews
by Guido van Rossum May 3, 2008

May 3, 2008

I'd like to get some more people trying out codereview.appspot.com, so I'm offering the first 10 people to submit a new patch there for my review to do the review by Monday. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

2 1

tracker issues? server error
by Gregory P. Smith May 3, 2008

May 3, 2008

trying to update a bug I get: Fri May 2 07:17:17 2008: An error occurred. Please check the server log for more infomation.

2 1

Summary of Tracker Issues
by Tracker May 2, 2008

May 2, 2008

ACTIVITY SUMMARY (04/25/08 - 05/02/08) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1850 open (+29) / 12712 closed (+16) / 14562 total (+45) Open issues with patches: 560 Average duration of open issues: 711 days. Median duration of open issues: 1347 days. Open Issues Breakdown open 1827 (+29) pending 23 ( +0) Issues Created Or Reopened (45) _______________________________ … [View More]

1 0

Should bytearray(b'a')[0] should 1-char or number?
by ocean May 2, 2008

May 2, 2008

# I tried to post this to bug tracker, but couldn't, so posted here... r62095 says >Fix and enable a skipped test: >with python 2.6, enumerating bytes yields 1-char strings, not numbers. > >Don't merge this into the py3k branch. This is true for bytes, but not for bytearray. >>> bytearray(b'a')[0] 97 [28493 refs] >>> b'a'[0] 'a' And this causes error on my environment like this. ====================================================================== FAIL: … [View More]

3 2

socket.try_reuse_address()
by Trent Nelson May 2, 2008

May 2, 2008

Since the recent changes to networking-oriented tests (issue 2550, r62234 and r62237), I think it's safe to say stability of the test suite on all the non-Windows platforms has improved significantly in this area (I don't recall seeing any socket errors in *nix buildbot logs since those commits). However, Windows buildbots are still periodically failing. More specifically, my Windows buildbots are still failing. One thing that's different about my buildbots is that two are being run at the … [View More]same time for both trunk and py3k -- one doing an x86 build, the other doing x64 build. Since the changes in the aforementioned revisions, the behaviour of my buildbots has definitely improved -- they no longer completely wedge on test_async(chat|core), mainly due to abolishing all use of SO_REUSEADDR as a socket option in any network-oriented tests. However, periodically, they're still dying/failing in a variety of ways -- see relevant log snippets at the end of this e-mail for examples. I attribute this to the fact that SO_REUSEADDR is still set as a socket option in asyncore.py and SocketServer.py. Basically, SO_REUSEADDR should *never* be set on Windows for TCP/IP sockets. Using asyncore.py as an example, here are two ways we could handle this: 1. Hard code the Windows opt-out: --- asyncore.py (revision 62509) +++ asyncore.py (working copy) @@ -267,6 +267,8 @@ def set_reuse_addr(self): # try to re-use a server port if possible + if os.name == 'nt' and self.socket.socket_type != socket.SOCK_DGRAM: + return try: self.socket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 2. Introduce socket.try_reuse_address(): --- asyncore.py (revision 62509) +++ asyncore.py (working copy) @@ -266,15 +266,7 @@ self.add_channel(map) def set_reuse_addr(self): - # try to re-use a server port if possible - try: - self.socket.setsockopt( - socket.SOL_SOCKET, socket.SO_REUSEADDR, - self.socket.getsockopt(socket.SOL_SOCKET, - socket.SO_REUSEADDR) | 1 - ) - except socket.error: - pass + self.socket.try_reuse_address() With try_use_address implemented as follows: --- socket.py (revision 62509) +++ socket.py (working copy) @@ -197,6 +197,10 @@ Return a new socket object connected to the same system resource.""" return _socketobject(_sock=self._sock) + def try_reuse_address(self): + if not (os.name == 'nt' and self._sock.type != SOCK_DGRAM): + self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) + def makefile(self, mode='r', bufsize=-1): """makefile([mode[, bufsize]]) -> file object I prefer the latter as it's cleaner, easier to document and encapsulates what we're trying to do relatively well. The affected modules would be asyncore.py, SocketServer.py and idlelib/rpc.py. Thoughts? Regards, Trent. <eg1> test_ftplib remoteFailed: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion. ] </eg1> <eg2> test_asynchat test test_asynchat failed -- errors occurred; run in verbose mode for details [snip to bottom of log where test_asynchat is re-run] 1 test failed: test_asynchat 33 tests skipped: test__locale test_aepack test_applesingle test_cProfile test_commands test_crypt test_curses test_dbm test_epoll test_fcntl test_fork1 test_gdbm test_grp test_ioctl test_kqueue test_macostools test_mhlib test_nis test_openpty test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd test_resource test_scriptpackages test_signal test_syslog test_threadsignals test_wait3 test_wait4 test_zipfile64 Those skips are all expected on win32. Re-running failed tests in verbose mode Re-running test 'test_asynchat' in verbose mode test_close_when_done (test.test_asynchat.TestAsynchat) ... ok test_empty_line (test.test_asynchat.TestAsynchat) ... ok test_line_terminator1 (test.test_asynchat.TestAsynchat) ... ok test_line_terminator2 (test.test_asynchat.TestAsynchat) ... ok test_line_terminator3 (test.test_asynchat.TestAsynchat) ... ok test_none_terminator (test.test_asynchat.TestAsynchat) ... ok test_numeric_terminator1 (test.test_asynchat.TestAsynchat) ... ok test_numeric_terminator2 (test.test_asynchat.TestAsynchat) ... ok test_simple_producer (test.test_asynchat.TestAsynchat) ... ok test_string_producer (test.test_asynchat.TestAsynchat) ... ok test_close_when_done (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_empty_line (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator1 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator2 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator3 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_none_terminator (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_numeric_terminator1 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_numeric_terminator2 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_simple_producer (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_string_producer (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_find_prefix_at_end (test.test_asynchat.TestHelperFunctions) ... ok test_basic (test.test_asynchat.TestFifo) ... ok test_given_list (test.test_asynchat.TestFifo) ... ok ---------------------------------------------------------------------- Ran 23 tests in 11.812s OK </eg2> (Note that re-running the test here didn't result in the test failing again.) <eg3> 1 test failed: test_smtplib Traceback (most recent call last): File "S:\buildbots\python\3.0.nelson-windows\build\lib\threading.py", line 493, in _bootstrap_inner self.run() File "S:\buildbots\python\3.0.nelson-windows\build\lib\threading.py", line 449, in run self._target(*self._args, **self._kwargs) File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_smtplib.py", line 106, in debugging_server poll_fun(0.01, asyncore.socket_map) File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 132, in poll read(obj) File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 72, in read obj.handle_error() File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 68, in read obj.handle_read_event() File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 390, in handle_read_event self.handle_read() File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_ssl.py", line 524, in handle_read data = self.recv(1024) File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 342, in recv data = self.socket.recv(buffer_size) File "S:\buildbots\python\3.0.nelson-windows\build\lib\ssl.py", line 247, in recv return self.read(buflen) File "S:\buildbots\python\3.0.nelson-windows\build\lib\ssl.py", line 162, in read v = self._sslobj.read(len or 1024) socket.error: [Errno 10053] An established connection was aborted by the software in your host machine </eg3> [View Less]

5 9

Re: [Python-Dev] Problems with the new super()
by Phillip J. Eby May 2, 2008

May 2, 2008

At 04:38 PM 5/1/2008 -0300, Facundo Batista wrote: >Has super() proved more useful than harmful? For me, yes. I use it all the time. The only time I use explicit-target upcalls is in __init__ methods, and there usually only to skip a subclass' init or to explicitly manage a tricky bit of multiple inheritance. (Note, by the way, that you cannot safely write an upcall in a mixin class without super, so it can't safely be done away with, anyway.)

2 1