<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16608" name=GENERATOR></HEAD>
<BODY>
<DIV><SPAN class=583055823-08042008><FONT face=Verdana color=#0000ff
size=2>Committed the patch in r62234. Hopefully the work paid off!
(He says moments before all the buildbots turn red...)</FONT></SPAN></DIV><BR>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Gregory P. Smith
[mailto:greg@krypto.org] <BR><B>Sent:</B> 08 April 2008 20:58<BR><B>To:</B>
Trent Nelson<BR><B>Cc:</B> python-dev@python.org<BR><B>Subject:</B> Re:
[Python-Dev] Changing all network-oriented tests to facilitate (virtually
unlimited) parallel test execution [Issue#: 2550]<BR></FONT><BR></DIV>
<DIV></DIV><BR><BR>
<DIV class=gmail_quote>On Tue, Apr 8, 2008 at 5:00 AM, Trent Nelson <<A
href="mailto:tnelson@onresolve.com">tnelson@onresolve.com</A>> wrote:<BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid"><BR>I've
forwarding my most recent update to issue 2550 here such that the proposed
patch (and in general, the approach to network-oriented test cases) can be
vetted by a wider audience:<BR><BR><A
href="http://bugs.python.org/file9980/trunk.2550-2.patch"
target=_blank>http://bugs.python.org/file9980/trunk.2550-2.patch</A><BR><BR>This
patch works towards fixing a large proportion of the tests that were written
in such a way that often leads to buildbot errors when port conflicts arise.
With this patch applied, I can run many instances of the test suite in
parallel and none of the network-oriented tests fail (which they do
currently if you try and do this).<BR><BR>Plenty of discussion (mostly me
replying to my own comments) on the subject at: <A
href="http://bugs.python.org/issue2550"
target=_blank>http://bugs.python.org/issue2550</A>.<BR><BR>Anyone have any
issues with this new approach? I'm particularily interested in whether
or not people disagree with the assumption I've drawn regarding never using
SO_REUSEADDR in tests for TCP/IP sockets (see below).<BR><BR>
Trent.</BLOCKQUOTE>
<DIV><BR>I think your assumptions are fair.<BR><BR>Note that not using
SO_REUSEADDR can reserve the port for a few minutes beyond the actual
unbinding destruction of the server socket on some OSes but with tens of
thousands of ports available and likely not more than 1-200 being needed by a
full test suite run and how long the test suite takes to run I think that is
acceptable and is not a problem we're likely to ever run into (fix it only
iffwe ever do).<BR><BR>-gps<BR> <BR></DIV>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid"><BR><BR><BR>________________________________________<BR>From:
python-bugs-list-bounces+tnelson=<A
href="mailto:onresolve.com@python.org">onresolve.com@python.org</A>
[python-bugs-list-bounces+tnelson=<A
href="mailto:onresolve.com@python.org">onresolve.com@python.org</A>] On
Behalf Of Trent Nelson [<A
href="mailto:report@bugs.python.org">report@bugs.python.org</A>]<BR>Sent: 08
April 2008 12:49<BR>To: <A
href="mailto:python-bugs-list@python.org">python-bugs-list@python.org</A><BR>Subject:
[issue2550] SO_REUSEADDR doesn't have the same semantics on Windows
as on Unix<BR><BR>Trent Nelson <<A
href="mailto:tnelson@onresolve.com">tnelson@onresolve.com</A>> added the
comment:<BR><BR>Invested quite a few cycles on this issue last night.
The more time I<BR>spent on it, the more I became convinced that every
single test working<BR>with sockets should be changed in one fell swoop in
order to facilitate<BR>(virtually unlimited) parallel test execution without
fear of port<BR>conflicts.<BR><BR>I've attached a second patch,
trunk.2550-2.patch, which is my progress<BR>so far on doing just this.
The main changes can be expressed by the<BR>following two
points:<BR><BR>a) do whatever it takes in network-oriented tests to
ensure<BR> unique ports are obtained (relying on the bind_port()
and<BR> find_unused_port() methods exposed by test_support)<BR><BR>b)
never, ever, ever call SO_REUSEADDR on a socket from a test;<BR>
because we're putting so much effort into obtaining a unique<BR> port,
this should never be necessary -- in the rare cases that<BR> our
attempts to obtain a unique port fail, then we absolutely<BR> should
fail with EADDRINUSE, as the ability to obtain a unique<BR> port for
the duration of a client/server test is an invariant<BR> that we
*must* be able to depend upon. If the invariant is<BR> broken,
fail immediately (don't mask the problem with<BR>
SO_REUSEADDR).<BR><BR>With this patch applied, I can spawn a handful of
Python processes and<BR>run the entire test suite (without -r, ensuring all
tests are run in<BR>the same order, which should encourage port conflicts
(if there were<BR>any)) without any errors*. Doing that now is
completely and utterly<BR>impossible.<BR><BR>[*] Well, almost without error.
All the I/O related tests that try and<BR>open @test fail.<BR><BR>I
believe there's still outstanding work to do with this patch with<BR>regards
to how the intracacies of SO_REUSEADDR and SO_EXCLUSIVEADDRUSE<BR>should be
handled in the rest of the stdlib. I'm still thinking about<BR>the
best approach for this. However, the patch as it currently
stands<BR>is still quite substantial so I wanted to get it out sooner rather
than<BR>later for review.<BR><BR>(I'll forward this to python-dev@ to try
and encourage more eyes from<BR>people with far more network-fu than
I.)<BR><BR>Added file: <A
href="http://bugs.python.org/file9980/trunk.2550-2.patch"
target=_blank>http://bugs.python.org/file9980/trunk.2550-2.patch</A><BR><BR>__________________________________<BR>Tracker
<<A
href="mailto:report@bugs.python.org">report@bugs.python.org</A>><BR><<A
href="http://bugs.python.org/issue2550"
target=_blank>http://bugs.python.org/issue2550</A>><BR>__________________________________<BR>_______________________________________________<BR>Python-bugs-list
mailing list<BR>Unsubscribe: <A
href="http://mail.python.org/mailman/options/python-bugs-list/tnelson%40onresolve.com"
target=_blank>http://mail.python.org/mailman/options/python-bugs-list/tnelson%40onresolve.com</A><BR>_______________________________________________<BR>Python-Dev
mailing list<BR><A
href="mailto:Python-Dev@python.org">Python-Dev@python.org</A><BR><A
href="http://mail.python.org/mailman/listinfo/python-dev"
target=_blank>http://mail.python.org/mailman/listinfo/python-dev</A><BR>Unsubscribe:
<A
href="http://mail.python.org/mailman/options/python-dev/greg%40krypto.org"
target=_blank>http://mail.python.org/mailman/options/python-dev/greg%40krypto.org</A><BR></BLOCKQUOTE></DIV><BR></BLOCKQUOTE></BODY></HTML>