[New-bugs-announce] [issue38697] test.support.find_unused_port() race condition: test_socket fails with OSError: [Errno 98] Address already in use

[STINNER Victor]

New submission from STINNER Victor <vstinner at python.org>:

When two tests using support.find_unused_port() run in parallel, it's possible to get the same port number and so the first process using it wins, and the other one fails. Example with test_socket running in parallel of multiple many other tests.

Example of fix to avoid support.find_unused_port():

diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index 1bf562a03d..4479a69943 100644
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -6202,10 +6202,11 @@ class CreateServerFunctionalTest(unittest.TestCase):
                      "dualstack_ipv6 not supported")
     @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test')
     def test_dual_stack_client_v6(self):
-        port = support.find_unused_port()
-        with socket.create_server(("", port), family=socket.AF_INET6,
+        with socket.create_server(("", 0), family=socket.AF_INET6,
                                   dualstack_ipv6=True) as sock:
             self.echo_server(sock)
+            port = sock.getsockname()[1]
+            print("PORT", port)
             self.echo_client(("::1", port), socket.AF_INET6)


AMD64 RHEL7 Refleaks 3.8:
https://buildbot.python.org/all/#/builders/299/builds/48

0:02:32 load avg: 7.37 [132/423] test_timeout passed (56.5 sec) -- running: test_regrtest (1 min 56 sec), test_logging (1 min 17 sec), test_mailbox (1 min 39 sec), test_statistics (33.8 sec), test_socket (55.8 sec)
...
0:02:56 load avg: 8.33 [153/423] test_urllib2net passed -- running: test_regrtest (2 min 20 sec), test_logging (1 min 40 sec), test_mailbox (2 min 2 sec), test_statistics (57.4 sec), test_socket (1 min 19 sec)
...
0:03:11 load avg: 8.30 [161/423] test_asyncore passed -- running: test_logging (1 min 55 sec), test_support (33.7 sec), test_mailbox (2 min 17 sec), test_statistics (1 min 12 sec), test_socket (1 min 34 sec)
...
0:03:24 load avg: 8.01 [167/423] test_support passed (46.7 sec) -- running: test_mailbox (2 min 31 sec), test_multiprocessing_forkserver (40.2 sec), test_socket (1 min 47 sec)
...
0:03:36 load avg: 7.78 [180/423] test_docxmlrpc passed -- running: test_decimal (30.7 sec), test_mailbox (2 min 42 sec), test_multiprocessing_forkserver (52.1 sec), test_socket (1 min 59 sec)
...
0:03:39 load avg: 7.88 [184/423/1] test_socket failed (2 min 2 sec) -- running: test_decimal (33.9 sec), test_mailbox (2 min 46 sec), test_multiprocessing_forkserver (55.2 sec)
...
0:03:39 load avg: 7.88 [184/423/1] test_socket failed (2 min 2 sec) -- running: test_decimal (33.9 sec), test_mailbox (2 min 46 sec), test_multiprocessing_forkserver (55.2 sec)
beginning 6 repetitions
123456
....

Warning -- threading._dangling was modified by test_socket
  Before: {<weakref at 0x7f08b2b48830; to '_MainThread' at 0x7f08b73a8140>}
  After:  {<weakref at 0x7f08b07258f0; to 'Thread' at 0x7f08aefb6a00>, <weakref at 0x7f08b07257d0; to 'Thread' at 0x7f08aefb6b40>, <weakref at 0x7f08b07259b0; to '_MainThread' at 0x7f08b73a8140>, <weakref at 0x7f08b0725710; to 'Thread' at 0x7f08aefb6be0>} 

test test_socket failed -- Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.8.cstratak-RHEL7-x86_64.refleak/build/Lib/test/test_socket.py", line 6206, in test_dual_stack_client_v6
    with socket.create_server(("", port), family=socket.AF_INET6,
  File "/home/buildbot/buildarea/3.8.cstratak-RHEL7-x86_64.refleak/build/Lib/socket.py", line 886, in create_server
    raise error(err.errno, msg) from None
OSError: [Errno 98] Address already in use (while attempting to bind on address ('', 45626))

----------
components: Tests
messages: 356022
nosy: vstinner
priority: normal
severity: normal
status: open
title: test.support.find_unused_port() race condition: test_socket fails with OSError: [Errno 98] Address already in use
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue38697>
_______________________________________