Le mercredi 04 août 2010 à 21:43 +1000, Richard Jones a écrit :

On Wed, Aug 4, 2010 at 8:48 PM, Paul Moore wrote:

...

On 3 August 2010 20:30, Barry Warsaw wrote:

...

Brian is looking at Windows now (the buildbots are a sad and sorry story).

There seems to be something distinctly wrong with the 3.x buildbots. A lot of test failures and timeouts. At first I assumed it was my buildslave going flaky (again :-() but it only affects the 3.x branch, and it seems to be hitting more than just my slave. From what I'm seeing, it's often test_io that's getting stalled and then sitting round until it times out.

I'm also quite confused by the test_smtpd failures that pop up on some of the test runs that I've had absolutely no luck reproducing locally under OS X or Solaris.

It happens when running test_smtplib before test_smtpb: $./python -m test.regrtest -v test_smtplib test_smtpd == CPython 3.2a1+ (py3k:83711M, Aug 4 2010, 13:23:20) [GCC 4.4.3] == Linux-2.6.33.5-desktop-2mnb-x86_64-with-mandrake-2010.1-Official == /home/antoine/py3k/__svn__/build/test_python_13320 [1/2] test_smtplib testBasic1 (test.test_smtplib.GeneralTests) ... ok testBasic2 (test.test_smtplib.GeneralTests) ... ok testLocalHostName (test.test_smtplib.GeneralTests) ... ok testTimeoutDefault (test.test_smtplib.GeneralTests) ... ok testTimeoutNone (test.test_smtplib.GeneralTests) ... ok testTimeoutValue (test.test_smtplib.GeneralTests) ... ok testBasic (test.test_smtplib.DebuggingServerTests) ... ok testHELP (test.test_smtplib.DebuggingServerTests) ... ok testNOOP (test.test_smtplib.DebuggingServerTests) ... ok testNotImplemented (test.test_smtplib.DebuggingServerTests) ... ok testRSET (test.test_smtplib.DebuggingServerTests) ... ok testSecondHELO (test.test_smtplib.DebuggingServerTests) ... ok testSend (test.test_smtplib.DebuggingServerTests) ... ok testVRFY (test.test_smtplib.DebuggingServerTests) ... ok testNonnumericPort (test.test_smtplib.NonConnectingTests) ... ok testNotConnected (test.test_smtplib.NonConnectingTests) ... ok testFailingHELO (test.test_smtplib.BadHELOServerTests) ... ok testAUTH_CRAM_MD5 (test.test_smtplib.SMTPSimTests) ... ok testAUTH_LOGIN (test.test_smtplib.SMTPSimTests) ... ok testAUTH_PLAIN (test.test_smtplib.SMTPSimTests) ... ok testBasic (test.test_smtplib.SMTPSimTests) ... ok testEHLO (test.test_smtplib.SMTPSimTests) ... ok testEXPN (test.test_smtplib.SMTPSimTests) ... ok testVRFY (test.test_smtplib.SMTPSimTests) ... ok ---------------------------------------------------------------------- Ran 24 tests in 0.107s OK [2/2] test_smtpd test_process_message_unimplemented (test.test_smtpd.SMTPDServerTest) ... FAIL test_DATA_syntax (test.test_smtpd.SMTPDChannelTest) ... FAIL test_EHLO_not_implemented (test.test_smtpd.SMTPDChannelTest) ... FAIL test_HELO (test.test_smtpd.SMTPDChannelTest) ... FAIL test_HELO_bad_syntax (test.test_smtpd.SMTPDChannelTest) ... FAIL test_HELO_duplicate (test.test_smtpd.SMTPDChannelTest) ... FAIL test_MAIL_chevrons (test.test_smtpd.SMTPDChannelTest) ... FAIL test_MAIL_missing_from (test.test_smtpd.SMTPDChannelTest) ... FAIL test_MAIL_syntax (test.test_smtpd.SMTPDChannelTest) ... FAIL test_NOOP (test.test_smtpd.SMTPDChannelTest) ... FAIL test_NOOP_bad_syntax (test.test_smtpd.SMTPDChannelTest) ... FAIL test_QUIT (test.test_smtpd.SMTPDChannelTest) ... FAIL test_QUIT_arg_ignored (test.test_smtpd.SMTPDChannelTest) ... FAIL test_RCPT_syntax (test.test_smtpd.SMTPDChannelTest) ... FAIL test_RSET (test.test_smtpd.SMTPDChannelTest) ... ERROR test_RSET_syntax (test.test_smtpd.SMTPDChannelTest) ... FAIL test_attribute_deprecations (test.test_smtpd.SMTPDChannelTest) ... ok test_bad_state (test.test_smtpd.SMTPDChannelTest) ... ok test_broken_connect (test.test_smtpd.SMTPDChannelTest) ... ok test_data_dialog (test.test_smtpd.SMTPDChannelTest) ... FAIL test_data_transparency_section_4_5_2 (test.test_smtpd.SMTPDChannelTest) ... FAIL test_manual_status (test.test_smtpd.SMTPDChannelTest) ... FAIL test_missing_data (test.test_smtpd.SMTPDChannelTest) ... FAIL test_multiple_RCPT (test.test_smtpd.SMTPDChannelTest) ... ERROR test_need_MAIL (test.test_smtpd.SMTPDChannelTest) ... FAIL test_need_RCPT (test.test_smtpd.SMTPDChannelTest) ... FAIL test_nested_MAIL (test.test_smtpd.SMTPDChannelTest) ... FAIL test_server_accept (test.test_smtpd.SMTPDChannelTest) ... ok ====================================================================== ERROR: test_RSET (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 212, in test_RSET self.assertEqual(self.server.messages[0], IndexError: list index out of range ====================================================================== ERROR: test_multiple_RCPT (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 192, in test_multiple_RCPT self.assertEqual(self.server.messages[-1], IndexError: list index out of range ====================================================================== FAIL: test_process_message_unimplemented (test.test_smtpd.SMTPDServerTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 45, in test_process_message_unimplemented self.assertRaises(NotImplementedError, write_line, b'spam\r\n.\r\n') AssertionError: NotImplementedError not raised by write_line ====================================================================== FAIL: test_DATA_syntax (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 177, in test_DATA_syntax self.assertEqual(self.channel.socket.last, b'501 Syntax: DATA\r\n') AssertionError: b'250 Ok\r\n' != b'501 Syntax: DATA\r\n' ====================================================================== FAIL: test_EHLO_not_implemented (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 80, in test_EHLO_not_implemented b'502 Error: command "EHLO" not implemented\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'502 Error: command "EHLO" not implemented\r\n' ====================================================================== FAIL: test_HELO (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 86, in test_HELO '250 {}\r\n'.format(name).encode('ascii')) AssertionError: b'502 Error: command "199" not implemented\r\n' != b'250 \r\n' ====================================================================== FAIL: test_HELO_bad_syntax (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 91, in test_HELO_bad_syntax b'501 Syntax: HELO hostname\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'501 Syntax: HELO hostname\r\n' ====================================================================== FAIL: test_HELO_duplicate (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 97, in test_HELO_duplicate b'503 Duplicate HELO/EHLO\r\n') AssertionError: b'250 \r\n' != b'503 Duplicate HELO/EHLO\r\n' ====================================================================== FAIL: test_MAIL_chevrons (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 139, in test_MAIL_chevrons self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'250 Ok\r\n' ====================================================================== FAIL: test_MAIL_missing_from (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 135, in test_MAIL_missing_from b'501 Syntax: MAIL FROM:<address>\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'501 Syntax: MAIL FROM:<address>\r\n' ====================================================================== FAIL: test_MAIL_syntax (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 130, in test_MAIL_syntax b'501 Syntax: MAIL FROM:<address>\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'501 Syntax: MAIL FROM:<address>\r\n' ====================================================================== FAIL: test_NOOP (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 101, in test_NOOP self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'250 Ok\r\n' ====================================================================== FAIL: test_NOOP_bad_syntax (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 106, in test_NOOP_bad_syntax b'501 Syntax: NOOP\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'501 Syntax: NOOP\r\n' ====================================================================== FAIL: test_QUIT (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 110, in test_QUIT self.assertEqual(self.channel.socket.last, b'221 Bye\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'221 Bye\r\n' ====================================================================== FAIL: test_QUIT_arg_ignored (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 114, in test_QUIT_arg_ignored self.assertEqual(self.channel.socket.last, b'221 Bye\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'221 Bye\r\n' ====================================================================== FAIL: test_RCPT_syntax (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 157, in test_RCPT_syntax b'501 Syntax: RCPT TO: <address>\r\n') AssertionError: b'250 Ok\r\n' != b'501 Syntax: RCPT TO: <address>\r\n' ====================================================================== FAIL: test_RSET_syntax (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 217, in test_RSET_syntax self.assertEqual(self.channel.socket.last, b'501 Syntax: RSET\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'501 Syntax: RSET\r\n' ====================================================================== FAIL: test_data_dialog (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 161, in test_data_dialog self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'250 Ok\r\n' ====================================================================== FAIL: test_data_transparency_section_4_5_2 (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 184, in test_data_transparency_section_4_5_2 self.assertEqual(self.channel.received_data, '.') AssertionError: '' != '.' + . ====================================================================== FAIL: test_manual_status (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 201, in test_manual_status self.assertEqual(self.channel.socket.last, b'250 Okish\r\n') AssertionError: b'354 End data with <CR><LF>.<CR><LF>\r\n' != b'250 Okish\r\n' ====================================================================== FAIL: test_missing_data (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 75, in test_missing_data b'500 Error: bad syntax\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'500 Error: bad syntax\r\n' ====================================================================== FAIL: test_need_MAIL (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 125, in test_need_MAIL b'503 Error: need MAIL command\r\n') AssertionError: b'502 Error: command "199" not implemented\r\n' != b'503 Error: need MAIL command\r\n' ====================================================================== FAIL: test_need_RCPT (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 151, in test_need_RCPT b'503 Error: need RCPT command\r\n') AssertionError: b'250 Ok\r\n' != b'503 Error: need RCPT command\r\n' ====================================================================== FAIL: test_nested_MAIL (test.test_smtpd.SMTPDChannelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_smtpd.py", line 145, in test_nested_MAIL b'503 Error: nested MAIL command\r\n') AssertionError: b'250 Ok\r\n' != b'503 Error: nested MAIL command\r\n' ---------------------------------------------------------------------- Ran 28 tests in 0.013s FAILED (failures=22, errors=2) test test_smtpd failed -- multiple errors occurred 1 test OK. 1 test failed: test_smtpd

On 4 August 2010 13:05, Antoine Pitrou wrote:

...

I'm also quite confused by the test_smtpd failures that pop up on some of the test runs that I've had absolutely no luck reproducing locally under OS X or Solaris.

It happens when running test_smtplib before test_smtpb:

Nice! How did you work that out? I'd like to learn how to diagnose this sort of thing, because it seems to come up a lot, and I'm not much use at the moment :-) Paul.

...

...

It happens when running test_smtplib before test_smtpb:

Nice! How did you work that out? I'd like to learn how to diagnose this sort of thing, because it seems to come up a lot, and I'm not much use at the moment :-)

I simply tried to run test_smtplib before test_smtpd.

A more deterministic (and more tedious) way is this: if you suspect that some failure might be due to the order of test cases, take a build log from the build bot where it fails, and run the tests in the exact same order. See whether the problem reproduces. If it does, drop tests from the test sequence until you end up with the minimum number of tests that you need to run in sequence (and yes, I had interworkings of three test modules at some point). Of course, educated guessing can accelerate the process. Regards, Martin

On Wed, 04 Aug 2010 19:53:55 +0200, =?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= wrote:

...

...

...

It happens when running test_smtplib before test_smtpb:

Nice! How did you work that out? I'd like to learn how to diagnose this sort of thing, because it seems to come up a lot, and I'm not much use at the moment :-)

I simply tried to run test_smtplib before test_smtpd.

A more deterministic (and more tedious) way is this: if you suspect that some failure might be due to the order of test cases, take a build log from the build bot where it fails, and run the tests in the exact same order. See whether the problem reproduces.

If it does, drop tests from the test sequence until you end up with the minimum number of tests that you need to run in sequence (and yes, I had interworkings of three test modules at some point).

Of course, educated guessing can accelerate the process.

Two bits of info to help you implement this process: The buildbots have regrtest print out the random seed used for the test run. The --randseed option of regtest can be used to repeat this run. That's the first step, which proves that the specific order produced the test failure. Then you cut and paste the test list into a file, and use the regrtest -f option to run the tests from that file, and do your binary search by breaking up the list into pieces. Hmm. If we added a 'binsearch' option to regrtest, you could just pass it the random seed and that option and off it would go... -- R. David Murray www.bitdance.com