[New-bugs-announce] [issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

New submission from Terry J. Reedy:

23 of the 62 idlelib/*.py files have an 'if __name__ ...' test that brings up a tkinter gui widget and waits for the human tester to do ... something. Problems:

0. They are a bit of a nuisance to run individually: either type
python -m idlelib.SomeLongNameYoumightMistype
or open in an editor and hit F5.

Running all is much worse given that I found no list; doing the above with all 62 files to find the 23 is something few would do.

1. About 5 do not run; at least 1 is a simple incomplete 2to3 fix (which means it has never been run with 3.x). I will post separately any that are not obvious.

2. They were written by and for the module authors; some are a puzzle. For example: python -m idlelib.FileList. I do not know what to do to perform the test.

3. About 4 of those that do run do not stop properly, depending on how run. Example: after python -m idlelib.ColorDelegator, the puzzling window (blank on 3.3, not 2.7) seems to close correctly, but open the same file and run with F5 (2.7 or 3.3, Windows) and pressing [x] in the window does nothing. But most do close so this is something different with a few.

4. Even when the test is fairly obvious, (python -m idlelib.MultiStatusBar) there is no way to indicate failure so that an  automated record of failures is collected.

My overall idea expands on the example of the dialog tests, such as
python -m idlelib.aboutDialog
Each opens a master test window with a start button that opens a window with the dialog be tested. Make a standardized master window for all tests, add an explanation of what to do (or a button to display it) and buttons to record the result: OK, NothingHappened, or Error (explain in text widget). A test method would set up the explanation and test item, connect both to a master window, run, retrieve result (I have no idea how to do this ;-), and end with AssertTrue(result, explanation).

I am thinking to suffix human-only tests with '_H' and put 'test_xyy_H' in a subdirectory such as idle_test/human. Some test files might test several different modules, such as 'test_dialog_H'.

----------
assignee: terry.reedy
messages: 190388
nosy: roger.serwy, terry.reedy
priority: normal
severity: normal
stage: needs patch
status: open
title: Idle: make human-mediated GUI tests usable
type: behavior
versions: Python 2.7, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue18104>
_______________________________________