[Python-Dev] We now have C code coverage!

[Terry Reedy]

On 6/22/2018 9:40 PM, Nathaniel Smith wrote:
> On Fri, Jun 22, 2018 at 6:16 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> I discovered the answer to 2. by shift-clicking on a text_x file to see
>> their coverage report for the file.  The colors actually do reflect the test
>> lines executed.  codecov.io excludes gui tests*, so the reported coverage
>> for tkinter, idlelib, and turtle is deceptive and bogus, and under-reports
>> the total cpython coverage by a percent or two.  It would be better to
>> exclude these modules.

>> * I assume that codecov.io uses linux servers.  I have read that there are
>> programs that simulate X-Windows so that gui code will execute without
>> actual terminals.
> 
> Codecov.io doesn't run any tests itself; it's just a service for
> aggregation and reporting. The coverage information is being gathered
> while running CPython's regular CI tests, and then uploaded to
> codecov.io to view.

Thank you for the information.

> So if you want to run the gui tests -- which seems like a good idea if
> possible! -- then the way to do that would be to make them run as part
> of the regular Travis/Appveyor/VSTS checks.

I have suggested that, and before that, the same for buildbots.  The 
reality is that tkinter, IDLE, or turtle could be disabled on *nix by 
regressions and the official testing would not notice.

-- 
Terry Jan Reedy