Re: [Python-Dev] We now have C code coverage!

On 6/22/2018 9:40 PM, Nathaniel Smith wrote:

On Fri, Jun 22, 2018 at 6:16 PM, Terry Reedy wrote:

...

I discovered the answer to 2. by shift-clicking on a text_x file to see their coverage report for the file. The colors actually do reflect the test lines executed. codecov.io excludes gui tests*, so the reported coverage for tkinter, idlelib, and turtle is deceptive and bogus, and under-reports the total cpython coverage by a percent or two. It would be better to exclude these modules.

...

* I assume that codecov.io uses linux servers. I have read that there are programs that simulate X-Windows so that gui code will execute without actual terminals.

Codecov.io doesn't run any tests itself; it's just a service for aggregation and reporting. The coverage information is being gathered while running CPython's regular CI tests, and then uploaded to codecov.io to view.

Thank you for the information.

So if you want to run the gui tests -- which seems like a good idea if possible! -- then the way to do that would be to make them run as part of the regular Travis/Appveyor/VSTS checks.

I have suggested that, and before that, the same for buildbots. The reality is that tkinter, IDLE, or turtle could be disabled on *nix by regressions and the official testing would not notice. -- Terry Jan Reedy