On 6/22/2018 6:21 PM, Brett Cannon wrote:

Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io http://codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).

And funny enough the coverage average changed less than 1%. :)

Questions: 1. Is it possible, given that we are not paying for those reports, to customize the 'exclude_lines' definitions? Without such, the idlelib measures are biased downward. 2. What do the colors of test files mean? Every line of nearly all the idlelib test files are executed, but over half are red. The Learn More page does not say anything about either. -- Terry Jan Reedy

On Fri, Jun 22, 2018 at 6:16 PM, Terry Reedy wrote:

On 6/22/2018 8:43 PM, Terry Reedy wrote:

...

On 6/22/2018 6:21 PM, Brett Cannon wrote:

...

Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io http://codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).

And funny enough the coverage average changed less than 1%. :)

Questions:

1. Is it possible, given that we are not paying for those reports, to customize the .coveragerc exclude_lines definitions? Without such, the idlelib measures are biased downward.

2. What do the colors of test files mean? Every line of nearly all the idlelib test files are executed, but over half are red.

The Learn More page does not say anything about either.

I discovered the answer to 2. by shift-clicking on a text_x file to see their coverage report for the file. The colors actually do reflect the test lines executed. codecov.io excludes gui tests*, so the reported coverage for tkinter, idlelib, and turtle is deceptive and bogus, and under-reports the total cpython coverage by a percent or two. It would be better to exclude these modules.

* I assume that codecov.io uses linux servers. I have read that there are programs that simulate X-Windows so that gui code will execute without actual terminals.

Codecov.io doesn't run any tests itself; it's just a service for aggregation and reporting. The coverage information is being gathered while running CPython's regular CI tests, and then uploaded to codecov.io to view. So if you want to run the gui tests -- which seems like a good idea if possible! -- then the way to do that would be to make them run as part of the regular Travis/Appveyor/VSTS checks. -n -- Nathaniel J. Smith -- https://vorpus.org

On 6/22/2018 9:40 PM, Nathaniel Smith wrote:

On Fri, Jun 22, 2018 at 6:16 PM, Terry Reedy wrote:

...

I discovered the answer to 2. by shift-clicking on a text_x file to see their coverage report for the file. The colors actually do reflect the test lines executed. codecov.io excludes gui tests*, so the reported coverage for tkinter, idlelib, and turtle is deceptive and bogus, and under-reports the total cpython coverage by a percent or two. It would be better to exclude these modules.

...

* I assume that codecov.io uses linux servers. I have read that there are programs that simulate X-Windows so that gui code will execute without actual terminals.

Codecov.io doesn't run any tests itself; it's just a service for aggregation and reporting. The coverage information is being gathered while running CPython's regular CI tests, and then uploaded to codecov.io to view.

Thank you for the information.

So if you want to run the gui tests -- which seems like a good idea if possible! -- then the way to do that would be to make them run as part of the regular Travis/Appveyor/VSTS checks.

I have suggested that, and before that, the same for buildbots. The reality is that tkinter, IDLE, or turtle could be disabled on *nix by regressions and the official testing would not notice. -- Terry Jan Reedy

On 23.06.2018 13:52, Paul Moore wrote:

On 22 June 2018 at 23:21, Brett Cannon wrote:

...

Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).

And funny enough the coverage average changed less than 1%. :) Nice!

One thing I noticed, code that's Windows-specific isn't covered. I assume that's because the coverage reports are based on runs of the test suite on Linux. Is it possible to merge in data from the Windows test runs? If not, what's the best way to address this? Should we be mocking things to attempt to test Windows-specific code even on Linux, or should we simply accept that we're not going to achieve 100% coverage and not worry about it?

AFAICS lcov is based on gcov which is GCC-specific.

Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- Regards, Ivan

On Sat, Jun 23, 2018 at 2:20 PM, Terry Reedy wrote:

Rechecking now, on Gentoo

test_idle appears and passed on these 3.6 and 3.7 pages http://buildbot.python.org/all/#/builders/82/builds/414/steps/5/logs/stdio

Neither Firefox nor Edge can find 'test_idle' on these 3.x pages http://buildbot.python.org/all/#/builders/103/builds/1149/steps/5/logs/stdio http://buildbot.python.org/all/#/builders/99/builds/1130/steps/4/logs/stdio

test_tk appears on 1130 but not 1149

For your 8.1 bot: test_idle passed for 3.7 http://buildbot.python.org/all/#/builders/133/builds/339/steps/3/logs/stdio

but does is not found on this 3.x page (neither is 'test_tk') http://buildbot.python.org/all/#/builders/12/builds/991/steps/3/logs/stdio

Click the magnifying glass icon ("load all data for use with the browser search tool") at the upper right of the console pane and try again; each of the above is present and passed. This does unfortunately seem to be another case of non-intuitive UI from buildbot.

Both Appveyor and my machine run test_idle when running the full 3.x test suite.

I am pleasantly surprised that AppVeyor does appear to be running the GUI tests :) -- Zach

This is awesome, thank you Ammar! On 23 June 2018 at 06:21, Brett Cannon wrote:

Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).

And funny enough the coverage average changed less than 1%. :)

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ dimaqq%40gmail.com