Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).
And funny enough the coverage average changed less than 1%. :)
On 6/22/2018 6:21 PM, Brett Cannon wrote:
Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io http://codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).
And funny enough the coverage average changed less than 1%. :)
Questions:
1. Is it possible, given that we are not paying for those reports, to customize the 'exclude_lines' definitions? Without such, the idlelib measures are biased downward.
2. What do the colors of test files mean? Every line of nearly all the idlelib test files are executed, but over half are red.
The Learn More page does not say anything about either.
On 6/22/2018 8:43 PM, Terry Reedy wrote:
On 6/22/2018 6:21 PM, Brett Cannon wrote:
Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io http://codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).
And funny enough the coverage average changed less than 1%. :)
Questions:
- Is it possible, given that we are not paying for those reports, to
customize the .coveragerc exclude_lines definitions? Without such, the idlelib measures are biased downward.
- What do the colors of test files mean? Every line of nearly all the
idlelib test files are executed, but over half are red.
The Learn More page does not say anything about either.
I discovered the answer to 2. by shift-clicking on a text_x file to see their coverage report for the file. The colors actually do reflect the test lines executed. codecov.io excludes gui tests*, so the reported coverage for tkinter, idlelib, and turtle is deceptive and bogus, and under-reports the total cpython coverage by a percent or two. It would be better to exclude these modules.
* I assume that codecov.io uses linux servers. I have read that there are programs that simulate X-Windows so that gui code will execute without actual terminals.
On Fri, Jun 22, 2018 at 6:16 PM, Terry Reedy tjreedy@udel.edu wrote:
On 6/22/2018 8:43 PM, Terry Reedy wrote:
On 6/22/2018 6:21 PM, Brett Cannon wrote:
Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io http://codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).
And funny enough the coverage average changed less than 1%. :)
Questions:
- Is it possible, given that we are not paying for those reports, to
customize the .coveragerc exclude_lines definitions? Without such, the idlelib measures are biased downward.
- What do the colors of test files mean? Every line of nearly all the
idlelib test files are executed, but over half are red.
The Learn More page does not say anything about either.
I discovered the answer to 2. by shift-clicking on a text_x file to see their coverage report for the file. The colors actually do reflect the test lines executed. codecov.io excludes gui tests*, so the reported coverage for tkinter, idlelib, and turtle is deceptive and bogus, and under-reports the total cpython coverage by a percent or two. It would be better to exclude these modules.
- I assume that codecov.io uses linux servers. I have read that there are
programs that simulate X-Windows so that gui code will execute without actual terminals.
Codecov.io doesn't run any tests itself; it's just a service for aggregation and reporting. The coverage information is being gathered while running CPython's regular CI tests, and then uploaded to codecov.io to view.
So if you want to run the gui tests -- which seems like a good idea if possible! -- then the way to do that would be to make them run as part of the regular Travis/Appveyor/VSTS checks.
-n
On 6/22/2018 9:40 PM, Nathaniel Smith wrote:
On Fri, Jun 22, 2018 at 6:16 PM, Terry Reedy tjreedy@udel.edu wrote:
I discovered the answer to 2. by shift-clicking on a text_x file to see their coverage report for the file. The colors actually do reflect the test lines executed. codecov.io excludes gui tests*, so the reported coverage for tkinter, idlelib, and turtle is deceptive and bogus, and under-reports the total cpython coverage by a percent or two. It would be better to exclude these modules.
- I assume that codecov.io uses linux servers. I have read that there are
programs that simulate X-Windows so that gui code will execute without actual terminals.
Codecov.io doesn't run any tests itself; it's just a service for aggregation and reporting. The coverage information is being gathered while running CPython's regular CI tests, and then uploaded to codecov.io to view.
Thank you for the information.
So if you want to run the gui tests -- which seems like a good idea if possible! -- then the way to do that would be to make them run as part of the regular Travis/Appveyor/VSTS checks.
I have suggested that, and before that, the same for buildbots. The reality is that tkinter, IDLE, or turtle could be disabled on *nix by regressions and the official testing would not notice.
On Sat, Jun 23, 2018 at 11:31 AM, Terry Reedy tjreedy@udel.edu wrote:
I have suggested that, and before that, the same for buildbots. The reality is that tkinter, IDLE, or turtle could be disabled on *nix by regressions and the official testing would not notice.
I'm looking into enabling the GUI tests on some of the CI hosts today, not sure how far I'll make it with that. GUI tests are currently enabled on my Gentoo [1] and Windows [2] builders, and have been for a couple of years at this point; I'm not sure if any other builders are running GUI tests.
[1] http://buildbot.python.org/all/#/workers/6 [2] http://buildbot.python.org/all/#/workers/11
On 6/23/2018 1:09 PM, Zachary Ware wrote:
On Sat, Jun 23, 2018 at 11:31 AM, Terry Reedy tjreedy@udel.edu wrote:
I have suggested that, and before that, the same for buildbots. The reality is that tkinter, IDLE, or turtle could be disabled on *nix by regressions and the official testing would not notice.
I'm looking into enabling the GUI tests on some of the CI hosts today, not sure how far I'll make it with that. GUI tests are currently enabled on my Gentoo [1] and Windows [2] builders, and have been for a couple of years at this point; I'm not sure if any other builders are running GUI tests.
I noticed your Gentoo builders running gui tests some years ago. When I re-checked perhaps a year ago, they either were not running or seem to have skipped test_idle, maybe the former.
[1] http://buildbot.python.org/all/#/workers/6 [2] http://buildbot.python.org/all/#/workers/11
Rechecking now, on Gentoo
test_idle appears and passed on these 3.6 and 3.7 pages http://buildbot.python.org/all/#/builders/82/builds/414/steps/5/logs/stdio
Neither Firefox nor Edge can find 'test_idle' on these 3.x pages http://buildbot.python.org/all/#/builders/103/builds/1149/steps/5/logs/stdio http://buildbot.python.org/all/#/builders/99/builds/1130/steps/4/logs/stdio
test_tk appears on 1130 but not 1149
For your 8.1 bot: test_idle passed for 3.7 http://buildbot.python.org/all/#/builders/133/builds/339/steps/3/logs/stdio
but does is not found on this 3.x page (neither is 'test_tk') http://buildbot.python.org/all/#/builders/12/builds/991/steps/3/logs/stdio
Both Appveyor and my machine run test_idle when running the full 3.x test suite.
On Sat, Jun 23, 2018 at 2:20 PM, Terry Reedy tjreedy@udel.edu wrote:
Rechecking now, on Gentoo
test_idle appears and passed on these 3.6 and 3.7 pages http://buildbot.python.org/all/#/builders/82/builds/414/steps/5/logs/stdio
Neither Firefox nor Edge can find 'test_idle' on these 3.x pages http://buildbot.python.org/all/#/builders/103/builds/1149/steps/5/logs/stdio http://buildbot.python.org/all/#/builders/99/builds/1130/steps/4/logs/stdio
test_tk appears on 1130 but not 1149
For your 8.1 bot: test_idle passed for 3.7 http://buildbot.python.org/all/#/builders/133/builds/339/steps/3/logs/stdio
but does is not found on this 3.x page (neither is 'test_tk') http://buildbot.python.org/all/#/builders/12/builds/991/steps/3/logs/stdio
Click the magnifying glass icon ("load all data for use with the browser search tool") at the upper right of the console pane and try again; each of the above is present and passed. This does unfortunately seem to be another case of non-intuitive UI from buildbot.
Both Appveyor and my machine run test_idle when running the full 3.x test suite.
I am pleasantly surprised that AppVeyor does appear to be running the GUI tests :)
On 6/23/2018 5:48 PM, Zachary Ware wrote:
On Sat, Jun 23, 2018 at 2:20 PM, Terry Reedy tjreedy@udel.edu wrote:
Rechecking now, on Gentoo
test_idle appears and passed on these 3.6 and 3.7 pages http://buildbot.python.org/all/#/builders/82/builds/414/steps/5/logs/stdio
Neither Firefox nor Edge can find 'test_idle' on these 3.x pages http://buildbot.python.org/all/#/builders/103/builds/1149/steps/5/logs/stdio http://buildbot.python.org/all/#/builders/99/builds/1130/steps/4/logs/stdio
test_tk appears on 1130 but not 1149
For your 8.1 bot: test_idle passed for 3.7 http://buildbot.python.org/all/#/builders/133/builds/339/steps/3/logs/stdio
but does is not found on this 3.x page (neither is 'test_tk') http://buildbot.python.org/all/#/builders/12/builds/991/steps/3/logs/stdio
Click the magnifying glass icon ("load all data for use with the browser search tool") at the upper right of the console pane and try again; each of the above is present and passed. This does unfortunately seem to be another case of non-intuitive UI from buildbot.
Presenting data on the screen that cannot be found is terrible. With Edge, I had to erase and retype the search word also. With Firefox, that sometimes worked without pressing the magnifier. I thought my copy of Firefox was broken until I tried Edge also.
Both Appveyor and my machine run test_idle when running the full 3.x test suite.
I am pleasantly surprised that AppVeyor does appear to be running the GUI tests :)
On 22 June 2018 at 23:21, Brett Cannon brett@python.org wrote:
Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).
And funny enough the coverage average changed less than 1%. :)
Nice!
One thing I noticed, code that's Windows-specific isn't covered. I assume that's because the coverage reports are based on runs of the test suite on Linux. Is it possible to merge in data from the Windows test runs? If not, what's the best way to address this? Should we be mocking things to attempt to test Windows-specific code even on Linux, or should we simply accept that we're not going to achieve 100% coverage and not worry about it?
Paul
On 23.06.2018 13:52, Paul Moore wrote:
On 22 June 2018 at 23:21, Brett Cannon brett@python.org wrote:
Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).
And funny enough the coverage average changed less than 1%. :)
Nice!
One thing I noticed, code that's Windows-specific isn't covered. I assume that's because the coverage reports are based on runs of the test suite on Linux. Is it possible to merge in data from the Windows test runs? If not, what's the best way to address this? Should we be mocking things to attempt to test Windows-specific code even on Linux, or should we simply accept that we're not going to achieve 100% coverage and not worry about it?
AFAICS lcov is based on gcov which is GCC-specific.
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru
This is awesome, thank you Ammar!
On 23 June 2018 at 06:21, Brett Cannon brett@python.org wrote:
Thanks to a PR from Ammar Askar we now run Python under lcov as part of the code coverage build. And thanks to codecov.io automatically merging code coverage reports we get a complete report of our coverage (the first results of which can now be seen at https://codecov.io/gh/python/cpython).
And funny enough the coverage average changed less than 1%. :)
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ dimaqq%40gmail.com
participants (7)
-
Brett Cannon -
Dima Tisnek -
Ivan Pozdeev -
Nathaniel Smith -
Paul Moore -
Terry Reedy -
Zachary Ware