Buildbot report (almost July)
Hi, Our buildbots are now even more stable than in my previous buildbot report. Many random failures have been fixed, even if there are still many rare random failures (most of them are related to multiprocessing). Search for issues created by "haypo" (me!) with the Component "Tests" to see all remaining issues. I'm trying to open an issue for *each* buildbot failure! So yeah, we have a long list of 44 open issues! Correct me if I'm wrong, but, for the first time, *all reference leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master), on *Linux and Windows*! Before, we mostly focused on the master branch (called "default" in Mercurial) on Linux. I also started to fix a few "memory block" leaks, most (or all?) of them should also be fixed (on all branches, on Linux and Windows). A new policy has been decided (on python-committers): if a commit breaks too many buildbots and cannot fixed easily nor quickly, the commit will be reverted, just to keep our buildbots "green". It doesn't mean that the commit will be rejected forever. It's a practical solution to give more time to write a proper fix, take time to review it, and not have to be stressed to fix buildbots "ASAP". Moreover, keeping green buildbots all the time allows to catch regressions more quickly, which ease debug in many ways. You have be warned! Now I will not hesitate to revert your change if you break my little buildbots ;-) I mostly care of Linux, Windows, macOS and FreeBSD (10 and CURRENT). Breaking other platforms is less important, since other platforms already have issues, but not enough developers interested to fix them. Obviously, I'm interested in keeping more buildbots green, if someone shows up and help me to fix remaining issues! Victor
On 6/29/2017 11:09 AM, Victor Stinner wrote:
Hi,
Our buildbots are now even more stable than in my previous buildbot report.
Many random failures have been fixed, even if there are still many rare random failures (most of them are related to multiprocessing).
Search for issues created by "haypo" (me!) with the Component "Tests" to see all remaining issues. I'm trying to open an issue for *each* buildbot failure! So yeah, we have a long list of 44 open issues!
Correct me if I'm wrong, but, for the first time, *all reference leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master), on *Linux and Windows*! Before, we mostly focused on the master branch (called "default" in Mercurial) on Linux.
I also started to fix a few "memory block" leaks, most (or all?) of them should also be fixed (on all branches, on Linux and Windows).
A new policy has been decided (on python-committers): if a commit breaks too many buildbots and cannot fixed easily nor quickly, the commit will be reverted, just to keep our buildbots "green". It doesn't mean that the commit will be rejected forever. It's a practical solution to give more time to write a proper fix, take time to review it, and not have to be stressed to fix buildbots "ASAP". Moreover, keeping green buildbots all the time allows to catch regressions more quickly, which ease debug in many ways.
You have be warned! Now I will not hesitate to revert your change if you break my little buildbots ;-)
I mostly care of Linux, Windows, macOS and FreeBSD (10 and CURRENT). Breaking other platforms is less important, since other platforms already have issues, but not enough developers interested to fix them. Obviously, I'm interested in keeping more buildbots green, if someone shows up and help me to fix remaining issues!
Great work. How about compiler warnings (and errors)? When I compile on Windows, there are a boatload of orange-yellow warnings. Some are about using a deprecated featured; some about dodgy casts; some (I presume) about other things. Should 'no warnings' be a goal? I believe you once fixed some, but new commits added more. Could 'no additional warnings' be a CI requirement for merging? I expect that a new "How to avoid compiler warnings on Windows" section of the devguide would be needed. -- Terry Jan Reedy
On 29Jun2017 1013, Terry Reedy wrote:
Great work.
Agreed. Thanks, Victor!
How about compiler warnings (and errors)? When I compile on Windows, there are a boatload of orange-yellow warnings. Some are about using a deprecated featured; some about dodgy casts; some (I presume) about other things. Should 'no warnings' be a goal?
Yes, I think that's a good goal. We're quickly getting there as well - I just merged two contributions yesterday that should significantly reduce the number of warnings.
I believe you once fixed some, but new commits added more. Could 'no additional warnings' be a CI requirement for merging? I expect that a new "How to avoid compiler warnings on Windows" section of the devguide would be needed.
This is probably not feasible without a *really good* section in the devguide. I would rather have warnings in the output than global suppressions, and suppressing warnings globally is the usual instinct. Locally suppressing warnings can be fairly hideous and is usually not portable. Some warnings are also complicated because of the nature of CPython. For example, the socket module exposes deprecated CRT functions (on Windows) directly because the API of the socket module promises to provide the function directly. Changing to the safer function would break the API guarantee (except sometimes it won't... hence "complicated"). Noting in PR builds that there are new warnings would be great if possible. I'd be concerned about it becoming a hard requirement though - I much prefer to leave the final decision in the hands of trusted people and provide them enough information to make a good decision. Cheers, Steve
On 6/29/2017 1:54 PM, Steve Dower wrote:
On 29Jun2017 1013, Terry Reedy wrote:
Great work.
Agreed. Thanks, Victor!
How about compiler warnings (and errors)? When I compile on Windows, there are a boatload of orange-yellow warnings. Some are about using a deprecated featured; some about dodgy casts; some (I presume) about other things. Should 'no warnings' be a goal?
Yes, I think that's a good goal. We're quickly getting there as well - I just merged two contributions yesterday that should significantly reduce the number of warnings.
Great. I recompiled changes for the last day or two (about 100 c files) and see none. _socket aside, I presume some of those files were the ones you fixed.
I believe you once fixed some, but new commits added more. Could 'no additional warnings' be a CI requirement for merging? I expect that a new "How to avoid compiler warnings on Windows" section of the devguide would be needed.
This is probably not feasible without a *really good* section in the devguide. I would rather have warnings in the output than global suppressions, and suppressing warnings globally is the usual instinct. Locally suppressing warnings can be fairly hideous and is usually not portable.
Some warnings are also complicated because of the nature of CPython. For example, the socket module exposes deprecated CRT functions (on Windows) directly because the API of the socket module promises to provide the function directly. Changing to the safer function would break the API guarantee (except sometimes it won't... hence "complicated").
Thanks for the explanation. Does 'deprecated' mean future removal in ms-speak? Perhaps after the compile, build.bat should print something like [Ignore any socket deprecation warnings]
Noting in PR builds that there are new warnings would be great if possible. I'd be concerned about it becoming a hard requirement though - I much prefer to leave the final decision in the hands of trusted people and provide them enough information to make a good decision.
A robot that could scan the appveyor compile log, compare with a 'current warnings' set, and post a comment requesting a fix and possible help sources would be better than leaving them unseen, which I expect is the current situation. -- Terry Jan Reedy
On 29Jun2017 1157, Terry Reedy wrote:
On 6/29/2017 1:54 PM, Steve Dower wrote:
Some warnings are also complicated because of the nature of CPython. For example, the socket module exposes deprecated CRT functions (on Windows) directly because the API of the socket module promises to provide the function directly. Changing to the safer function would break the API guarantee (except sometimes it won't... hence "complicated").
Thanks for the explanation. Does 'deprecated' mean future removal in ms-speak? Perhaps after the compile, build.bat should print something like [Ignore any socket deprecation warnings]
Probably not future removal, considering the vast quantities of legacy code that we don't want to break, but renaming is possible (generally for standards compliance, such as the timezone->_timezone rename) and new platforms (e.g. ARM) are not guaranteed to have the APIs when there is essentially no existing code to worry about. In many (most?) cases, deprecations in MS libraries indicate that there are known security issues either inherently in the function itself or with many/most uses of it, and there is a better option (e.g. strcpy is deprecated in favour of strcpy_s, which requires/infers the length of the destination buffer and ensures you never overflow). I'm still in favour of local suppression over ignoring warnings - we just need to promote the macro at https://github.com/python/cpython/blob/master/Modules/socketmodule.c#L510 to somewhere global to make it consistent (and maybe extend it to handle other compilers where it matters). But this is why I'm not against the general idea, but it requires that the devguide entry be well written. Let's clean up all the warnings, but in the right way, not the way people use when faced with arbitrary rules. Cheers, Steve
2017-06-29 19:13 GMT+02:00 Terry Reedy <tjreedy@udel.edu>:
How about compiler warnings (and errors)?
That would be nice to have, but I don't have the bandwidth to handle this goal.
When I compile on Windows, there are a boatload of orange-yellow warnings. Some are about using a deprecated featured; some about dodgy casts; some (I presume) about other things. Should 'no warnings' be a goal?
Last years, I tried to fix a few integer downcast warnings on Windows 64-bit, but they are still a few ones. It's not easy to distinguish warnings from real bugs (integer overflow). There are a few open issues about Windows 64-bit warnings. Segev Finer just wrote an interesting but big PR: https://github.com/python/cpython/pull/2492 The deprecated warnings is another interesting topic :-)
I believe you once fixed some, but new commits added more. Could 'no additional warnings' be a CI requirement for merging? I expect that a new "How to avoid compiler warnings on Windows" section of the devguide would be needed.
My policy is more to start by fixing all warnings, before starting to check for regressions. I began by fixing the most common test failures, then references leaks, and now memory leaks. So it becomes much simpler to check for regressions on these checks. Victor
Hello, I¹m new to the list, and contributing to Python specifically, and I¹m interested in getting master and 3.6 branches building and working ³better² on UNIX. I¹ve been looking at a problem building 3.6 on HP-UX and see a PR was merged into master, https://github.com/python/cpython/pull/1351 and I¹d like to see it applied to 3.6. I¹m happy to create a PR with a cherry-picked commit, and/or test. Rob Boehne Senior Software Architect | Datalogics, Inc. +1.312.853.8351 <tel:3128538399> | robb@datalogics.com <mailto:jonw@datalogics.com> datalogics.com <https://www.datalogics.com/> | blogs.datalogics.com <https://blogs.datalogics.com/>Connect with us: Facebook <https://www.facebook.com/DatalogicsInc> | Twitter <https://twitter.com/DatalogicsInc> | LinkedIn <https://www.linkedin.com/company/datalogics> | YouTube <https://www.youtube.com/user/Datalogics> PROUDLY CELEBRATING 50 YEARS! <https://www.datalogics.com/datalogics-celebrates-50-years-of-embracing-cha nge-and-providing-high-quality-software> On 6/29/17, 10:09 AM, "Python-Dev on behalf of Victor Stinner" <python-dev-bounces+robb=datalogics.com@python.org on behalf of victor.stinner@gmail.com> wrote:
Hi,
Our buildbots are now even more stable than in my previous buildbot report.
Many random failures have been fixed, even if there are still many rare random failures (most of them are related to multiprocessing).
Search for issues created by "haypo" (me!) with the Component "Tests" to see all remaining issues. I'm trying to open an issue for *each* buildbot failure! So yeah, we have a long list of 44 open issues!
Correct me if I'm wrong, but, for the first time, *all reference leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master), on *Linux and Windows*! Before, we mostly focused on the master branch (called "default" in Mercurial) on Linux.
I also started to fix a few "memory block" leaks, most (or all?) of them should also be fixed (on all branches, on Linux and Windows).
A new policy has been decided (on python-committers): if a commit breaks too many buildbots and cannot fixed easily nor quickly, the commit will be reverted, just to keep our buildbots "green". It doesn't mean that the commit will be rejected forever. It's a practical solution to give more time to write a proper fix, take time to review it, and not have to be stressed to fix buildbots "ASAP". Moreover, keeping green buildbots all the time allows to catch regressions more quickly, which ease debug in many ways.
You have be warned! Now I will not hesitate to revert your change if you break my little buildbots ;-)
I mostly care of Linux, Windows, macOS and FreeBSD (10 and CURRENT). Breaking other platforms is less important, since other platforms already have issues, but not enough developers interested to fix them. Obviously, I'm interested in keeping more buildbots green, if someone shows up and help me to fix remaining issues!
Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/robb%40datalogics.com
On 6/29/2017 5:34 PM, Rob Boehne wrote:
Hello,
I¹m new to the list, and contributing to Python specifically, and I¹m interested in getting master and 3.6 branches building and working ³better² on UNIX. I¹ve been looking at a problem building 3.6 on HP-UX and see a PR was merged into master, https://github.com/python/cpython/pull/1351 and
The issue is https://bugs.python.org/issue30183 3.6 and 3.5 are marked as possible backport targets. I¹d
like to see it applied to 3.6. I¹m happy to create a PR with a cherry-picked commit, and/or test.
Ask on the issue or PR whether David of Victor are planning to do a backport or if they know some reason why it would be difficult, and repeat you offer to try it. -- Terry Jan Reedy
2017-06-29 17:09 GMT+02:00 Victor Stinner <victor.stinner@gmail.com>:
Correct me if I'm wrong, but, for the first time, *all reference leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master), on *Linux and Windows*! Before, we mostly focused on the master branch (called "default" in Mercurial) on Linux.
I also started to fix a few "memory block" leaks, most (or all?) of them should also be fixed (on all branches, on Linux and Windows).
The "AMD64 Windows8.1 Refleaks" now pass on 2.7, 3.5, 3.6 and master branches. I finished to backport my changes to fix false alarms in hunting reference and memory leaks. The next introduced reference leak should now send an email to the buildbot-status mailing list ;-) Victor
participants (4)
-
Rob Boehne -
Steve Dower -
Terry Reedy -
Victor Stinner