Multiple reference leaks in master
Hi, Me (and Victor) have not been able to attend the buildbots for a while unfortunately and today I was checking them out after some fixes to the SSL tests and sadly the refleaks buildbots have many independent issues. At least these tests are failing due to reference leaks: test__xxsubinterpreters, test_atexit, test_capi, test_httpservers, test_threading I am trying to slowly fix them but finding them and debugging often takes several hours per leak. I was tracking them in https://bugs.python.org/issue38962 until I discover that there are multiple sources of leaks so I may open more issues. Please, when you merge a PR, take a look at the buildbots (especially the refleak buildbots that do not report still to the PR in case of failure) so the issues don't keep piling up, as multiple refleaks together are much difficult to deal with. Thanks, everyone! Regards from cloudy London, Pablo Galindo Salgado
How do I find the refleak buildbots? I went to the devguide and searched for "buildbot", which pointed to https://www.python.org/dev/buildbot/ -- but searching there for "refleak" finds nothing. On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado <pablogsal@gmail.com> wrote:
Hi,
Me (and Victor) have not been able to attend the buildbots for a while unfortunately and today I was checking them out after some fixes to the SSL tests and sadly the refleaks buildbots have many independent issues. At least these tests are failing due to reference leaks:
test__xxsubinterpreters, test_atexit, test_capi, test_httpservers, test_threading
I am trying to slowly fix them but finding them and debugging often takes several hours per leak. I was tracking them in https://bugs.python.org/issue38962 until I discover that there are multiple sources of leaks so I may open more issues.
Please, when you merge a PR, take a look at the buildbots (especially the refleak buildbots that do not report still to the PR in case of failure) so the issues don't keep piling up, as multiple refleaks together are much difficult to deal with.
Thanks, everyone!
Regards from cloudy London, Pablo Galindo Salgado
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZG... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
How do I find the refleak buildbots?
In this page: https://buildbot.python.org/all/#/builders all the buildbots that have "Refleaks" in the name. You can click on the tags to filter only the "stable" and the ones in master "3.x". On Tue, 3 Dec 2019 at 22:11, Guido van Rossum <guido@python.org> wrote:
How do I find the refleak buildbots? I went to the devguide and searched for "buildbot", which pointed to https://www.python.org/dev/buildbot/ -- but searching there for "refleak" finds nothing.
On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado <pablogsal@gmail.com> wrote:
Hi,
Me (and Victor) have not been able to attend the buildbots for a while unfortunately and today I was checking them out after some fixes to the SSL tests and sadly the refleaks buildbots have many independent issues. At least these tests are failing due to reference leaks:
test__xxsubinterpreters, test_atexit, test_capi, test_httpservers, test_threading
I am trying to slowly fix them but finding them and debugging often takes several hours per leak. I was tracking them in https://bugs.python.org/issue38962 until I discover that there are multiple sources of leaks so I may open more issues.
Please, when you merge a PR, take a look at the buildbots (especially the refleak buildbots that do not report still to the PR in case of failure) so the issues don't keep piling up, as multiple refleaks together are much difficult to deal with.
Thanks, everyone!
Regards from cloudy London, Pablo Galindo Salgado
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZG... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Maybe you or Victor can update the devguide with this information? This is not the first time that we've been asked to pay attention to the refleaks buildbots, and it's frustrating that they are so hard to find. On Tue, Dec 3, 2019 at 2:17 PM Pablo Galindo Salgado <pablogsal@gmail.com> wrote:
How do I find the refleak buildbots?
In this page:
https://buildbot.python.org/all/#/builders
all the buildbots that have "Refleaks" in the name. You can click on the tags to filter only the "stable" and the ones in master "3.x".
On Tue, 3 Dec 2019 at 22:11, Guido van Rossum <guido@python.org> wrote:
How do I find the refleak buildbots? I went to the devguide and searched for "buildbot", which pointed to https://www.python.org/dev/buildbot/ -- but searching there for "refleak" finds nothing.
On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado <pablogsal@gmail.com> wrote:
Hi,
Me (and Victor) have not been able to attend the buildbots for a while unfortunately and today I was checking them out after some fixes to the SSL tests and sadly the refleaks buildbots have many independent issues. At least these tests are failing due to reference leaks:
test__xxsubinterpreters, test_atexit, test_capi, test_httpservers, test_threading
I am trying to slowly fix them but finding them and debugging often takes several hours per leak. I was tracking them in https://bugs.python.org/issue38962 until I discover that there are multiple sources of leaks so I may open more issues.
Please, when you merge a PR, take a look at the buildbots (especially the refleak buildbots that do not report still to the PR in case of failure) so the issues don't keep piling up, as multiple refleaks together are much difficult to deal with.
Thanks, everyone!
Regards from cloudy London, Pablo Galindo Salgado
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZG... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Maybe you or Victor can update the devguide with this information? This is not the first time that we've been asked to pay attention to the refleaks buildbots, and it's frustrating that they are so hard to find.
Good point! I will make a PR to the dev guide improving the existing documentation and I will think of other ways to make discovery easier. On Wed, 4 Dec 2019 at 00:00, Guido van Rossum <guido@python.org> wrote:
Maybe you or Victor can update the devguide with this information? This is not the first time that we've been asked to pay attention to the refleaks buildbots, and it's frustrating that they are so hard to find.
On Tue, Dec 3, 2019 at 2:17 PM Pablo Galindo Salgado <pablogsal@gmail.com> wrote:
How do I find the refleak buildbots?
In this page:
https://buildbot.python.org/all/#/builders
all the buildbots that have "Refleaks" in the name. You can click on the tags to filter only the "stable" and the ones in master "3.x".
On Tue, 3 Dec 2019 at 22:11, Guido van Rossum <guido@python.org> wrote:
How do I find the refleak buildbots? I went to the devguide and searched for "buildbot", which pointed to https://www.python.org/dev/buildbot/ -- but searching there for "refleak" finds nothing.
On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado < pablogsal@gmail.com> wrote:
Hi,
Me (and Victor) have not been able to attend the buildbots for a while unfortunately and today I was checking them out after some fixes to the SSL tests and sadly the refleaks buildbots have many independent issues. At least these tests are failing due to reference leaks:
test__xxsubinterpreters, test_atexit, test_capi, test_httpservers, test_threading
I am trying to slowly fix them but finding them and debugging often takes several hours per leak. I was tracking them in https://bugs.python.org/issue38962 until I discover that there are multiple sources of leaks so I may open more issues.
Please, when you merge a PR, take a look at the buildbots (especially the refleak buildbots that do not report still to the PR in case of failure) so the issues don't keep piling up, as multiple refleaks together are much difficult to deal with.
Thanks, everyone!
Regards from cloudy London, Pablo Galindo Salgado
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZG... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
----- Original Message -----
From: "Pablo Galindo Salgado" <pablogsal@gmail.com> To: "Python Dev" <python-dev@python.org> Sent: Tuesday, December 3, 2019 10:14:01 PM Subject: [Python-Dev] Multiple reference leaks in master
Hi,
Me (and Victor) have not been able to attend the buildbots for a while unfortunately and today I was checking them out after some fixes to the SSL tests and sadly the refleaks buildbots have many independent issues. At least these tests are failing due to reference leaks:
test__xxsubinterpreters, test_atexit, test_capi, test_httpservers, test_threading
I am trying to slowly fix them but finding them and debugging often takes several hours per leak. I was tracking them in https://bugs.python.org/issue38962 until I discover that there are multiple sources of leaks so I may open more issues.
Please, when you merge a PR, take a look at the buildbots (especially the refleak buildbots that do not report still to the PR in case of failure) so the issues don't keep piling up, as multiple refleaks together are much difficult to deal with.
Thanks, everyone!
Regards from cloudy London, Pablo Galindo Salgado
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZG... Code of Conduct: http://python.org/psf/codeofconduct/
Also noting here that the reference leaks builds are triggered once a day so many times it's not possible to find the commit that triggered a leak through the web ui of buildbot. In this case it's good to also check the previous builds of the same worker to figure out what changed. -- Regards, Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat
Le mer. 4 déc. 2019 à 01:45, Charalampos Stratakis <cstratak@redhat.com> a écrit :
Also noting here that the reference leaks builds are triggered once a day so many times it's not possible to find the commit that triggered a leak through the web ui of buildbot. In this case it's good to also check the previous builds of the same worker to figure out what changed.
Once a day is a tradeoff to not stress buildbot workers too often. In my experience, it's quite easy to look at changes made the day before and use bisect tools to identify the leak. I'm using different tools: * Refleak buildbots tell you which tests leak * ./python -m test.bisect_cmd -R 3:3 test_xxx: identify one test method which is enough to trigger the leak in test_xxx. This speedup a lot following steps to check if a commit has the leak or not (git bisect), since -R 3:3 makes tests 6x slower. * git bisect helps a lot to identify which commit introduced the leak Overall, it's rare than it takes me longer than 30 minutes to identify the commit and the leaking tests and test methods. Sometimes, it's done in under 10 minutes. It's just a few simple command and wait until the computer works for me :-) (test.bisect_cmd that I wrote helps me a lot!) Pablo and me are opened issues to reference leak regressions. If possible, we comment the issue introducing the regression, or we open new issues if it's too hard to identify the leak. The worst is when the leak is not a recent regression, but it was hidden for whatever reasons and can now be reproduced in Python 6 months ago, if not older. But that's the least common case. Note: test.bisect_cmd can also be used to identify the test method responsible for a "altered environment" (ENV_CHANGED) if you add --fail-env-changed option. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
participants (4)
-
Charalampos Stratakis -
Guido van Rossum -
Pablo Galindo Salgado -
Victor Stinner