Hi,
We don't have a PR check for refleaks. I would like one.
The buildbots are far too slow and unreliable to provide a sensible way to check for refleaks. It is nigh impossible to pick out errors in a PR from flaky tests, flaky machines and pre-existing errors.
Stopping refleaks at source would be great. But we need a PR checks for refleaks.
Unfortunately, refleaks tests are currently very slow. By splitting the tests into 3 or 4 groups we should be able to run refleak tests without slowing turn around too much.
I would rather have to wait a while to merge good code, than merge bad code quickly. I don't think the cost would be prohibitive.
Cheers, Mark.
It is nigh impossible to pick out errors in a PR from flaky tests, flaky machines and pre-existing errors.
I agree with the sentiment but is certainly not that dramatic. We (people watching the build bots) do it on a regular basis, and many contributors do it very efficiently as well. I agree that flaky tests and machines do not help here, but those are generally just a small subset (although a noisy one). Pre-existing errors should be fixed, so one could argue that these errors being annoying is a positive thing under some optics.
I think the best approach for a CI check is to link every source file to a particular test and then use the diff to select what tests to run. Notice this is still not enough because some tests (like test_asyncio) will take a huge amount of time on their own in refleak mode.
Additionally, some files will virtually touch all tests (like changing the VM) and some refleaks will only be visible with an unknown subset of them.
I would rather have to wait a while to merge good code, than merge bad code
quickly.
With that logic, seems that waiting for buildbots is not bad ;)
On Fri, 28 Jan 2022 at 11:46, Mark Shannon mark@hotpy.org wrote:
Hi,
We don't have a PR check for refleaks. I would like one.
The buildbots are far too slow and unreliable to provide a sensible way to check for refleaks. It is nigh impossible to pick out errors in a PR from flaky tests, flaky machines and pre-existing errors.
Stopping refleaks at source would be great. But we need a PR checks for refleaks.
Unfortunately, refleaks tests are currently very slow. By splitting the tests into 3 or 4 groups we should be able to run refleak tests without slowing turn around too much.
I would rather have to wait a while to merge good code, than merge bad code quickly. I don't think the cost would be prohibitive.
Cheers, Mark.
python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/4... Code of Conduct: https://www.python.org/psf/codeofconduct/
Is there a possible middle ground here? Rather than a required PR check (or a full buildbot run), maybe we could just add a new “magic” label that runs a single refleak job using the GitHub actions runners.
Is there a possible middle ground here? Rather than a required PR check (or a full buildbot run), maybe we could just add a new “magic” label that runs a single refleak job using the GitHub actions runners.
That sounds like a good compromise and it will be strictly better than the current setup.
In addition, I still think that for the slower tests, the git diff based test run can be quite beneficial.
We just need to bear in mind that those tests will always be partial, so we should not be surprised if refleaks are still detected in buildbots.
On Fri, 28 Jan 2022 at 14:25, Brandt Bucher brandtbucher@gmail.com wrote:
Is there a possible middle ground here? Rather than a required PR check (or a full buildbot run), maybe we could just add a new “magic” label that runs a single refleak job using the GitHub actions runners.
python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/N... Code of Conduct: https://www.python.org/psf/codeofconduct/
participants (3)
-
Brandt Bucher
-
Mark Shannon
-
Pablo Galindo Salgado