It is nigh impossible to pick out errors in a PR from flaky tests, flaky machines and pre-existing errors.
I agree with the sentiment but is certainly not that dramatic. We (people watching the build bots) do it on a regular basis, and many contributors do it very efficiently as well. I agree that flaky tests and machines do not help here, but those are generally just a small subset (although a noisy one). Pre-existing errors should be fixed, so one could argue that these errors being annoying is a positive thing under some optics.
I think the best approach for a CI check is to link every source file to a particular test and then use the diff to select what tests to run. Notice this is still not enough because some tests (like test_asyncio) will take a huge amount of time on their own in refleak mode.
Additionally, some files will virtually touch all tests (like changing the VM) and some refleaks will only be visible with an unknown subset of them.
I would rather have to wait a while to merge good code, than merge bad code
quickly.
With that logic, seems that waiting for buildbots is not bad ;)
On Fri, 28 Jan 2022 at 11:46, Mark Shannon mark@hotpy.org wrote:
Hi,
We don't have a PR check for refleaks. I would like one.
The buildbots are far too slow and unreliable to provide a sensible way to check for refleaks. It is nigh impossible to pick out errors in a PR from flaky tests, flaky machines and pre-existing errors.
Stopping refleaks at source would be great. But we need a PR checks for refleaks.
Unfortunately, refleaks tests are currently very slow. By splitting the tests into 3 or 4 groups we should be able to run refleak tests without slowing turn around too much.
I would rather have to wait a while to merge good code, than merge bad code quickly. I don't think the cost would be prohibitive.
Cheers, Mark.
python-committers mailing list -- python-committers@python.org To unsubscribe send an email to python-committers-leave@python.org https://mail.python.org/mailman3/lists/python-committers.python.org/ Message archived at https://mail.python.org/archives/list/python-committers@python.org/message/4... Code of Conduct: https://www.python.org/psf/codeofconduct/