Need help with strange test failure for pylint-django
Hi folks, I need help debugging a strange test failure in pylint-django that started happening after the upgrade to pylint/astroid 2.0. For all I can tell we've already adjusted the code and this failure is happening randomly. pylint-django's test used uses the test_functional.LintModuleTest class from pylint (I know it isn't public interface) and executes against input files comparing the linter results with expected. The error that we get is: E Failed: Wrong results for file "func_noerror_foreignkeys": E E Unexpected in testdata: E 27: no-member The strange thing is that on one test job in Travis this passed with Python 3.4 and Django 2.0 (see job #447.3): https://travis-ci.org/PyCQA/pylint-django/builds/407129036 and on the next commit this failed on Python 3.4/Django 2.0 (see #448.3): https://travis-ci.org/PyCQA/pylint-django/builds/407800899 however it passed on 3.5 + Django 2.0. The two test jobs use the same code (minor differences in commits due to squash/conflicts). When the linter is executed manually against the offending file it produces a 10/10 score. If I modify the test suite like so: --- a/pylint_django/tests/test_func.py +++ b/pylint_django/tests/test_func.py @@ -43,6 +43,8 @@ def get_tests(input_dir='input', sort=False): suite = [] for fname in os.listdir(input_dir): + if fname.find('func_noerror_foreign') == -1: + continue if fname != '__init__.py' and fname.endswith('.py'): suite.append(test_functional.FunctionalTestFile(input_dir, fname)) then I get only a subset of the tests executed (including the offending one) and this time it passes. Anyone seen such erratic behaviour ? I think this is a flaky test but I have no idea how to approach debugging that. For the record this is blocking us from releasing a new version of pylint-django so any help is appreciated. -- Alex
Update for those interested, the solution is here with explanation: https://github.com/PyCQA/pylint-django/pull/183/commits/538546044ac5a9785dc8... Cheers, James On Tue, 24 Jul 2018, at 10:23 PM, Alexander Todorov wrote:
Hi folks, I need help debugging a strange test failure in pylint-django that started happening after the upgrade to pylint/astroid 2.0.
For all I can tell we've already adjusted the code and this failure is happening randomly. pylint-django's test used uses the test_functional.LintModuleTest class from pylint (I know it isn't public interface) and executes against input files comparing the linter results with expected.
The error that we get is:
E Failed: Wrong results for file "func_noerror_foreignkeys": E E Unexpected in testdata: E 27: no-member
The strange thing is that on one test job in Travis this passed with Python 3.4 and Django 2.0 (see job #447.3): https://travis-ci.org/PyCQA/pylint-django/builds/407129036
and on the next commit this failed on Python 3.4/Django 2.0 (see #448.3): https://travis-ci.org/PyCQA/pylint-django/builds/407800899
however it passed on 3.5 + Django 2.0.
The two test jobs use the same code (minor differences in commits due to squash/conflicts).
When the linter is executed manually against the offending file it produces a 10/10 score.
If I modify the test suite like so:
--- a/pylint_django/tests/test_func.py +++ b/pylint_django/tests/test_func.py @@ -43,6 +43,8 @@ def get_tests(input_dir='input', sort=False):
suite = [] for fname in os.listdir(input_dir): + if fname.find('func_noerror_foreign') == -1: + continue if fname != '__init__.py' and fname.endswith('.py'): suite.append(test_functional.FunctionalTestFile(input_dir, fname))
then I get only a subset of the tests executed (including the offending one) and this time it passes.
Anyone seen such erratic behaviour ? I think this is a flaky test but I have no idea how to approach debugging that.
For the record this is blocking us from releasing a new version of pylint-django so any help is appreciated.
-- Alex
_______________________________________________ code-quality mailing list code-quality@python.org https://mail.python.org/mailman/listinfo/code-quality
participants (2)
-
Alexander Todorov -
James Cooke