Hi folks, I need help debugging a strange test failure in pylint-django that started happening after the upgrade to pylint/astroid 2.0. For all I can tell we've already adjusted the code and this failure is happening randomly. pylint-django's test used uses the test_functional.LintModuleTest class from pylint (I know it isn't public interface) and executes against input files comparing the linter results with expected. The error that we get is: E Failed: Wrong results for file "func_noerror_foreignkeys": E E Unexpected in testdata: E 27: no-member The strange thing is that on one test job in Travis this passed with Python 3.4 and Django 2.0 (see job #447.3): https://travis-ci.org/PyCQA/pylint-django/builds/407129036 and on the next commit this failed on Python 3.4/Django 2.0 (see #448.3): https://travis-ci.org/PyCQA/pylint-django/builds/407800899 however it passed on 3.5 + Django 2.0. The two test jobs use the same code (minor differences in commits due to squash/conflicts). When the linter is executed manually against the offending file it produces a 10/10 score. If I modify the test suite like so: --- a/pylint_django/tests/test_func.py +++ b/pylint_django/tests/test_func.py @@ -43,6 +43,8 @@ def get_tests(input_dir='input', sort=False): suite = [] for fname in os.listdir(input_dir): + if fname.find('func_noerror_foreign') == -1: + continue if fname != '__init__.py' and fname.endswith('.py'): suite.append(test_functional.FunctionalTestFile(input_dir, fname)) then I get only a subset of the tests executed (including the offending one) and this time it passes. Anyone seen such erratic behaviour ? I think this is a flaky test but I have no idea how to approach debugging that. For the record this is blocking us from releasing a new version of pylint-django so any help is appreciated. -- Alex