[code-quality] Make pylint faster by caching modules that passed without warnings

[Ned Batchelder]

I tried this on my large project (Open edX), and it generally worked 
great, but was one crash:

   File "../more/cached_pylint.py", line 252, in <module>
     sys.exit(main())
   File "../more/cached_pylint.py", line 248, in main
     run = CachedRun(sys.argv[1:], exit=False)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", 
line 1332, in __init__
     linter.check(args)
   File "../more/cached_pylint.py", line 137, in check
     PyLinter.check(self, files_or_modules)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", 
line 747, in check
     self._do_check(files_or_modules)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", 
line 869, in _do_check
     self.check_astroid_module(ast_node, walker, rawcheckers, tokencheckers)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", 
line 946, in check_astroid_module
     walker.walk(ast_node)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/utils.py", 
line 874, in walk
     self.walk(child)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/utils.py", 
line 871, in walk
     cb(astroid)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/checkers/variables.py", 
line 904, in visit_from
     self._check_module_attrs(node, module, name.split('.'))
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/checkers/variables.py", 
line 974, in _check_module_attrs
     module = next(module.getattr(name)[0].infer())
StopIteration

Any clue what might be causing that?

--Ned.

On 11/5/15 6:24 AM, Ned Batchelder wrote:
> On 11/5/15 5:21 AM, Noam Yorav-Raphael wrote:
>> Hi,
>>
>> Take a look at cached_pylint.py at 
>> https://gist.github.com/noamraph/c933b32deb8304ac7ccd
>>
>> It records module dependencies, and when a module passes with no 
>> warnings, it records it together with the sha1 of the contents of 
>> each of its dependencies, so when run again it could pass without 
>> checking it.
>>
>> Dependencies are recorded by using infer and by using ImportsChecker, 
>> so if module A imports something from module B that was actually 
>> defined in module C, both B and C will be recorded as dependencies of A.
>>
>> We have a medium sized code repository that took two minutes for each 
>> check. Now pylint only checks the modified modules, and it takes two 
>> seconds.
>>
> This sounds like a great feature.  Have you considered contributing it 
> as a pull request into pylint?
>
> --Ned.
>
>> I tested it with pylint 1.4.3.
>>
>> Hope this helps someone,
>>
>> Noam