Mailman 3 Make pylint faster by caching modules that passed without warnings - code-quality

Make pylint faster by caching modules that passed without warnings

Noam Yorav-Raphael

5 Nov 2015 5 Nov '15

3:51 p.m.

Hi, Take a look at cached_pylint.py at https://gist.github.com/noamraph/c933b32deb8304ac7ccd It records module dependencies, and when a module passes with no warnings, it records it together with the sha1 of the contents of each of its dependencies, so when run again it could pass without checking it. Dependencies are recorded by using infer and by using ImportsChecker, so if module A imports something from module B that was actually defined in module C, both B and C will be recorded as dependencies of A. We have a medium sized code repository that took two minutes for each check. Now pylint only checks the modified modules, and it takes two seconds. I tested it with pylint 1.4.3. Hope this helps someone, Noam

Attachments:

attachment.htm (text/html — 984 bytes)

Show replies by thread

Ned Batchelder

5 Nov 5 Nov

4:54 p.m.

On 11/5/15 5:21 AM, Noam Yorav-Raphael wrote:

...

Hi,

Take a look at cached_pylint.py at https://gist.github.com/noamraph/c933b32deb8304ac7ccd

It records module dependencies, and when a module passes with no warnings, it records it together with the sha1 of the contents of each of its dependencies, so when run again it could pass without checking it.

Dependencies are recorded by using infer and by using ImportsChecker, so if module A imports something from module B that was actually defined in module C, both B and C will be recorded as dependencies of A.

We have a medium sized code repository that took two minutes for each check. Now pylint only checks the modified modules, and it takes two seconds.

This sounds like a great feature. Have you considered contributing it as a pull request into pylint? --Ned.

...

I tested it with pylint 1.4.3.

Hope this helps someone,

Noam

Ned Batchelder

9:25 p.m.

I tried this on my large project (Open edX), and it generally worked great, but was one crash: File "../more/cached_pylint.py", line 252, in <module> sys.exit(main()) File "../more/cached_pylint.py", line 248, in main run = CachedRun(sys.argv[1:], exit=False) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", line 1332, in __init__ linter.check(args) File "../more/cached_pylint.py", line 137, in check PyLinter.check(self, files_or_modules) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", line 747, in check self._do_check(files_or_modules) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", line 869, in _do_check self.check_astroid_module(ast_node, walker, rawcheckers, tokencheckers) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", line 946, in check_astroid_module walker.walk(ast_node) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/utils.py", line 874, in walk self.walk(child) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/utils.py", line 871, in walk cb(astroid) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/checkers/variables.py", line 904, in visit_from self._check_module_attrs(node, module, name.split('.')) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/checkers/variables.py", line 974, in _check_module_attrs module = next(module.getattr(name)[0].infer()) StopIteration Any clue what might be causing that? --Ned. On 11/5/15 6:24 AM, Ned Batchelder wrote:

...

On 11/5/15 5:21 AM, Noam Yorav-Raphael wrote:

...
Hi,

Take a look at cached_pylint.py at https://gist.github.com/noamraph/c933b32deb8304ac7ccd

It records module dependencies, and when a module passes with no warnings, it records it together with the sha1 of the contents of each of its dependencies, so when run again it could pass without checking it.

Dependencies are recorded by using infer and by using ImportsChecker, so if module A imports something from module B that was actually defined in module C, both B and C will be recorded as dependencies of A.

We have a medium sized code repository that took two minutes for each check. Now pylint only checks the modified modules, and it takes two seconds.

This sounds like a great feature. Have you considered contributing it as a pull request into pylint?

--Ned.

...
I tested it with pylint 1.4.3.

Hope this helps someone,

Noam

Noam Yorav-Raphael

8 Nov 8 Nov

2:34 a.m.

(I'm sorry, I accidentally had mail delivery turned off, so I reply to myself)

...

This sounds like a great feature. Have you considered contributing it as a pull request into pylint?

I will be glad to - you can use the code for whatever you like - but I think that a member of the team should review my code and say what he thinks and how it should be integrated into the main code.

...

I tried this on my large project (Open edX), and it generally worked great, but was one crash:

I updated the gist so now I hope it would work - can you check? Cheers, Noam On Thu, Nov 5, 2015 at 12:21 PM, Noam Yorav-Raphael wrote:

...

Hi,

Take a look at cached_pylint.py at https://gist.github.com/noamraph/c933b32deb8304ac7ccd

It records module dependencies, and when a module passes with no warnings, it records it together with the sha1 of the contents of each of its dependencies, so when run again it could pass without checking it.

Dependencies are recorded by using infer and by using ImportsChecker, so if module A imports something from module B that was actually defined in module C, both B and C will be recorded as dependencies of A.

We have a medium sized code repository that took two minutes for each check. Now pylint only checks the modified modules, and it takes two seconds.

I tested it with pylint 1.4.3.

Hope this helps someone,

Noam

Ned Batchelder

5:21 p.m.

...

(I'm sorry, I accidentally had mail delivery turned off, so I reply to myself)

...
This sounds like a great feature. Have you considered contributing it as a pull request into pylint?

I will be glad to - you can use the code for whatever you like - but I think that a member of the team should review my code and say what he thinks and how it should be integrated into the main code.

...
I tried this on my large project (Open edX), and it generally worked great, but was one crash:

I updated the gist so now I hope it would work - can you check? Thanks, the crash is fixed. On our code, which has a ton of violations

On 11/7/15 4:04 PM, Noam Yorav-Raphael wrote: that we are slowly working through, it's about the same speed as regular pylint though. I think we need to shift the balance so that more files have no violations before the caching pays off. Thanks, this is very interesting, and I hope it's pursued as a pylint feature. --Ned.

...

Cheers, Noam

On Thu, Nov 5, 2015 at 12:21 PM, Noam Yorav-Raphael mailto:noamraph@gmail.com> wrote:

Hi,

Take a look at cached_pylint.py at https://gist.github.com/noamraph/c933b32deb8304ac7ccd

It records module dependencies, and when a module passes with no warnings, it records it together with the sha1 of the contents of each of its dependencies, so when run again it could pass without checking it.

Dependencies are recorded by using infer and by using ImportsChecker, so if module A imports something from module B that was actually defined in module C, both B and C will be recorded as dependencies of A.

We have a medium sized code repository that took two minutes for each check. Now pylint only checks the modified modules, and it takes two seconds.

I tested it with pylint 1.4.3.

Hope this helps someone,

Noam

3085

Age (days ago)

3088

Last active (days ago)

List overview

Download

4 comments

2 participants

participants (2)

Ned Batchelder
Noam Yorav-Raphael

Make pylint faster by caching modules that passed without warnings

Noam Yorav-Raphael

Ned Batchelder

Ned Batchelder

Noam Yorav-Raphael

Ned Batchelder

tags

participants (2)