code-quality November 2014

code-quality@python.org

22 participants
13 discussions

Advice regarding caching values as a side-effect of visiting classes
by Anne Mulhern 04 Dec '14

04 Dec '14

Hi! I've implemented a very simple analysis of the kind described in this enhancement request: https://bitbucket.org/logilab/pylint/issue/201/pointless-attribute-override…. The analysis is neither sound nor complete, but has proven useful for removing of bits of dead code. It can be made more sound by altering the algorithm so that it accumulates data by visiting classes, and subsequently processes that accumulated data, preferably after all classes have been visited, in order to determine messages to report. Is there some infrastructure in pylint that can accomodate an analysis that works that way? If there is, is there a working example that uses it? Thanks for any help, - mulhern

2 2

pylint can't deduce the type of an object
by Paul Smith 29 Nov '14

29 Nov '14

Hi all; as mentioned in a previous post, I am looking into updating my curring pylint (1.0.1) to newer pylint 1.4.0. I've got one significant problem left and I'm not sure how to resolve it: in the new pylint I'm seeing a number of errors like this: mymod.py:227: [E1101(no-member), MyMod.__str__] Instance of 'int' has no 'get_val' member The code in question is something like: self.someobj.get_val() The problem appears to be that pylint cannot tell the type of the object stored in "someobj", so it assumes "int" and gives an error. The value of self.someobj is passed in via the constructor, and so this module really _doesn't_ know the type. I checked what pylint 1.0.1 is doing and it seems to be ignoring that value altogether, so it doesn't give the above incorrect error but it also doesn't complain if, for example, I change the call to "self.someobj.getval()" (or any other non-existent method). Can I do something to get pylint to understand this type? If it cannot figure it out automatically, is there some way I can hint or annotate the type of this member so pylint will understand what it is? Is there a way to disable the "assume an int" processing and ignore unknown members, like the old pylint did, without disabling E1101 altogether? Pylint is run automatically as part of our static analysis tests so I can't upgrade until I can get the new version to run clean.

2 3

Lots of E1101 in standard modules after upgrading pylint to 1.4.0
by Paul Smith 27 Nov '14

27 Nov '14

Hi all; I just upgraded to a newer version of pylint and now I'm seeing many spurious E1101 errors. Before I was using: * Python 2.7.6 * Pylint 1.1.0 * Astroid 1.0.1 * logilab-common 0.61.0 Now I've upgraded to: * Python 2.7.8 * Pylint 1.4.0 * Astroid 1.3.2 * logilab-common 0.63.0 Everything about this new install appears to work fine, EXCEPT pylint. I'm seeing tons of strange E1101 error on standard modules, like time.sleep() and others (readline, etc.) For example: $ cat sl.py import time time.sleep(1) $ python sl.py <sleeps for 1s successfully> $ pylint sl.py ************* Module sl C: 1, 0: Missing module docstring (missing-docstring) E: 2, 0: Module 'time' has no 'sleep' member (no-member) If I run this with my old setup (older python/pylint/etc.), it doesn't complain at all. Why am I seeing this invalid error? Thanks!

2 8

Re: [code-quality] [Pylint-dev] [ANN] Pylint 1.4 released
by Michal Nowikowski 27 Nov '14

27 Nov '14

Hello, Just submitted pull request with a fix: https://bitbucket.org/logilab/pylint/pull-request/205/fixed-reading-list-of… Regards, Godfryd On Thu, Nov 27, 2014 at 7:00 PM, Ned Batchelder <ned(a)nedbatchelder.com> wrote: > On 11/27/14 11:59 AM, Julien Cristau wrote: > >> On Thu, Nov 27, 2014 at 10:19:28 -0500, Ned Batchelder wrote: >> >> I found a problem with the new spell feature, but the issue tracker >>> (http://www.bytebucket.org/logilab/pylint/issues) seems broken: >>> everything I tried ended at a 403 CSRF validation failure page. >>> >>> Not sure where that url came from, afaik the tracker is >> https://bitbucket.org/logilab/pylint/issues >> >> > Hmm, not sure where bytebucket came from either! Sorry for the noise... > > Cheers, >> Julien >> >> > > -- > Ned Batchelder, http://nedbatchelder.com > > _______________________________________________ > Pylint-dev mailing list > Pylint-dev(a)lists.logilab.org > http://lists.logilab.org/mailman/listinfo/pylint-dev >

1 0

[ANN] Pylint 1.4 released
by Claudiu Popa 27 Nov '14

27 Nov '14

Hello! On behalf of the Pylint development team, I'm happy to announce that Pylint 1.4 has been released. This release has a lot of improvements over the last one. One of the main differences is that support for Python versions < 2.7 has been droped, which allows us to support Python 2.7 and 3.3+ from a single codebase. Other important changes: * A Python 3 porting checker, activated by the new flag '--py3k'. This mode will disable all other checkers and will emit warnings and errors for constructs which are invalid or removed in Python 3. * New options for controlling the loading of C extensions. By default, only C extensions from the stdlib will be loaded into the active Python interpreter for inspection, because they can run arbitrary code on import. The option `--extension-pkg-whitelist` can be used to specify modules or packages that are safe to load. * An experimental support for using multiple workers to process files and packages, activated with the new --jobs flag. * A new spelling checker (disabled by default). * New warnings: boolean-datetime, logging-format-interpolation, inherit-non-class, mixed-line-endings * A lot of other small improvements and bug fixes. If you find any bugs, don't hesitate to open a new issue on our issue tracker. Enjoy!

3 3

PyLint catching comprehension binding leaks (was: Warn on list comprehension variables used outside list comprehensions?)
by Ian Cordasco 27 Nov '14

27 Nov '14

On Wed, Nov 26, 2014 at 10:51 AM, Keith Derrick <keith.derrick(a)lge.com> wrote: > I couldn't find anything in their bug list about this, so thought I'd check. > > Just ran pylint at the command line with pylint 1.4 on this file > (docstrings and import just to suppress miscellaneous complaints) > >> 1 ''' Test file for pylint ''' >> 2 from __future__ import print_function >> 3 >> 4 def method1(): >> 5 ''' Let x bleed from comprehension ''' >> 6 [x for x in range(3)] >> 7 print(x) >> 8 >> 9 def method2(): >> 10 ''' reuse/hide x from local scope in comprehension ''' >> 11 y = 10 >> 12 [y for y in range(3)] >> 13 print(y) >> 14 > > For python 2.7.6 >> ************* Module bug >> W: 6, 4: Expression "[x for x in range(3)]" is assigned to nothing >> (expression-not-assigned) >> W: 7,10: Using possibly undefined loop variable 'x' >> (undefined-loop-variable) >> C: 11, 4: Invalid variable name "y" (invalid-name) >> W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing >> (expression-not-assigned) > For python 3.4.0 >> ************* Module bug >> W: 6, 4: Expression "[x for x in range(3)]" is assigned to nothing >> (expression-not-assigned) >> E: 7,10: Undefined variable 'x' (undefined-variable) >> C: 11, 4: Invalid variable name "y" (invalid-name) >> W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing >> (expression-not-assigned) > > So, it warns about the first case in py2 and gives an error in py3 which > is expected. > > But it seems to be confused by the second case, flagging "y" as an > invalid variable-name at the assignment point for both versions. > > Keith Derrick | Principal Engineer, Connected Platform | Engineering > LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054 > Office: 408.610-5746 | Mobile: 831.383.9567 | LG.com > At this point, I'd recommend repling to the retitled message to catch the PyLint maintainer's focus. > On 11/26/2014 07:50 AM, Ian Cordasco wrote: >> On Wed, Nov 26, 2014 at 9:27 AM, Keith Derrick <keith.derrick(a)lge.com> wrote: >>> Definitely seems like something that should be warned about. >>> >>> For python2 it ought to be an error (which can always be turned off) for >>> either code snippet. >> PyFlakes doesn't allow you to ignore errors. Flake8 does so this is a >> moot point. >> >>> For python3, I'd say the two line snippet should flag an error - yes, it >>> will fail at run-time, but pydev in eclipse runs pylint and saves me a >>> lot of grief by spotting this type of thing before I ever save the file >>> (though it doesn't catch either of these, presumably because they are >>> valid python). >> If PyLint doesn't catch this I wonder if it's either never been raised >> or been decided to be a bad idea. Could someone search their bugs? >> >>> The three line snippet should probably give a warning ("Are you sure you >>> mean this?") >>> >>> The question is, how do we know which version of Python the code under >>> inspection is targeted for, and hence how to treat this case. >>> >>> Some thoughts on that: >>> >>> * Anyone serious enough to be using pyflakes, pylint, or other static >>> analysis tools is also probably using virtualenv. >>> * pyflakes is therefore most likely running under the same python >>> version for which the code is targeted. >>> * code which is meant to run under multiple python versions is probably >>> tested under something like tox - so again, pyflakes is run under the >>> targeted python version (certainly my tox.ini runs it in every target >>> environment) >>> >>> There's a lot of "probably"'s in there, but I don't think they are >>> unreasonable assumptions. >>> >>> So, could pyflakes reasonably assume that the interpreter under which >>> it is running is also the intended target version for the code under >>> inspection? At least for the major version. >> It already works this way. If you use pyflakes on Python 2 but are >> writing code only valid in Python 3 you will have a bad time. You have >> to install pyflakes on the version of Python you want it to lint for >> since pyflakes works only with the in-built ast module of Python. >> >>> This could then be overridden by a suitable command line option to >>> specify the target version(s) >> No. This isn't how PyFlakes is intended to work and this is extraneous. >> >>> Also, in cases like this, possibly a "This will behave differently in >>> Python 2 and 3" warning is a worth while alternative. >> Again, PyFlakes is very different from PyLint. We don't issue warnings >> like this. >> >>> Keith Derrick | Principal Engineer, Connected Platform | Engineering >>> LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054 >>> Office: 408.610-5746 | Mobile: 831.383.9567 | LG.com >>> >>> On 11/26/2014 06:57 AM, Skip Montanaro wrote: >>>>> That's not the point of this check. The point of this check is that on >>>>> Python 2, the binding to x in the comprehension bleeds outside of the >>>>> comprehension scope >>>> Got it. >>>> >>>> Still, this code: >>>> >>>>> x = 10 >>>>> [x for x in range(3)] >>>>> print(x + 1) >>>> will run differently in Python 2 than Python3, so even if that was a >>>> conscious choice by the author, a --py3k flag should cause a message >>>> for this code. This is even worse than the simpler >>>> >>>>> [x for x in range(3)] >>>>> print(x + 1) >>>> because at least that will raise a NameError when run in Python 3. The >>>> three-line construct will still run, though produce different output. >>>> >>>> Skip >>>> _______________________________________________ >>>> code-quality mailing list >>>> code-quality(a)python.org >>>> https://mail.python.org/mailman/listinfo/code-quality >>> _______________________________________________ >>> code-quality mailing list >>> code-quality(a)python.org >>> https://mail.python.org/mailman/listinfo/code-quality

2 1

Warn on list comprehension variables used outside list comprehensions?
by code-quality.wting＠xoxy.net 26 Nov '14

26 Nov '14

Python binds list comprehension variables to the local scope which has caused some subtle bugs. Is it possible to add a warning for this in pyflakes? I haven't implemented it yet, but here are the example tests: def test_listCompVariableUsedOutsideListComp(self): """ Test that a variable defined in a list comprehension is not used outside of the list comprehension. """ self.flakes(''' [x for x in range(3)] print x ''', m.VariableUsedOutsideListComp) self.flakes(''' [x for x in range(3)] [x for _ in range(3)] ''', m.VariableUsedOutsideListComp) def test_listCompVariableAllowReuse(self): """ Test that list comprehension variables are allowed to be reused if redefined. """ self.flakes(''' [x for x in range(3)] [x for x in range(3)]''') - William

5 9

Problems running pylint 1.4.0
by Skip Montanaro 26 Nov '14

26 Nov '14

We have a rather peculiar software distribution system here at work. I downloaded, packaged, and installed astroid, logilab-common, and pylint. When I run pylint now (1.1.0 worked just fine), I get this error: Traceback (most recent call last): File "/opt/local/bin/pylint", line 6, in <module> from pkg_resources import load_entry_point File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", line 2805, in <module> working_set.require(__requires__) File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", line 696, in require needed = self.resolve(parse_requirements(requirements)) File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", line 594, in resolve raise DistributionNotFound(req) pkg_resources.DistributionNotFound: pylint==1.4.0 I see the (I think) necessary pylint bits in our site-packages directory: % ls -ld /opt/local/lib/python2.7/site-packages/pylint* drwxr-xr-x 6 root root 4096 Nov 25 14:43 /opt/local/lib/python2.7/site-packages/pylint drwxr-xr-x 2 root root 4096 Nov 25 14:43 /opt/local/lib/python2.7/site-packages/pylint-1.4.0-py2.7.egg-info and they seem to have reasonable contents: % ls -lL /opt/local/lib/python2.7/site-packages/pylint total 324 -rw-rw-r-- 1 root root 1458 Nov 15 12:18 __init__.py -rw-rw-r-- 1 root root 1838 Nov 25 14:42 __init__.pyc -rw-rw-r-- 1 root root 56 Nov 15 12:18 __main__.py -rw-rw-r-- 1 root root 226 Nov 25 14:42 __main__.pyc -rw-rw-r-- 1 root root 2925 Nov 23 03:03 __pkginfo__.py -rw-rw-r-- 1 root root 2439 Nov 25 14:42 __pkginfo__.pyc drwxr-xr-x 2 root root 4096 Nov 25 14:43 checkers -rw-rw-r-- 1 root root 5782 Nov 21 05:23 config.py -rw-rw-r-- 1 root root 5407 Nov 25 14:42 config.pyc -rw-rw-r-- 1 root root 6445 Nov 15 12:18 epylint.py -rw-rw-r-- 1 root root 5848 Nov 25 14:42 epylint.pyc -rw-rw-r-- 1 root root 19401 Nov 21 05:23 gui.py -rw-rw-r-- 1 root root 18419 Nov 25 14:42 gui.pyc -rw-rw-r-- 1 root root 2828 Nov 15 12:18 interfaces.py -rw-rw-r-- 1 root root 4143 Nov 25 14:42 interfaces.pyc -rw-rw-r-- 1 root root 57390 Nov 23 02:46 lint.py -rw-rw-r-- 1 root root 49003 Nov 25 14:42 lint.pyc drwxr-xr-x 2 root root 4096 Nov 25 14:43 pyreverse drwxr-xr-x 2 root root 4096 Nov 25 14:43 reporters drwxr-xr-x 7 root root 4096 Nov 25 14:43 test -rw-rw-r-- 1 root root 13371 Nov 21 05:23 testutils.py -rw-rw-r-- 1 root root 36111 Nov 21 05:23 utils.py -rw-rw-r-- 1 root root 34047 Nov 25 14:42 utils.pyc blade% ls -lL /opt/local/lib/python2.7/site-packages/pylint-1.4.0-py2.7.egg-info/ total 44 -rw-rw-r-- 1 root root 3227 Nov 25 14:42 PKG-INFO -rw-rw-r-- 1 root root 22242 Nov 25 14:42 SOURCES.txt -rw-rw-r-- 1 root root 1 Nov 25 14:42 dependency_links.txt -rw-rw-r-- 1 root root 172 Nov 25 14:42 entry_points.txt -rw-rw-r-- 1 root root 45 Nov 25 14:42 requires.txt -rw-rw-r-- 1 root root 7 Nov 25 14:42 top_level.txt The whole distutils/setuptools/pkg_resources stuff has always eluded my understanding. Can someone give me some suggestions about what's missing? Thanks, Skip Montanaro

3 4

Static checker for common Python programming errors
by Stefan Bucur 25 Nov '14

25 Nov '14

I'm developing a Python static analysis tool that flags common programming errors in Python programs. The tool is meant to complement other tools like Pylint (which perform checks at lexical and AST level) by going deeper with the code analysis and keeping track of the possible control flow paths in the program (path-sensitive analysis). For instance, a path-sensitive analysis detects that the following snippet of code would raise an AttributeError exception: if object is None: # If the True branch is taken, we know the object is None object.doSomething() # ... so this statement would always fail I wanted first to tap into people's experience and get a sense of what common pitfalls in the language & its standard library such a static checker should look for. Just as an example of what I mean, here [1] is a list of static checks for the C++ language, as part of the Clang static analyzer project. My preliminary list of Python checks is quite rudimentary, but maybe could serve as a discussion starter: * Proper Unicode handling (for 2.x) - encode() is not called on str object - decode() is not called on unicode object * Check for integer division by zero * Check for None object dereferences Thanks a lot, Stefan Bucur [1] http://clang-analyzer.llvm.org/available_checks.html

5 8

RedBaron, a bottom-up refactoring lib/tool for python
by Laurent Peuch 23 Nov '14

23 Nov '14

Hello everyone, Someone has suggested me to talk about the project I'm working on right now on this mailing list because this has a lot of chances to interest you. This tool is an answer to a frustration that I've had while trying to build tools for python and projects I was working on. While there is already good capacities in python to analyse code (ast.py, astroid (while it wasn't out at that time)), the "write code that modify source code" was really missing (in my opinion and my knowledge of the existing tools). I wanted a very intuitive and easy to use library that allows me to query and modify my source code only in the place I wanted to modify it without touching the rest of the code. So I've built what can be describe as "the BeautifulSoup of python source code". To do so, I've built what can be called "a lossless AST" for python (designed to be used by humans), an AST that satisfy this equation: source_code == ast_to_source(source_to_ast(source_code)) It produces json-serializable python data structures (because data structures are easier to use and don't hide anything from you). And now the part that should interest you more: on top on that AST, I've built an high level "query and modification" library that wraps those AST nodes into objects. I've put a lot of efforts in making this library intuitive and very easy to use while removing you the burden of having to deal with low level details. This "BeautifulSoup of the python source code" is called Redbaron. It looks like this: from redbaron import RedBaron # simple API # pass string red = RedBaron("some_value = 42") # get string back red.dumps() Queries are like BeautifulSoup: red.find("int", value=4) red.find_all("def", name="stuff") (You can pass lambda/regex/special syntaxe for globs/regex etc... to queries, they should be powerful enough for the vast majorities of your needs). Nodes modification is very simple: just pass source code stored in string and "voilà": red = RedBaron("some_value = 42") red[0].value = "1 + 1" # some_value = 1 + 1 red = RedBaron("def stuff():\n plop") red[0].value = "some_code" # def stuff():\n some_code # notice that the input is correctly formatting, indented and it # also takes care of not breaking the next node indentation # works too with decorators and where you expect it to works (It is possible to pass it ast datastructure or RedBaron objects to). And I've made an abstraction on top of "list of things" so you don't have to take care about when you need to put a separator (for eg: a "," in a list): red = RedBaron("[1, 2, 3]") red[0].append("plop") # [1, 2, 3, plop] # want to add a new django app to INSTALLED_APPS? just do: red.find("assignment", target=lambda x: x.dumps() == "INSTALLED_APPLICATIONS").value.append("'another_app'") # notice that the formatting of the list is detected # want to add "@profile" to every function of the root level for # line_profiler? red('def', recursive=False).map(lambda x: x.decorators.insert(0, '@profile')) # and remove them red("decorator", lambda x: x.dumps() == "@decorator").map(lambda x: x.parent.parent.decorators.remove(x)) # convert every "print a" to "logger.debug(a) red('print', value=lambda x: len(x) == 1).map(lambda x: x.replace('logger.debug(%s)' % x.value.dumps()) # and print a, b, c to logger.debug("%s %s %s" % (a, b, c)) red('print', value=lambda x: len(x) == 1).map(lambda x: x.replace('logger.debug("%s" % (%s))' % (" ".join('%s' * len(x.value))) Both library and fully tested (more than 2000 tests in total), fully *documented* (with lots of examples) and under freesoftware licences. I consider RedBaron to be in alpha stage, it is already very stable but a significant number of edge cases are probably not handled yet. Important point: RedBaron is not and will not do static analysis, I'm probably going to integrate (or integrate RedBaron into) a tool that already do that like astroid or rope. Links: * RedBaron tutorial: https://redbaron.readthedocs.org/en/latest/tuto.html * RedBaron documentation: https://redbaron.readthedocs.org * RedBaron source code: https://github.com/psycojoker/redbaron * Baron (the AST) source code: https://github.com/psycojoker/baron * Baron documentation: https://baron.readthedocs.org I hope that I have trigger your interest and I'm very interested by your feedback, Have a nice day and thanks for your time, PS: I've only been aware of the capacities of lib2to3 since 2 months and was very unhappy to discover it so late (I've spent months or googling before deciding to start this project), I'll probably swap my parser with lib2to3 one in the future. -- Laurent Peuch -- Bram

5 11

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

code-quality November 2014