From amulhern at redhat.com  Thu Dec  4 22:17:57 2014
From: amulhern at redhat.com (Anne Mulhern)
Date: Thu, 4 Dec 2014 16:17:57 -0500 (EST)
Subject: [code-quality] Possible to make extensions subdirectory a separate
	package?
In-Reply-To: <308313554.8965600.1417727609095.JavaMail.zimbra@redhat.com>
Message-ID: <1541591574.8967139.1417727877327.JavaMail.zimbra@redhat.com>

Hi!

I think that making contents of proposed extensions subdirectory a separate
package from pylint itself would be a great convenience for users who may
not be ready to update to most recent version of pylint, but are interested
in some optional checker.

Please let me know what you think.

Thanks!

- mulhern

From graffatcolmingov at gmail.com  Thu Dec  4 22:46:08 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Thu, 4 Dec 2014 15:46:08 -0600
Subject: [code-quality] Possible to make extensions subdirectory a
 separate package?
In-Reply-To: <1541591574.8967139.1417727877327.JavaMail.zimbra@redhat.com>
References: <308313554.8965600.1417727609095.JavaMail.zimbra@redhat.com>
 <1541591574.8967139.1417727877327.JavaMail.zimbra@redhat.com>
Message-ID: <CAN-Kwu1NhKPGopj+mzAWoq5QqcQBLa5kLrmT_X8YfFEPe9J5ew@mail.gmail.com>

On Dec 4, 2014 3:19 PM, "Anne Mulhern" <amulhern at redhat.com> wrote:
>
> Hi!
>
> I think that making contents of proposed extensions subdirectory a
separate
> package from pylint itself would be a great convenience for users who may
> not be ready to update to most recent version of pylint, but are
interested
> in some optional checker.
>
> Please let me know what you think.
>
> Thanks!
>
> - mulhern
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality

I don't use them but testing them as third-party packages could also help
determine if and when they should become part of pylint proper. This is an
excellent suggestion Anne!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141204/aca83cc5/attachment.html>

From amulhern at redhat.com  Thu Dec  4 22:22:02 2014
From: amulhern at redhat.com (Anne Mulhern)
Date: Thu, 4 Dec 2014 16:22:02 -0500 (EST)
Subject: [code-quality] Advice regarding caching values as a side-effect
 of visiting classes
In-Reply-To: <CADq85CQeEwzEdeWaasXwc-Oimt+98Thym1m_yvu=CfSvr27t5Q@mail.gmail.com>
References: <1114454924.2638246.1416591897742.JavaMail.zimbra@redhat.com>
 <812774972.2645807.1416592693466.JavaMail.zimbra@redhat.com>
 <CADq85CQeEwzEdeWaasXwc-Oimt+98Thym1m_yvu=CfSvr27t5Q@mail.gmail.com>
Message-ID: <1320972880.8968476.1417728122745.JavaMail.zimbra@redhat.com>

Thanks!

I ended up getting less ambitious with the analysis,
but the definition of close() method in Misdesign checker
seemed like a good working example to start with.

- mulhern

----- Original Message -----
> From: "Torsten Marek" <shlomme at gmail.com>
> To: "Anne Mulhern" <amulhern at redhat.com>
> Cc: "code-quality" <code-quality at python.org>
> Sent: Friday, November 21, 2014 5:32:15 PM
> Subject: Re: [code-quality] Advice regarding caching values as a side-effect of visiting classes
> 
> Hi Anne,
> 
> you can write a checker that simply stores all classes it encounters and
> then walks the inheritance tree to find all violations in `close`. This
> checker method is called after all modules specified on the commandline
> have been checked, but before reports are generated, so emitting warnings
> at this point is possible.
> 
> // Torsten
> 
> 
> 
> 2014-11-21 18:58 GMT+01:00 Anne Mulhern <amulhern at redhat.com>:
> 
> > Hi!
> >
> > I've implemented a very simple analysis of the kind described in this
> > enhancement
> > request:
> > https://bitbucket.org/logilab/pylint/issue/201/pointless-attribute-override-checker
> > .
> > The analysis is neither sound nor complete, but has proven useful for
> > removing of bits of
> > dead code.
> >
> > It can be made more sound by altering the algorithm so that it accumulates
> > data by visiting classes,
> > and subsequently processes that accumulated data, preferably after all
> > classes have been visited,
> > in order to determine messages to report.
> >
> > Is there some infrastructure in pylint that can accomodate an analysis
> > that works that way?
> > If there is, is there a working example that uses it?
> >
> > Thanks for any help,
> >
> > - mulhern
> >
> > _______________________________________________
> > code-quality mailing list
> > code-quality at python.org
> > https://mail.python.org/mailman/listinfo/code-quality
> >
> 

From amulhern at redhat.com  Thu Dec  4 22:41:26 2014
From: amulhern at redhat.com (Anne Mulhern)
Date: Thu, 4 Dec 2014 16:41:26 -0500 (EST)
Subject: [code-quality] Time cost for pylint analyses?
In-Reply-To: <1302775550.8974493.1417729186785.JavaMail.zimbra@redhat.com>
Message-ID: <1210013895.8974985.1417729286090.JavaMail.zimbra@redhat.com>

Hi!

Would it be possible to be able to extend pylint somehow to
record a time cost for individual pylint analyses?

- mulhern

From shlomme at gmail.com  Sun Dec  7 10:51:01 2014
From: shlomme at gmail.com (Torsten Marek)
Date: Sun, 7 Dec 2014 10:51:01 +0100
Subject: [code-quality] Time cost for pylint analyses?
In-Reply-To: <1210013895.8974985.1417729286090.JavaMail.zimbra@redhat.com>
References: <1302775550.8974493.1417729186785.JavaMail.zimbra@redhat.com>
 <1210013895.8974985.1417729286090.JavaMail.zimbra@redhat.com>
Message-ID: <CADq85CT4oR07Rv3Aexv=9aQrtMZ+kEj0ukPg7D==ja=QxvUj4w@mail.gmail.com>

Hi Anne,

should be fairly simple, yes. What kind of breakdown do you mean, by
module/package or by checker?

// Torsten

2014-12-04 22:41 GMT+01:00 Anne Mulhern <amulhern at redhat.com>:

> Hi!
>
> Would it be possible to be able to extend pylint somehow to
> record a time cost for individual pylint analyses?
>
> - mulhern
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141207/41b0aaa5/attachment.html>

From skip.montanaro at gmail.com  Mon Dec  8 19:16:45 2014
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 8 Dec 2014 12:16:45 -0600
Subject: [code-quality] This seems like a good warning candidate...
Message-ID: <CANc-5Uywz1TtQiOeo8o_HdoM_TaasAfQfMpbDtqL3_wwWtvOtA@mail.gmail.com>

I just bumped into a bug, reusing the same loop variable in two nested for
loops. Here's an example:

def f(x):
    for i in range(x):
        for i in [1, 2, 3]:
            print i

Neither pylint nor flake8 warned about this problematic usage. Had i or x
shadowed a global of the same name, pylint would have complained, e.g.:

x = 7
def f(x):
    for i in range(x):
        for i in [1, 2, 3]:
            x = i
            print i, x

Using the same variable in nested loops within the same function seems at
least as serious as a shadowed global (or builtin) name.

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141208/3b767da2/attachment.html>

From graffatcolmingov at gmail.com  Mon Dec  8 19:24:54 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 8 Dec 2014 12:24:54 -0600
Subject: [code-quality] This seems like a good warning candidate...
In-Reply-To: <CANc-5Uywz1TtQiOeo8o_HdoM_TaasAfQfMpbDtqL3_wwWtvOtA@mail.gmail.com>
References: <CANc-5Uywz1TtQiOeo8o_HdoM_TaasAfQfMpbDtqL3_wwWtvOtA@mail.gmail.com>
Message-ID: <CAN-Kwu2ZNhBCVCH9yt21gfnOjCzEn1jbiH_mqQF1tGgdsxzvWA@mail.gmail.com>

On Dec 8, 2014 12:17 PM, "Skip Montanaro" <skip.montanaro at gmail.com> wrote:
>
> I just bumped into a bug, reusing the same loop variable in two nested
for loops. Here's an example:
>
> def f(x):
>     for i in range(x):
>         for i in [1, 2, 3]:
>             print i
>
> Neither pylint nor flake8 warned about this problematic usage. Had i or x
shadowed a global of the same name, pylint would have complained, e.g.:
>
> x = 7
> def f(x):
>     for i in range(x):
>         for i in [1, 2, 3]:
>             x = i
>             print i, x
>
> Using the same variable in nested loops within the same function seems at
least as serious as a shadowed global (or builtin) name.
>
> Skip
>
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>

I agree. I think this belongs in PyFlakes (as far as flake8 is concerned)
but Phil may disagree.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141208/76db3c52/attachment.html>

From indigo at bitglue.com  Mon Dec  8 20:32:09 2014
From: indigo at bitglue.com (Phil Frost)
Date: Mon, 8 Dec 2014 14:32:09 -0500
Subject: [code-quality] This seems like a good warning candidate...
In-Reply-To: <CAN-Kwu2ZNhBCVCH9yt21gfnOjCzEn1jbiH_mqQF1tGgdsxzvWA@mail.gmail.com>
References: <CANc-5Uywz1TtQiOeo8o_HdoM_TaasAfQfMpbDtqL3_wwWtvOtA@mail.gmail.com>
 <CAN-Kwu2ZNhBCVCH9yt21gfnOjCzEn1jbiH_mqQF1tGgdsxzvWA@mail.gmail.com>
Message-ID: <CAP4Jji5Ogvp5YOxR1v+KAGEOzyGV+KRseKT6rirNo5hiMebcOA@mail.gmail.com>

On Mon, Dec 8, 2014 at 1:24 PM, Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

>
> On Dec 8, 2014 12:17 PM, "Skip Montanaro" <skip.montanaro at gmail.com>
> wrote:
> >
> > Using the same variable in nested loops within the same function seems
> at least as serious as a shadowed global (or builtin) name.
>
> I agree. I think this belongs in PyFlakes (as far as flake8 is concerned)
> but Phil may disagree.
>

Maybe. As a matter of principle, Pyflakes should only emit a warning for
things that it is sure is an error.

Currently, pyflakes only emits a warning if you redefine an import without
ever using it. For example:

import x
def f():
  x = 2
  print x
<stdin>:1: 'x' imported but unused
<stdin>:3: redefinition of unused 'x' from line 1

It does not do this for other kinds of definitions:

x = 1
def f():
  x = 2
  print x
[no errors]

Also note that as long as the imported "x" is used somewhere, then there
are no warnings, even if "x" is shadowed in some scopes:

import x
def f():
  x = 2
  print x

def g():
  print x
[no errors]

The reasoning here is that if you import a thing and then redefine it
without using it ever, that's just the same as importing a thing and not
using it.

However, I can come up with use cases for shadowing variables from function
scope or in nested loops. For example, event-based libraries frequently
expect callbacks that take one parameter which the callback may or may not
care about. Or, we may use a for loop to do something a number of times
without caring about the iteration count. I tend to use "_" for these sorts
of things, a convention I got from lisp.

def thingHappened(_):
  # don't care about the thing that happened, just that it happened

class Sheldon:
  def visit_penny(self):
    for _ in xrange(3):
      for _ in xrange(3):
        knock()
      say('penny')

I wouldn't want a warning in these cases, as they are legitimate and
correct. Can you think of any additional reasoning we could apply that
would catch your case, but not these?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141208/71a9205a/attachment.html>

From skip.montanaro at gmail.com  Mon Dec  8 21:07:34 2014
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 8 Dec 2014 14:07:34 -0600
Subject: [code-quality] This seems like a good warning candidate...
In-Reply-To: <CAP4Jji5Ogvp5YOxR1v+KAGEOzyGV+KRseKT6rirNo5hiMebcOA@mail.gmail.com>
References: <CANc-5Uywz1TtQiOeo8o_HdoM_TaasAfQfMpbDtqL3_wwWtvOtA@mail.gmail.com>
 <CAN-Kwu2ZNhBCVCH9yt21gfnOjCzEn1jbiH_mqQF1tGgdsxzvWA@mail.gmail.com>
 <CAP4Jji5Ogvp5YOxR1v+KAGEOzyGV+KRseKT6rirNo5hiMebcOA@mail.gmail.com>
Message-ID: <CANc-5UxhK84zxBXEx6c7vybNj1adBtE0igy3sPHGMq=F6H7kXw@mail.gmail.com>

On Mon, Dec 8, 2014 at 1:32 PM, Phil Frost <indigo at bitglue.com> wrote:

> Maybe. As a matter of principle, Pyflakes should only emit a warning for
> things that it is sure is an error.


The avoidance of false positives is a noble goal, but given the dynamic
nature of Python, I'm happy to put up with a few if it means flagging stuff
that is likely to be an error.

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141208/45434a18/attachment.html>

From graffatcolmingov at gmail.com  Mon Dec  8 21:21:58 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 8 Dec 2014 14:21:58 -0600
Subject: [code-quality] This seems like a good warning candidate...
In-Reply-To: <CANc-5UxhK84zxBXEx6c7vybNj1adBtE0igy3sPHGMq=F6H7kXw@mail.gmail.com>
References: <CANc-5Uywz1TtQiOeo8o_HdoM_TaasAfQfMpbDtqL3_wwWtvOtA@mail.gmail.com>
 <CAN-Kwu2ZNhBCVCH9yt21gfnOjCzEn1jbiH_mqQF1tGgdsxzvWA@mail.gmail.com>
 <CAP4Jji5Ogvp5YOxR1v+KAGEOzyGV+KRseKT6rirNo5hiMebcOA@mail.gmail.com>
 <CANc-5UxhK84zxBXEx6c7vybNj1adBtE0igy3sPHGMq=F6H7kXw@mail.gmail.com>
Message-ID: <CAN-Kwu3LxVZNAthjsERxWOX+PiY_RFE9ojdQPggDVyA-W0gfmQ@mail.gmail.com>

On Mon, Dec 8, 2014 at 2:07 PM, Skip Montanaro <skip.montanaro at gmail.com> wrote:
>
> On Mon, Dec 8, 2014 at 1:32 PM, Phil Frost <indigo at bitglue.com> wrote:
>>
>> Maybe. As a matter of principle, Pyflakes should only emit a warning for
>> things that it is sure is an error.
>
>
> The avoidance of false positives is a noble goal, but given the dynamic
> nature of Python, I'm happy to put up with a few if it means flagging stuff
> that is likely to be an error.
>
> Skip

I'm not sure in what case code like what Skip shared wouldn't result
in an error:

    for i in range(1, 10):
        for i in ['foo', 'frob', 'bob', 'bogus', 'smogus']:
            do_stuff(i)
        do_other_stuff(i)

do_other_stuff will probably raise a TypeError if it's expecting an
int instead of a string (and is doing something that cannot be done to
both). I'm fairly confident this will be an error in most cases and
for the people for whom it will be a false positive, it will maybe
teach them to write better loop variable names.

From skip.montanaro at gmail.com  Mon Dec  8 22:10:10 2014
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 8 Dec 2014 15:10:10 -0600
Subject: [code-quality] The union of all tools
Message-ID: <CANc-5UwjZMJJxxW68kybbyUpLbKYP=sDmkeBPTnHkjxLf+ZVjg@mail.gmail.com>

Thinking about the discussion I (inadvertently) started about what sorts of
issues should cause tools like pylint and pyflakes to emit messages, got me
thinking about how I try to use these tools. I don't know what the current
suite of all available static checking tools is, but let me just assume for
the moment that we have three tools: pylint, pyflakes, and (the probably
now defunct) pychecker. Each one has different goals, which naturally
affects the messages it can possibly emit. For example, because pychecker
imports the modules mentioned in the code it analyzes, it is able to build
symbol tables for extension modules. OTOH, it could be slower, the user
might suffer if importing modules had side effects, and to get the most out
of it, you were tied to the platforms for which you had compiled extension
modules. I don't think pylint actually imports the code it analyzes, so it
can be more platform-independent, but not as comprehensive as pychecker
when trying to emit warnings about cross-module issues. It can, however,
identify other problematic constructs.

As a user of these tools, recognizing that none will ever be "complete", I
try to use multiple tools precisely so I can see the union of all messages.
In fact, some while ago (at least a couple years, probably longer), I wrote
a shell script wrapper which invokes both pychecker and pylint, presenting
me with the sorted (and very lightly massaged) output. If pyflakes is so
conservative in what it will emit that it tries to avoid all possible false
positives, it's less useful to me, precisely because the odds are low that
it will report problems the other available tools won't catch. That is:

   pylint ? pychecker ? pylint ? pychecker ? pyflakes

making the incremental benefit to me of using pyflakes small.

This is not to say that Phil doesn't have excellent reasons for his
approach, just that there is a non-trivial trade-off in that approach.

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141208/15647312/attachment.html>

From carl.crowder at gmail.com  Mon Dec  8 22:34:46 2014
From: carl.crowder at gmail.com (Carl Crowder)
Date: Mon, 8 Dec 2014 22:34:46 +0100
Subject: [code-quality] The union of all tools
In-Reply-To: <CANc-5UwjZMJJxxW68kybbyUpLbKYP=sDmkeBPTnHkjxLf+ZVjg@mail.gmail.com>
References: <CANc-5UwjZMJJxxW68kybbyUpLbKYP=sDmkeBPTnHkjxLf+ZVjg@mail.gmail.com>
Message-ID: <CAK2BMQvGGKFj5uhBuxO6Z093Gw6OAwAho=hq9A_eJaFzuJHT2A@mail.gmail.com>

Hi Skip,

This kind of argument was on my mind a little when I made prospector
(https://github.com/landscapeio/prospector) although probably from a
different angle. There are many tools out there and using them all
means you can get the maximal coverage of all concepts, with some
overlap. Each tool has a spectrum of certainty and usefulness, and
what you really want is the biggest problems in a specific code base.
"Biggest problem" as a concept varies from codebase to codebase.

Of course the ideal tool figures out exactly what you want and what
level of scrutiny you want on your code, and delivers the most
important problems first. Such a tool is impossible! I think you can
get there though, if you merge each tool (and therefore, each opinion
of what is important) enough.

The main concern is how to filter the wheat from the chaff. So far I
have simply made opinionated defaults and there is a way to make your
own configuration; that level of effort isn't that different to simply
configuring each tool yourself!

I'd love some feedback on this concept in general. I think each tool
and each error and each codebase and each user is a unique point and
it's difficult to create something that caters to all tastes. However,
having several starting positions tailored to individual use cases
could be helpful.

Cheers,
Carl

On 8 December 2014 at 22:10, Skip Montanaro <skip.montanaro at gmail.com> wrote:
> Thinking about the discussion I (inadvertently) started about what sorts of
> issues should cause tools like pylint and pyflakes to emit messages, got me
> thinking about how I try to use these tools. I don't know what the current
> suite of all available static checking tools is, but let me just assume for
> the moment that we have three tools: pylint, pyflakes, and (the probably now
> defunct) pychecker. Each one has different goals, which naturally affects
> the messages it can possibly emit. For example, because pychecker imports
> the modules mentioned in the code it analyzes, it is able to build symbol
> tables for extension modules. OTOH, it could be slower, the user might
> suffer if importing modules had side effects, and to get the most out of it,
> you were tied to the platforms for which you had compiled extension modules.
> I don't think pylint actually imports the code it analyzes, so it can be
> more platform-independent, but not as comprehensive as pychecker when trying
> to emit warnings about cross-module issues. It can, however, identify other
> problematic constructs.
>
> As a user of these tools, recognizing that none will ever be "complete", I
> try to use multiple tools precisely so I can see the union of all messages.
> In fact, some while ago (at least a couple years, probably longer), I wrote
> a shell script wrapper which invokes both pychecker and pylint, presenting
> me with the sorted (and very lightly massaged) output. If pyflakes is so
> conservative in what it will emit that it tries to avoid all possible false
> positives, it's less useful to me, precisely because the odds are low that
> it will report problems the other available tools won't catch. That is:
>
>    pylint ? pychecker ? pylint ? pychecker ? pyflakes
>
> making the incremental benefit to me of using pyflakes small.
>
> This is not to say that Phil doesn't have excellent reasons for his
> approach, just that there is a non-trivial trade-off in that approach.
>
> Skip
>
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>

From indigo at bitglue.com  Mon Dec  8 22:44:46 2014
From: indigo at bitglue.com (Phil Frost)
Date: Mon, 8 Dec 2014 16:44:46 -0500
Subject: [code-quality] This seems like a good warning candidate...
In-Reply-To: <CAN-Kwu3LxVZNAthjsERxWOX+PiY_RFE9ojdQPggDVyA-W0gfmQ@mail.gmail.com>
References: <CANc-5Uywz1TtQiOeo8o_HdoM_TaasAfQfMpbDtqL3_wwWtvOtA@mail.gmail.com>
 <CAN-Kwu2ZNhBCVCH9yt21gfnOjCzEn1jbiH_mqQF1tGgdsxzvWA@mail.gmail.com>
 <CAP4Jji5Ogvp5YOxR1v+KAGEOzyGV+KRseKT6rirNo5hiMebcOA@mail.gmail.com>
 <CANc-5UxhK84zxBXEx6c7vybNj1adBtE0igy3sPHGMq=F6H7kXw@mail.gmail.com>
 <CAN-Kwu3LxVZNAthjsERxWOX+PiY_RFE9ojdQPggDVyA-W0gfmQ@mail.gmail.com>
Message-ID: <CAP4Jji4qAxK8qXEfDOxzu9S+5=C4qurDfgTwROchWtOo4MZJJA@mail.gmail.com>

On Mon, Dec 8, 2014 at 3:21 PM, Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

> I'm not sure in what case code like what Skip shared wouldn't result
> in an error:
>
>     for i in range(1, 10):
>         for i in ['foo', 'frob', 'bob', 'bogus', 'smogus']:
>             do_stuff(i)
>         do_other_stuff(i)
>

A slight variation that is probably correct:

for i in range(1, 10)
  do_other_stuff(i)
  for i in ['foo', 'frob', 'bob', 'bogus', 'smogus']:
    do_stuff(i)

The difference here being that the outer `i` is used before it's reassigned
by the inner loop. We could only emit a warning if the loop variable is
re-defined if it's not yet used, but then what about this:

# https://www.youtube.com/watch?v=jrzUsHNGZHc
for _ in xrange(3):
    for _ in xrange(3):  # `_` is unused and redefined, but who cares?
        knock()
    say "Penny"

We might consider re-using a loop variable outside the loop as the problem,
but then what about:

for thing in some_things():
    if is_the_one_im_seeking(thing):
        break
else:
    raise CouldntFindTheThingException
frob(thing)

I agree, it would be good to catch this error, but I haven't thought of a
way to do it that doesn't run afoul of false positives.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141208/9586b78c/attachment.html>

From graffatcolmingov at gmail.com  Mon Dec  8 23:12:23 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 8 Dec 2014 16:12:23 -0600
Subject: [code-quality] This seems like a good warning candidate...
In-Reply-To: <CAP4Jji4qAxK8qXEfDOxzu9S+5=C4qurDfgTwROchWtOo4MZJJA@mail.gmail.com>
References: <CANc-5Uywz1TtQiOeo8o_HdoM_TaasAfQfMpbDtqL3_wwWtvOtA@mail.gmail.com>
 <CAN-Kwu2ZNhBCVCH9yt21gfnOjCzEn1jbiH_mqQF1tGgdsxzvWA@mail.gmail.com>
 <CAP4Jji5Ogvp5YOxR1v+KAGEOzyGV+KRseKT6rirNo5hiMebcOA@mail.gmail.com>
 <CANc-5UxhK84zxBXEx6c7vybNj1adBtE0igy3sPHGMq=F6H7kXw@mail.gmail.com>
 <CAN-Kwu3LxVZNAthjsERxWOX+PiY_RFE9ojdQPggDVyA-W0gfmQ@mail.gmail.com>
 <CAP4Jji4qAxK8qXEfDOxzu9S+5=C4qurDfgTwROchWtOo4MZJJA@mail.gmail.com>
Message-ID: <CAN-Kwu3hdKb76VTVCSBoyKWgObZ0N_bnJbSMkPjMY7Dm27tXCQ@mail.gmail.com>

On Mon, Dec 8, 2014 at 3:44 PM, Phil Frost <indigo at bitglue.com> wrote:
> On Mon, Dec 8, 2014 at 3:21 PM, Ian Cordasco <graffatcolmingov at gmail.com>
> wrote:
>>
>> I'm not sure in what case code like what Skip shared wouldn't result
>> in an error:
>>
>>     for i in range(1, 10):
>>         for i in ['foo', 'frob', 'bob', 'bogus', 'smogus']:
>>             do_stuff(i)
>>         do_other_stuff(i)
>
>
> A slight variation that is probably correct:
>
> for i in range(1, 10)
>   do_other_stuff(i)
>   for i in ['foo', 'frob', 'bob', 'bogus', 'smogus']:
>     do_stuff(i)
>
> The difference here being that the outer `i` is used before it's reassigned
> by the inner loop. We could only emit a warning if the loop variable is
> re-defined if it's not yet used, but then what about this:
>
> # https://www.youtube.com/watch?v=jrzUsHNGZHc
> for _ in xrange(3):
>     for _ in xrange(3):  # `_` is unused and redefined, but who cares?
>         knock()
>     say "Penny"
>
> We might consider re-using a loop variable outside the loop as the problem,
> but then what about:
>
> for thing in some_things():
>     if is_the_one_im_seeking(thing):
>         break
> else:
>     raise CouldntFindTheThingException
> frob(thing)
>
> I agree, it would be good to catch this error, but I haven't thought of a
> way to do it that doesn't run afoul of false positives.
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>

Your second example is moot because PyFlakes already silences certain
warnings if _ is the variable name. I also think you're blurring the
lines too much. Re-using the loop variable outside a loop is common
(and often intentional). This doesn't even relate to the original
request or its validity.

From amulhern at redhat.com  Tue Dec  9 00:14:51 2014
From: amulhern at redhat.com (Anne Mulhern)
Date: Mon, 8 Dec 2014 18:14:51 -0500 (EST)
Subject: [code-quality] Time cost for pylint analyses?
In-Reply-To: <CADq85CT4oR07Rv3Aexv=9aQrtMZ+kEj0ukPg7D==ja=QxvUj4w@mail.gmail.com>
References: <1302775550.8974493.1417729186785.JavaMail.zimbra@redhat.com>
 <1210013895.8974985.1417729286090.JavaMail.zimbra@redhat.com>
 <CADq85CT4oR07Rv3Aexv=9aQrtMZ+kEj0ukPg7D==ja=QxvUj4w@mail.gmail.com>
Message-ID: <1565228923.10579778.1418080491382.JavaMail.zimbra@redhat.com>

I was thinking by checker...but now that you point
it out I realize that by module being checked
would be interesting, too.

- mulhern

----- Original Message -----
> From: "Torsten Marek" <shlomme at gmail.com>
> To: "Anne Mulhern" <amulhern at redhat.com>
> Cc: "code-quality" <code-quality at python.org>
> Sent: Sunday, December 7, 2014 4:51:01 AM
> Subject: Re: [code-quality] Time cost for pylint analyses?
> 
> Hi Anne,
> 
> should be fairly simple, yes. What kind of breakdown do you mean, by
> module/package or by checker?
> 
> // Torsten
> 
> 2014-12-04 22:41 GMT+01:00 Anne Mulhern <amulhern at redhat.com>:
> 
> > Hi!
> >
> > Would it be possible to be able to extend pylint somehow to
> > record a time cost for individual pylint analyses?
> >
> > - mulhern
> > _______________________________________________
> > code-quality mailing list
> > code-quality at python.org
> > https://mail.python.org/mailman/listinfo/code-quality
> >
> 

From indigo at bitglue.com  Tue Dec  9 17:52:08 2014
From: indigo at bitglue.com (Phil Frost)
Date: Tue, 9 Dec 2014 11:52:08 -0500
Subject: [code-quality] The union of all tools
In-Reply-To: <CANc-5UwjZMJJxxW68kybbyUpLbKYP=sDmkeBPTnHkjxLf+ZVjg@mail.gmail.com>
References: <CANc-5UwjZMJJxxW68kybbyUpLbKYP=sDmkeBPTnHkjxLf+ZVjg@mail.gmail.com>
Message-ID: <CAP4Jji5GjySQDfJcYefj8z-Pq6pLfStD1vGmdkiFiNiPe9se7w@mail.gmail.com>

On Mon, Dec 8, 2014 at 4:10 PM, Skip Montanaro <skip.montanaro at gmail.com>
wrote:

>
> This is not to say that Phil doesn't have excellent reasons for his
> approach, just that there is a non-trivial trade-off in that approach.
>

It's a feature, not a bug. You can run pyflakes against some ugly code that
an intern wrote 3 years ago, and every single thing it tells you is a bug
in the program (or a bug in pyflakes). It won't complain about how the
intern didn't understand that classes should be named in CamelCase, or
gripe that every iteration variable is named "i", even though the code
works. Without any up-front effort or configuration, pyflakes will point
you immediately at the areas of code that have problems instead of sending
you off on a 2-week search-and-replace mission to rename all the variables
and twiddle whitespace which is more likely to introduce bugs than to fix
them.

The premise of pyflakes is that you are smarter than your linting tool, you
write tests to check the correctness of your program, and you audit for
readability in code review. You can also run it against really good code
without any configuration effort, and still everything you get is a real
problem in the code, and pyflakes will find things that a unit test will
not.

In fact, when I wrote pyflakes, pylint and pychecker already existed. I was
working at Divmod, which was staffed by many of the core contributors of
Twisted. You can look at the Twisted code to get an idea of the climate:
unit test coverage is nearly 100%, there is a brutal review process, and
all the developers have years of experience and not a single intern in
sight. The code quality is very high.

However, we had a few problems where sometimes we'd add an import to a
file, and suddenly the module couldn't be imported because of a cycle in
the import dependencies. Frequently, the cycle was due to an import which
served no purpose, maybe left over from a function that had moved
elsewhere. So, the first and easiest step to reducing the cycles would be
to remove all these unnecessary imports.

I tried pychecker and pylint, and they each had problems:

1) they were slow.
2) they spewed *thousands* of "problems".

Remember, this is a code base that is super well tested. There weren't
thousands of problems. I suppose I could have taken the time to figure out
how to configure these tools to not complain about how Divmod's naming
conventions didn't match the tool's defaults, and disable the dozens of
other "style suggestions" and warnings about things that "might" be bad but
which, because of the tests and code review, I knew were not. But given
that the existing tools were already slow, I figured I could do better, and
pyflakes was born.

After I wrote pyflakes and we ran it against the code, we also found some
code paths which were useless. They didn't show up in unit test code
coverage reports because they were executed, but maybe they'd calculate a
variable that never got used. We were then able to remove these bits of
useless code. Between Divmod and Twisted I had a large corpus of
well-tested, rigorously reviewed, high-quality code, so I reasoned that if
pyflakes found a false-positive in this code, it was a bug in pyflakes.

So, I'd argue that the incremental benefit of using pylint is small if you
have tests.

I should also point out that a lot of people seems to agree with this
philosophy, but still want to enforce conformance to some kind of naming
and whitespace convention, like PEP8. There's a program that does exactly
that, called (unsurprisingly) pep8. There's also flake8, which combines
pyflakes and pep8. Based on PyPI download stats, It's a fair guess that the
most popular use of pyflakes is as a dependency of flake8.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141209/524de77c/attachment.html>

From marius at gedmin.as  Fri Dec 12 09:25:05 2014
From: marius at gedmin.as (Marius Gedminas)
Date: Fri, 12 Dec 2014 10:25:05 +0200
Subject: [code-quality] flake8 and per-file doctest checking
Message-ID: <20141212082505.GA6902@platonas>

flake8 has the doctests option that enables doctest checking.
Unfortunately it's global.

In objgraph I have a file (objgraph.py) where the doctest syntax is used
for examples, and another file (tests.py) where doctests are used for
actual test code.

If I set doctests = yes, I get false positives on objgraph.py about all
the undefined names in my examples.

If I set doctests = no, I get a false positive about 'import gc' in
test.py being unused, despite gc.collect() being called in many doctests.
Also, the code in my doctests doesn't get checked.

Do you think it would make sense to be able to enable/disable doctest
checking per file?  I'm thinking either directives inside the files
themselves, say:

  # flake8: +doctests

or

  # flake8: -doctests

or perhaps a list of glob patterns in setup.cfg, e.g.

  [flake8]
  doctests = tests.py

(while keeping support for the boolean values, of course)

Marius Gedminas
-- 
We have enough youth, how about a fountain of SMART?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141212/82f8f184/attachment.sig>

From graffatcolmingov at gmail.com  Fri Dec 12 16:19:10 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Fri, 12 Dec 2014 09:19:10 -0600
Subject: [code-quality] flake8 and per-file doctest checking
In-Reply-To: <20141212082505.GA6902@platonas>
References: <20141212082505.GA6902@platonas>
Message-ID: <CAN-Kwu0=QQw54vCyn1Q3cnAxQo5OoF4177DmW7vZ32CPay7b0g@mail.gmail.com>

On Fri, Dec 12, 2014 at 2:25 AM, Marius Gedminas <marius at gedmin.as> wrote:
> flake8 has the doctests option that enables doctest checking.
> Unfortunately it's global.

This is actually a feature of pep8. It's available in flake8 because
it is available in pep8.

> In objgraph I have a file (objgraph.py) where the doctest syntax is used
> for examples, and another file (tests.py) where doctests are used for
> actual test code.
>
> If I set doctests = yes, I get false positives on objgraph.py about all
> the undefined names in my examples.
>
> If I set doctests = no, I get a false positive about 'import gc' in
> test.py being unused, despite gc.collect() being called in many doctests.
> Also, the code in my doctests doesn't get checked.
>
> Do you think it would make sense to be able to enable/disable doctest
> checking per file?

Frankly, no. The current machinery we have around `# noqa` (and `#
flake8: noqa`) is very simplistic. Pretty much if we see either of
those things, we skip either the line or the file. There's no real, on
or off switch. Having per-file options is an oft-requested feature but
the maintenance and architecture of it is beyond the scope of flake8
as a project.

> I'm thinking either directives inside the files
> themselves, say:
>
>   # flake8: +doctests
>
> or
>
>   # flake8: -doctests
>
> or perhaps a list of glob patterns in setup.cfg, e.g.
>
>   [flake8]
>   doctests = tests.py

Since this is a pep8 flag, you would have to request something like
this for pep8. This may have a better chance of being accepted (and
then once pep8 is released with the feature, you would be able to use
it in flake8 without issue).

> (while keeping support for the boolean values, of course)
>
> Marius Gedminas
> --
> We have enough youth, how about a fountain of SMART?
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>

From jbc.develop at gmail.com  Fri Dec 12 20:32:46 2014
From: jbc.develop at gmail.com (Juan BC)
Date: Fri, 12 Dec 2014 16:32:46 -0300
Subject: [code-quality] check for files with bad styles
Message-ID: <548B42DE.2090900@gmail.com>

i actually writing a test case who check code style with flake8

i interested in retrieve only the name of the files with errors not the 
error itself, so after some research i trying to capture the stdout of 
flake8 and write the next code

out, err = sys.stdout, sys.stderr
dummy = StringIO()
sys.stdout, sys.stderr = dummy, dummy
pep8 = engine.get_style_guide()
pep8.paths = [....]
report = pep8.check_files()
sys.stdout, sys.stderr = out, err

Always dummy.getvalue() comes empty. Its another way to retrieve the 
list of files who fail the check?

also i try to write my own report class who inherit from BaseQReport and 
add redefining a error method with a multiprocessing.Queue for 
communication but don't work

Sorry for my poor english grammar

regards.

From jbc.develop at gmail.com  Fri Dec 12 20:37:17 2014
From: jbc.develop at gmail.com (Juan BC)
Date: Fri, 12 Dec 2014 16:37:17 -0300
Subject: [code-quality] check for files with bad styles
Message-ID: <CAFT7ZuhJ7XAqZ4EiupoWGhUiGYzMJUzAb5jxQ2ui0ojO7m726Q@mail.gmail.com>

i actually writing a test case who check code style with flake8

i interested in retrieve only the name of the files with errors not the
error itself, so after some research i trying to capture the stdout of
flake8 and write the next code

out, err = sys.stdout, sys.stderr
dummy = StringIO()
sys.stdout, sys.stderr = dummy, dummy
pep8 = engine.get_style_guide()
pep8.paths = [....]
report = pep8.check_files()
sys.stdout, sys.stderr = out, err

Always dummy.getvalue() comes empty. Its another way to retrieve the list
of files who fail the check?

also i try to write my own report class who inherit from BaseQReport and
add redefining a error method with a multiprocessing.Queue for
communication but don't work

Sorry for my poor english grammar

regards.


-- 
Juan B Cabral
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141212/fd666609/attachment.html>

From marius at gedmin.as  Mon Dec 15 07:03:03 2014
From: marius at gedmin.as (Marius Gedminas)
Date: Mon, 15 Dec 2014 08:03:03 +0200
Subject: [code-quality] flake8 and per-file doctest checking
In-Reply-To: <CAN-Kwu0=QQw54vCyn1Q3cnAxQo5OoF4177DmW7vZ32CPay7b0g@mail.gmail.com>
References: <20141212082505.GA6902@platonas>
 <CAN-Kwu0=QQw54vCyn1Q3cnAxQo5OoF4177DmW7vZ32CPay7b0g@mail.gmail.com>
Message-ID: <20141215060303.GB5453@platonas>

On Fri, Dec 12, 2014 at 09:19:10AM -0600, Ian Cordasco wrote:
> On Fri, Dec 12, 2014 at 2:25 AM, Marius Gedminas <marius at gedmin.as> wrote:
> > flake8 has the doctests option that enables doctest checking.
> > Unfortunately it's global.
> 
> This is actually a feature of pep8. It's available in flake8 because
> it is available in pep8.

I think you meant pyflakes, not pep8.

> > In objgraph I have a file (objgraph.py) where the doctest syntax is used
> > for examples, and another file (tests.py) where doctests are used for
> > actual test code.
> >
> > If I set doctests = yes, I get false positives on objgraph.py about all
> > the undefined names in my examples.
> >
> > If I set doctests = no, I get a false positive about 'import gc' in
> > test.py being unused, despite gc.collect() being called in many doctests.
> > Also, the code in my doctests doesn't get checked.
> >
> > Do you think it would make sense to be able to enable/disable doctest
> > checking per file?
> 
> Frankly, no. The current machinery we have around `# noqa` (and `#
> flake8: noqa`) is very simplistic. Pretty much if we see either of
> those things, we skip either the line or the file. There's no real, on
> or off switch. Having per-file options is an oft-requested feature but
> the maintenance and architecture of it is beyond the scope of flake8
> as a project.
> 
> > I'm thinking either directives inside the files
> > themselves, say:
> >
> >   # flake8: +doctests
> >
> > or
> >
> >   # flake8: -doctests
> >
> > or perhaps a list of glob patterns in setup.cfg, e.g.
> >
> >   [flake8]
> >   doctests = tests.py
> 
> Since this is a pep8 flag, you would have to request something like
> this for pep8. This may have a better chance of being accepted (and
> then once pep8 is released with the feature, you would be able to use
> it in flake8 without issue).

Last time I asked for a pyflakes feature to selectively ignore some
warnings, I was told to use flake8 and annotate those lines with '#noqa'.
:)

> > (while keeping support for the boolean values, of course)
> >
> > Marius Gedminas

Marius Gedminas
-- 
The death rate on Earth is: .... (computing) .... One per person.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141215/9c24ac58/attachment.sig>

From amulhern at redhat.com  Mon Dec 15 14:24:29 2014
From: amulhern at redhat.com (Anne Mulhern)
Date: Mon, 15 Dec 2014 08:24:29 -0500 (EST)
Subject: [code-quality] Possible to make extensions subdirectory a
 separate package?
In-Reply-To: <CAN-Kwu1NhKPGopj+mzAWoq5QqcQBLa5kLrmT_X8YfFEPe9J5ew@mail.gmail.com>
References: <308313554.8965600.1417727609095.JavaMail.zimbra@redhat.com>
 <1541591574.8967139.1417727877327.JavaMail.zimbra@redhat.com>
 <CAN-Kwu1NhKPGopj+mzAWoq5QqcQBLa5kLrmT_X8YfFEPe9J5ew@mail.gmail.com>
Message-ID: <1483846737.13414412.1418649869738.JavaMail.zimbra@redhat.com>





----- Original Message -----
> Sent: Thursday, December 4, 2014 4:46:08 PM
> Subject: Re: [code-quality] Possible to make extensions subdirectory a separate package?
> 
> On Dec 4, 2014 3:19 PM, "Anne Mulhern" <amulhern at redhat.com> wrote:
> >
> > Hi!
> >
> > I think that making contents of proposed extensions subdirectory a
> separate
> > package from pylint itself would be a great convenience for users who may
> > not be ready to update to most recent version of pylint, but are
> interested
> > in some optional checker.
> >
> > Please let me know what you think.
> >
> > Thanks!
> >
> > - mulhern
> > _______________________________________________
> > code-quality mailing list
> > code-quality at python.org
> > https://mail.python.org/mailman/listinfo/code-quality
> 
> I don't use them but testing them as third-party packages could also help
> determine if and when they should become part of pylint proper. This is an
> excellent suggestion Anne!
> 

Is there anybody else besides me who thinks they might have a checker or
so to contribute to this package? I _think_ I might have two, one about
overrides of methods or attributes which give the same value as what they
override and so are effectively pointless, and one about internationalization.

- mulhern

From graffatcolmingov at gmail.com  Mon Dec 15 15:43:15 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 15 Dec 2014 08:43:15 -0600
Subject: [code-quality] flake8 and per-file doctest checking
In-Reply-To: <20141215060303.GB5453@platonas>
References: <20141212082505.GA6902@platonas>
 <CAN-Kwu0=QQw54vCyn1Q3cnAxQo5OoF4177DmW7vZ32CPay7b0g@mail.gmail.com>
 <20141215060303.GB5453@platonas>
Message-ID: <CAN-Kwu16G2uR49+Cjhiw3hSBnNAxXDm5vz+M7F40XOsG4SHwdA@mail.gmail.com>

On Mon, Dec 15, 2014 at 12:03 AM, Marius Gedminas <marius at gedmin.as> wrote:
> On Fri, Dec 12, 2014 at 09:19:10AM -0600, Ian Cordasco wrote:
>> On Fri, Dec 12, 2014 at 2:25 AM, Marius Gedminas <marius at gedmin.as> wrote:
>> > flake8 has the doctests option that enables doctest checking.
>> > Unfortunately it's global.
>>
>> This is actually a feature of pep8. It's available in flake8 because
>> it is available in pep8.
>
> I think you meant pyflakes, not pep8.

You're right. I was confusing the fact that pep8 runs doctests on
itself with the idea that it provided the --doctest option. You'll
notice though, that neither pyflakes nor pep8 actually provide a
command-line flag for this option. This is entirely something flake8
adds even though the in-built pyflakes Checker supports this.

>> > In objgraph I have a file (objgraph.py) where the doctest syntax is used
>> > for examples, and another file (tests.py) where doctests are used for
>> > actual test code.
>> >
>> > If I set doctests = yes, I get false positives on objgraph.py about all
>> > the undefined names in my examples.
>> >
>> > If I set doctests = no, I get a false positive about 'import gc' in
>> > test.py being unused, despite gc.collect() being called in many doctests.
>> > Also, the code in my doctests doesn't get checked.
>> >
>> > Do you think it would make sense to be able to enable/disable doctest
>> > checking per file?
>>
>> Frankly, no. The current machinery we have around `# noqa` (and `#
>> flake8: noqa`) is very simplistic. Pretty much if we see either of
>> those things, we skip either the line or the file. There's no real, on
>> or off switch. Having per-file options is an oft-requested feature but
>> the maintenance and architecture of it is beyond the scope of flake8
>> as a project.
>>
>> > I'm thinking either directives inside the files
>> > themselves, say:
>> >
>> >   # flake8: +doctests
>> >
>> > or
>> >
>> >   # flake8: -doctests
>> >
>> > or perhaps a list of glob patterns in setup.cfg, e.g.
>> >
>> >   [flake8]
>> >   doctests = tests.py
>>
>> Since this is a pep8 flag, you would have to request something like
>> this for pep8. This may have a better chance of being accepted (and
>> then once pep8 is released with the feature, you would be able to use
>> it in flake8 without issue).
>
> Last time I asked for a pyflakes feature to selectively ignore some
> warnings, I was told to use flake8 and annotate those lines with '#noqa'.
> :)
>

So since this is entirely within my wheelhouse (and discretion), I am
going to try to figure out the best way to allow you to skip files and
doctests without having to skip all checks. I'm heavily leaning
towards the glob patterns in the config file (whether you use tox.ini,
setup.cfg, or the other supported options).

I've created an issue for this on Flake8's issue tracker:
https://gitlab.com/pycqa/flake8/issues/16

>> > (while keeping support for the boolean values, of course)
>> >
>> > Marius Gedminas
>
> Marius Gedminas
> --
> The death rate on Earth is: .... (computing) .... One per person.
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>

From amulhern at redhat.com  Mon Dec 15 17:27:35 2014
From: amulhern at redhat.com (Anne Mulhern)
Date: Mon, 15 Dec 2014 11:27:35 -0500 (EST)
Subject: [code-quality] Whole program analysis
In-Reply-To: <1408205846.13538741.1418659775330.JavaMail.zimbra@redhat.com>
Message-ID: <1828349269.13551029.1418660855401.JavaMail.zimbra@redhat.com>

Hi!

Pretend I have classes:

class A(object):
    pass

class B(object):
    pass

class C(A,B):
    pass

class D(object):
    pass

The MRO linearizations are:

L(object) = [object]
L(A) = [A, object]
L(B) = [B, object]
L(C) = [C, A, B, object]
L(D) = [D, object]

I'll say that L(X) > L(Y) if every
c in L(Y) occurs in L(X). By that rule
L(C) and L(D) are maximal linearizations.

I want to write an analysis which would
work best if I could gather up all the
linearization of all classes in a possibly
large package distributed over possibly
many files and find only
the maximal linearizations.

This is really a kind of whole-program analysis.

I don't believe that this fits the pylint model
very well, since it is reasonable to expect that
pylint would be run separately on each individual
file in a library.

Are there other analyses around that make use of
astroid and are more whole-program like?

Thanks!

- mulhern

From pcmanticore at gmail.com  Mon Dec 15 17:49:05 2014
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Mon, 15 Dec 2014 18:49:05 +0200
Subject: [code-quality] Whole program analysis
In-Reply-To: <1828349269.13551029.1418660855401.JavaMail.zimbra@redhat.com>
References: <1408205846.13538741.1418659775330.JavaMail.zimbra@redhat.com>
 <1828349269.13551029.1418660855401.JavaMail.zimbra@redhat.com>
Message-ID: <CAMy=CLq9o2a3yVJDhABfiyrrDaH3xbBQZvqRUThHJ=DQ3RrN3w@mail.gmail.com>

On Mon, Dec 15, 2014 at 6:27 PM, Anne Mulhern <amulhern at redhat.com> wrote:
> Hi!
>
> Pretend I have classes:
>
> class A(object):
>     pass
>
> class B(object):
>     pass
>
> class C(A,B):
>     pass
>
> class D(object):
>     pass
>
> The MRO linearizations are:
>
> L(object) = [object]
> L(A) = [A, object]
> L(B) = [B, object]
> L(C) = [C, A, B, object]
> L(D) = [D, object]
>
> I'll say that L(X) > L(Y) if every
> c in L(Y) occurs in L(X). By that rule
> L(C) and L(D) are maximal linearizations.
>
> I want to write an analysis which would
> work best if I could gather up all the
> linearization of all classes in a possibly
> large package distributed over possibly
> many files and find only
> the maximal linearizations.
>
> This is really a kind of whole-program analysis.
>
> I don't believe that this fits the pylint model
> very well, since it is reasonable to expect that
> pylint would be run separately on each individual
> file in a library.
>
> Are there other analyses around that make use of
> astroid and are more whole-program like?
>
> Thanks!


Hi, Anne.

I'm currently working on adding mro support in astroid (need it to
solve a false positive in Pylint). After this, it will be easy to use
astroid for your task, by walking each file in your package, build an
astroid.module with AstroidBuilder.file_build, get all the classes
using Module.nodes_of_class and finally calling .mro for each one.
I'll push the change to astroid tonight.

From jiangzhe at weizoom.com  Wed Dec 17 16:43:06 2014
From: jiangzhe at weizoom.com (=?gb2312?B?va3V3A==?=)
Date: Wed, 17 Dec 2014 23:43:06 +0800
Subject: [code-quality] Some thing about
Message-ID: <DC47DEA3-A1F9-4388-A349-3F33304BE0B5@weizoom.com>

hi everybody,

I have a checker name webapp.py in my current directory. When I used pylint with param ?load-plugins webapp, i got an errer like this:
webapp None
Traceback (most recent call last):
  File "/usr/local/bin/pylint", line 11, in <module>
    sys.exit(run_pylint())
  File "/Library/Python/2.7/site-packages/pylint/__init__.py", line 23, in run_pylint
    Run(sys.argv[1:])
  File "/Library/Python/2.7/site-packages/pylint/lint.py", line 1193, in __init__
    linter.load_plugin_modules(self._plugins)
  File "/Library/Python/2.7/site-packages/pylint/lint.py", line 469, in load_plugin_modules
    module = load_module_from_name(modname)
  File "/Library/Python/2.7/site-packages/astroid/modutils.py", line 135, in load_module_from_name
    return load_module_from_modpath(dotted_name.split('.'), path, use_sys)
  File "/Library/Python/2.7/site-packages/astroid/modutils.py", line 178, in load_module_from_modpath
    mp_file, mp_filename, mp_desc = imp.find_module(part, path)
ImportError: No module named webapp
first line is variable part, path in the method astroid.modutils.load_module_from_modpath

Then I changed the method pylint.lint.load_plugin_modules like this and it well works well.


My pylint version is 1.4.0 and astroid version is 1.3.2, os is Mac 10.10. Maybe it?s a bug need to fixed.

Kind regards,

?? developer of 
www.weizoom.com <http://www.weizoom.com/>
email: jiangzhe at weizoom.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141217/e71dfe59/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.tiff
Type: image/tiff
Size: 60462 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141217/e71dfe59/attachment-0001.tiff>

From graffatcolmingov at gmail.com  Thu Dec 18 22:15:56 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Thu, 18 Dec 2014 15:15:56 -0600
Subject: [code-quality] [flake8] [idea] Optional Checks in Flake8
Message-ID: <CAN-Kwu3Ck6VCU7UL_KY8ox1xEhbWOKN0oeJuzMYL+6zRbQs-4A@mail.gmail.com>

Hey all,

I was working with Joe Gordon (who's subscribed to this list) and we
realized there's a very real desire for the ability to register a
check but not have it turned on by default. Thanks to some free time
today, I've been able to throw together an example of what that would
look like in flake8: https://gitlab.com/pycqa/flake8/merge_requests/14

I'm mainly looking for community and user based feedback. Are there
objections to a feature like this? Are there pitfalls? The approach is
a bit hack-ish, granted, but I doubt this is something pep8 has any
need to absorb.

Cheers,
Ian

From ch.martin at gmail.com  Mon Dec 22 02:16:57 2014
From: ch.martin at gmail.com (Chris Martin)
Date: Mon, 22 Dec 2014 01:16:57 +0000
Subject: [code-quality] Speeding up pylint
Message-ID: <CALMnaTMVL_EPdmXHnnF_kAYMR70MGh215V6DcxTHLkpf7W9zLQ@mail.gmail.com>

Pylint runs for several minutes on my project; I'd like to add it to my CI
build, but I'm not sure I want to introduce that much delay.

Is there any trick to optimizing it? Are there some particularly slow
inspections that I could disable?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141222/f6386088/attachment.html>

From skip.montanaro at gmail.com  Mon Dec 22 16:37:21 2014
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 22 Dec 2014 09:37:21 -0600
Subject: [code-quality] Speeding up pylint
In-Reply-To: <CALMnaTMVL_EPdmXHnnF_kAYMR70MGh215V6DcxTHLkpf7W9zLQ@mail.gmail.com>
References: <CALMnaTMVL_EPdmXHnnF_kAYMR70MGh215V6DcxTHLkpf7W9zLQ@mail.gmail.com>
Message-ID: <CANc-5UyRWddTmJ_dRSdu_4k83nk3Gb2GAc391=6C6ogT7x8W4A@mail.gmail.com>

On Sun, Dec 21, 2014 at 7:16 PM, Chris Martin <ch.martin at gmail.com> wrote:
> Pylint runs for several minutes on my project; I'd like to add it to my CI
> build, but I'm not sure I want to introduce that much delay.
>
> Is there any trick to optimizing it? Are there some particularly slow
> inspections that I could disable?

I'll let others identify slower checks. I've never found pylint to be
a big deal performance-wise, especially when compared to traditional
compiled systems. (Perhaps you've never built large C++ systems?)

Here are my thoughts:

* If you find it useful, why disable something just because it's a bit
slow? Would you prefer to miss possible problems in favor of running a
bit faster?

* Couldn't you just run pylint on the subset of files which have changed?

* Are you running one massive pylint across all modules in your
system? Perhaps break it into multiple runs on (possibly overlapping)
subsets of the system which would either run faster one-by-one, or
could be run in parallel.

* Doesn't this stuff run off on its own while you're busy doing other
things? Is it really holding you up, or is it just perception?

My 2?...

Skip

From pcmanticore at gmail.com  Mon Dec 22 16:27:08 2014
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Mon, 22 Dec 2014 17:27:08 +0200
Subject: [code-quality] Speeding up pylint
In-Reply-To: <CALMnaTMVL_EPdmXHnnF_kAYMR70MGh215V6DcxTHLkpf7W9zLQ@mail.gmail.com>
References: <CALMnaTMVL_EPdmXHnnF_kAYMR70MGh215V6DcxTHLkpf7W9zLQ@mail.gmail.com>
Message-ID: <CAMy=CLq24jVwL1wud7ndq+J1iAe_hju67f4BdnejrUk6OtrKug@mail.gmail.com>

On Mon, Dec 22, 2014 at 3:16 AM, Chris Martin <ch.martin at gmail.com> wrote:
> Pylint runs for several minutes on my project; I'd like to add it to my CI
> build, but I'm not sure I want to introduce that much delay.
>
> Is there any trick to optimizing it? Are there some particularly slow
> inspections that I could disable?
>

Hi,

Maybe it's related to this issue:
https://bitbucket.org/logilab/pylint/issue/395/horrible-performance-related-to-inspect.
I didn't have that much time to take a look at it, but it's on my
priority list for 1.4.1. The problem is not inspect-specific, but it's
something fishy in the inference engine, which wasn't there before.

From joe.gordon0 at gmail.com  Tue Dec 23 02:27:28 2014
From: joe.gordon0 at gmail.com (Joe Gordon)
Date: Mon, 22 Dec 2014 17:27:28 -0800
Subject: [code-quality] [flake8] [idea] Optional Checks in Flake8
In-Reply-To: <CAN-Kwu3Ck6VCU7UL_KY8ox1xEhbWOKN0oeJuzMYL+6zRbQs-4A@mail.gmail.com>
References: <CAN-Kwu3Ck6VCU7UL_KY8ox1xEhbWOKN0oeJuzMYL+6zRbQs-4A@mail.gmail.com>
Message-ID: <CAHXdxOek4meRBYNqU1ji4Occo2Me4MkYTD__Y_=fUYMn982LFQ@mail.gmail.com>

On Thu, Dec 18, 2014 at 1:15 PM, Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

> Hey all,
>
> I was working with Joe Gordon (who's subscribed to this list) and we
> realized there's a very real desire for the ability to register a
> check but not have it turned on by default. Thanks to some free time
> today, I've been able to throw together an example of what that would
> look like in flake8: https://gitlab.com/pycqa/flake8/merge_requests/14


I just tried this patch out on hacking, OpenStacks style guide flake8
plugin and it works well: http://paste.openstack.org/show/154033


>
>
> I'm mainly looking for community and user based feedback. Are there
> objections to a feature like this? Are there pitfalls? The approach is
> a bit hack-ish, granted, but I doubt this is something pep8 has any
> need to absorb.
>

I like this feature, because it is an easy way to allow repos to opt-in to
new rules without feeling like they are forced to use them. We currently
work around this issue by having local rules in each repo but this has
resulted in a lot of duplicate code.


>
> Cheers,
> Ian
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141222/8289b306/attachment.html>

From ned at nedbatchelder.com  Tue Dec 23 11:45:19 2014
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Tue, 23 Dec 2014 05:45:19 -0500
Subject: [code-quality] Speeding up pylint
In-Reply-To: <CANc-5UyRWddTmJ_dRSdu_4k83nk3Gb2GAc391=6C6ogT7x8W4A@mail.gmail.com>
References: <CALMnaTMVL_EPdmXHnnF_kAYMR70MGh215V6DcxTHLkpf7W9zLQ@mail.gmail.com>
 <CANc-5UyRWddTmJ_dRSdu_4k83nk3Gb2GAc391=6C6ogT7x8W4A@mail.gmail.com>
Message-ID: <549947BF.1080309@nedbatchelder.com>


On 12/22/14 10:37 AM, Skip Montanaro wrote:
> * Couldn't you just run pylint on the subset of files which have changed?
>
BTW, diff-cover (https://github.com/edx/diff-cover) is a tool that runs 
pylint, pep8, and coverage on just your recent changes, rather than on 
the entire working tree resulting from your changes.  This gives you an 
achievable goal (each patch should be clean) that can help move a messy 
project toward goodness in a satisfying way.

--Ned.

From graffatcolmingov at gmail.com  Sat Dec 27 19:32:09 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Sat, 27 Dec 2014 12:32:09 -0600
Subject: [code-quality] [Meta] IRC Channel Now available
Message-ID: <CAN-Kwu1e55WmVL0tmneE1p4UQEx04H830zhmoaFY78Qwsv3qrw@mail.gmail.com>

Hey all,

I've just registered ##python-code-quality on irc.freenode.net so we
have more real-time discussions there if desired.

See you there,
Ian

From flagist0 at gmail.com  Sun Dec 28 19:42:32 2014
From: flagist0 at gmail.com (Alexandr Presniakov)
Date: Sun, 28 Dec 2014 22:42:32 +0400
Subject: [code-quality] Pylint custom transformer: suppress some error
	messages
Message-ID: <CAMj0QcZ_2NMnPEWGWU4LBHLVWe2LLnNzBjcXOynt53jPv1hXZA@mail.gmail.com>

Hello!
I'm implementing pylint plugin for web2py: it executes user code in a
special environment populated with predefined objects.
My transformer adds imports for pylint not to complain about undefined
objects.
But after that pylint complains about unused-wildcard-import.
Is there any way to suppress some messages or at least to add some
'#pylint: disable=...' line to module from the transformer?

Thanks in advance, Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141228/1260c77a/attachment.html>

From jsnklln at gmail.com  Mon Dec 29 20:57:27 2014
From: jsnklln at gmail.com (Jason Killen)
Date: Mon, 29 Dec 2014 14:57:27 -0500
Subject: [code-quality] flake8 config file usage
Message-ID: <CAFe7a4Qionu00=kX2nLPv7=qXc3d8Q9fTKvOrxYPmKOjR9T_GQ@mail.gmail.com>

I've just started using flake8.  I'm using 2.2.5 (pep8: 1.5.7, pyflakes:
0.8.1, mccabe: 0.3, flake8-debugger: 1.3.2) CPython 2.7.1 on Darwin

Config files don't seem to be being read in.  I've tried defining it
(config.cfg) using --config, putting it in ., putting it in everything up
from ., and putting it in ~/.config/flake8.  No matter what I try flake8
doesn't seem to be picking them up.  Using --ignore works fine.

Here's my config file:
[flake8]
max-line-length = 200
# E251 unexpected spaces around keyword / parameter equals
# E711 comparison to None should be if cond is not None
# E501 line too long (203 > 200 characters)
# E303 too many blank lines (2)
# E712 comparison to True should be 'if cond is not True:' or 'if not cond:'
# E302 expected 2 blank lines, found 1
# E226 missing whitespace around arithmetic operator
# W291 trailing whitespace
# W293 blank line contains whitespace
# W391 blank line at end of file
# E111 indentation is not a multiple of four
# E702 multiple statements on one line (semicolon)
ignore = E251,E711,E501,E303,E712,E302,E226,W291,W293,W391,E111,E702,E265

Help please?

-- 
Jason Killen   jsnklln at gmail.com
Pain is inevitable, misery is optional.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20141229/55c03fa9/attachment.html>

From graffatcolmingov at gmail.com  Mon Dec 29 22:08:09 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 29 Dec 2014 15:08:09 -0600
Subject: [code-quality] flake8 config file usage
In-Reply-To: <CAFe7a4Qionu00=kX2nLPv7=qXc3d8Q9fTKvOrxYPmKOjR9T_GQ@mail.gmail.com>
References: <CAFe7a4Qionu00=kX2nLPv7=qXc3d8Q9fTKvOrxYPmKOjR9T_GQ@mail.gmail.com>
Message-ID: <CAN-Kwu1N2WSAEMucf-_rGqB8hyoa0ZcZjZiK=WQ+fHgA8tzV9w@mail.gmail.com>

On Mon, Dec 29, 2014 at 1:57 PM, Jason Killen <jsnklln at gmail.com> wrote:
> I've just started using flake8.  I'm using 2.2.5 (pep8: 1.5.7, pyflakes:
> 0.8.1, mccabe: 0.3, flake8-debugger: 1.3.2) CPython 2.7.1 on Darwin
>
> Config files don't seem to be being read in.  I've tried defining it
> (config.cfg) using --config, putting it in ., putting it in everything up
> from ., and putting it in ~/.config/flake8.  No matter what I try flake8
> doesn't seem to be picking them up.  Using --ignore works fine.
>
> Here's my config file:
> [flake8]
> max-line-length = 200
> # E251 unexpected spaces around keyword / parameter equals
> # E711 comparison to None should be if cond is not None
> # E501 line too long (203 > 200 characters)
> # E303 too many blank lines (2)
> # E712 comparison to True should be 'if cond is not True:' or 'if not cond:'
> # E302 expected 2 blank lines, found 1
> # E226 missing whitespace around arithmetic operator
> # W291 trailing whitespace
> # W293 blank line contains whitespace
> # W391 blank line at end of file
> # E111 indentation is not a multiple of four
> # E702 multiple statements on one line (semicolon)
> ignore = E251,E711,E501,E303,E712,E302,E226,W291,W293,W391,E111,E702,E265
>
> Help please?

Hey Jason,

pep8 handles discovery and parsing of config files for flake8. Looking
at what pep8 tries to find, your options are ~/.config/pep8,
./tox.ini, and ./setup.cfg. As for why using --config doesn't work, I
haven't tried that but that would also be a bug found in pep8. I'll
see if I can reproduce that though.

Cheers,
Ian
flake8 maintainer

From graffatcolmingov at gmail.com  Mon Dec 29 22:27:31 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 29 Dec 2014 15:27:31 -0600
Subject: [code-quality] flake8 config file usage
In-Reply-To: <CAN-Kwu1N2WSAEMucf-_rGqB8hyoa0ZcZjZiK=WQ+fHgA8tzV9w@mail.gmail.com>
References: <CAFe7a4Qionu00=kX2nLPv7=qXc3d8Q9fTKvOrxYPmKOjR9T_GQ@mail.gmail.com>
 <CAN-Kwu1N2WSAEMucf-_rGqB8hyoa0ZcZjZiK=WQ+fHgA8tzV9w@mail.gmail.com>
Message-ID: <CAN-Kwu0B2fKk5zbKEU-CaU9G7aZL8uKc5vML=8ZRAR0umcjp9Q@mail.gmail.com>

On Mon, Dec 29, 2014 at 3:08 PM, Ian Cordasco
<graffatcolmingov at gmail.com> wrote:
> On Mon, Dec 29, 2014 at 1:57 PM, Jason Killen <jsnklln at gmail.com> wrote:
>> I've just started using flake8.  I'm using 2.2.5 (pep8: 1.5.7, pyflakes:
>> 0.8.1, mccabe: 0.3, flake8-debugger: 1.3.2) CPython 2.7.1 on Darwin
>>
>> Config files don't seem to be being read in.  I've tried defining it
>> (config.cfg) using --config, putting it in ., putting it in everything up
>> from ., and putting it in ~/.config/flake8.  No matter what I try flake8
>> doesn't seem to be picking them up.  Using --ignore works fine.
>>
>> Here's my config file:
>> [flake8]
>> max-line-length = 200
>> # E251 unexpected spaces around keyword / parameter equals
>> # E711 comparison to None should be if cond is not None
>> # E501 line too long (203 > 200 characters)
>> # E303 too many blank lines (2)
>> # E712 comparison to True should be 'if cond is not True:' or 'if not cond:'
>> # E302 expected 2 blank lines, found 1
>> # E226 missing whitespace around arithmetic operator
>> # W291 trailing whitespace
>> # W293 blank line contains whitespace
>> # W391 blank line at end of file
>> # E111 indentation is not a multiple of four
>> # E702 multiple statements on one line (semicolon)
>> ignore = E251,E711,E501,E303,E712,E302,E226,W291,W293,W391,E111,E702,E265
>>
>> Help please?
>
> Hey Jason,
>
> pep8 handles discovery and parsing of config files for flake8. Looking
> at what pep8 tries to find, your options are ~/.config/pep8,
> ./tox.ini, and ./setup.cfg. As for why using --config doesn't work, I
> haven't tried that but that would also be a bug found in pep8. I'll
> see if I can reproduce that though.
>
> Cheers,
> Ian
> flake8 maintainer

So I just did the following:

$ mktmpenv
$ pip install flake8 flake8-docstrings
$ flake8 --version
2.2.5 (pep8: 1.5.7, pyflakes: 0.8.1, pep257: 0.2.1, mccabe: 0.3)
CPython 2.7.9 on Darwin
$ cd ~/sandbox/github3.py

I know my project has a lot of docstring (pep257) violations and that
one of the common ones is D100 so I wrote

[flake8]
ignore = D100

In test.cfg and did

$ flake8 | wc -l
     881
$ flake8 --config test.cfg | wc -l
     868

So I think that --config is working just fine.

If I update test.cfg to

[flake8]
ignore = D100,D101,D102

$ flake8 --config test.cfg | wc -l
     422


My final iteration of test.cfg is

[flake8]
max-line-length = 100
# Comment
ignore = D100,D101,D102

And that also works just fine.

Can you share the exact commands you're running? Perhaps this is some
weird side-effect of flake8-debugger.