[Catalog-sig] Package Quality Measurement for packages on Pypi

[Robert Kern]

On 2009-11-18 17:02 PM, David Lyon wrote:
> On Wed, 18 Nov 2009 14:33:27 -0600, Robert Kern<robert.kern at gmail.com>
> wrote:
>> Personally, I don't want to see any aggregates of incommensurable
>> observations
>> ever. I don't mind seeing a dashboard of individual observations (even if
> I
>>
>> disagree with many of the individual measurements), but aggregating them
>> with
>> arbitrary weights into a single score is simply wrong. I disagree with
>> including
>> user ratings, too, for much the same reasons.
>
> I'm not sure if CPANTS displays their findings/ratings to package users
> on CPAN either. I think you have to navigate to a seperate site to see
> the grade.
>
> The purpose of testing packages isn't to warn users off a package, say,
> because it has no docstrings. It's about taking their package, running the
> internal test suite on a number of different platforms (windows, linux,
> mac),
> checking that it installs properly with
> distribute/setuptools/distutils/pip.
>
> After that, to probe it and put some numbers (ratings) on what is
> and isn't done. Like documentation, tests, pylint, pep8.
>
> Any new package writer would expect to submit a package and get a rating
> in the C or D range (if graded with letters). With some extra polishing,
> you'd expect them to be interested in moving their package up into the
> A or B range.

Personally, I am entirely uninterested in moving up grades. I am interested in 
having good, discoverable documentation, easy and robust builds, good test 
coverage, etc. I object thoroughly to the idea that I *should* care about such a 
meaningless aggregate grade rather than the specific, individual issues 
uncovered by the tests.

> I can't see why it would be so wrong to give them tools that would allow
> them to do something like that. Otherwise, there's no incentive to try
> to make things good. Because it looks like nobody cares.

Making your package work well has a plethora of incentives that are entirely 
unrelated to, and are not helped by, an automated grading system. Of course 
people care, and all package makers understand that. If we didn't think people 
cared about our packages, we wouldn't release them.

An automated build/test/style-check service is great. By all means, continue 
your work. It provides actual value to the community. Just don't try to mash 
together a variety of disparate measurements into an arbitrary, meaningless grade.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco