[Python-Dev] Tracker Stats

[Ezio Melotti]

On Tue, Jun 24, 2014 at 1:25 AM, A.M. Kuchling <amk at amk.ca> wrote:
> On Mon, Jun 23, 2014 at 04:12:24PM -0400, R. David Murray wrote:
>> The stats graphs are based on the data generated for the
>> weekly issue report.  I have a patched version of that
>> report that adds the bug/enhancement info.
>
> After PyCon, I started working on a scraper that would produce a bunch
> of different lists and charts.  My ideas were:
>
> * pie charts of issues by status and type.
>
> * list or histogram of open library issues by module, perhaps limited to the
>   top N modules
>

We don't have module-specific tags yet (see the core-workflow ML for
discussions about that), but I have other scripts that analyze all the
patches and divide them by module.
I didn't have time to integrate this in the tracker though.

> * list of N oldest issues with no subsequent activity (the unreviewed ones)
>

You can search for issues with only one message:
http://bugs.python.org/issue?%40sort0=activity&%40sort1=&%40group0=&%40group1=&%40columns=title%2Cid%2Cactivity%2Cstatus&%40filter=status%2Cmessage_count&status=1&message_count=1&%40pagesize=50&%40startwith=0

> * list of N people with the most open issues assigned to them
>

And then poke them with a goad until they fix them? :)

> The idea is to provide charts that help us direct effort to particular
> subsets of bugs.
>

If someone wants to experiment with and/or improve the tracker stats,
this is how it works:
1) The roundup-summary script [0] analyzes the issues once a week and
produce the weekly report and a  static JSON file [1];
2) The stats page [2] request the JSON file and uses the data to
generate the charts client-side.

Now there are two ways to improve it:
1) the easy way is just to use the roundup-summary script to expose
more of its data or to find new ones and add them to the JSON file
(and possibly to the summary too);
2) the hard way is to decouple the roundup-summary and the stats page
and either make another weekly (or daily/hourly) script to generate
the JSON file, or a template page that generates the data in
real-time.

Once the data are in the JSON file is quite easy to use jqPlot [4] to
make any kind of charts.
Keep in mind that some things are trivial to get out from the DB (e.g.
number of issues for each status/type), but other things are a bit
more complicated (e.g. things involving specific periods of time) and
currently the roundup-summary takes a few minutes to analyze all the
issues.
I also tried to include just a few useful charts on the stats page --
at first I had several more charts but then I removed them.
Feel free to ping me on IRC (#python-dev at Freenode) if you have questions.

Best Regards,
Ezio Melotti

[0]: http://hg.python.org/tracker/python-dev/file/default/scripts/roundup-summary
[1]: http://bugs.python.org/@@file/issue.stats.json
[2]: http://hg.python.org/tracker/python-dev/file/bbbe6c190a99/html/issue.stats.html#l20
[3]: http://www.jqplot.com/tests/

> --amk