[Python-Dev] automated testing system [was: Hundreds of warnings]

Skip Montanaro skip@pobox.com
Wed, 15 May 2002 18:45:10 -0500

    Neil> Now I want to build a distributed system.  There would be some
    Neil> cross-platform software that people could install on machines they
    Neil> have laying around.  The more and weirder the better.  The system
    Neil> would receive signals to start a build and regression test and
    Neil> send the results to some collector.  The signal could be based on
    Neil> the CVS checkin list.  The collector could generate summaries and
    Neil> post them to python-dev.

I created a little XML-RPC server sometime ago to catch instruction set
frequency information (-DDXPAIRs in ceval.c, etc) and store them in a
database.  I think this would be the ideal sort of interface for a
regression test reporting system.  The only change I had to make to my
application to make it "aware" was

    def send_instruction_counts(appname, email):
        if not hasattr(sys, "getdxp"):
        dxpserver = xmlrpclib.ServerProxy("http://manatee.mojam.com:7304")
        dxpserver.add_dx_info(appname, email, sys.version_info[:3],

    atexit.register(send_instruction_counts, "musi-cal", "skip@pobox.com")

As for posting to python-dev I think the results would just get lost or
would be so voluminous after awhile that people would quickly get
desensitized to them.  I'd rather see a tabular summary on a website
somewhere with some color coding (read/yellow/green) based the number or
percentage of tests that passed without problem.  If necessary, a post to
python-dev could remind people about the results page with perhaps a simple
two- or three-line summary identifying the overall change from the previous
day's or week's results, something like:

     3 new configurations
   -13 green (no fails)
   +13 yellow (< 3 fails)
   + 3 red (>= 3 fails)

I'd be willing to implement the recording and reporting stuff if someone
wants to help specify the API.