[Tutor] Multi-thread environments

[Liam Clarke]

Hi all,

I'm working in my first multi-threaded environments, and I think I
might have just been bitten by that.

I having issue with one method in a parser of mine. Only the code I
think is causing the issue is included below, I can do a full post to
RAFB if requested, but there's a bit of it.

The parser:

class Parser:
    def __init__(self, Q):
    self.Q = Q
    self.players = {}
    self.teams = {}


    def parsePack(self, packet):
        #Do stuff to packet
        #Few conditionals removed, below is example actions.
        self.players["A_value"] += self.players["A_value"]
        self.teams["As_above"] += self.players["As_above"]

    def sendData(self):
        if not self.players or not self.teams: return
        self.Q.put((self.players, self.teams))
        self.resetStats()

    def resetStats():
        for key in self.players:
            self.players[key] = 0
        for key in self.teams:
            self.teams[key] = 0

The thread in question:

class ParseThread(threading.Thread):

    def __init__(self, Q, daoQ, cfg):
        self.Q = Q
        self.parser = parserx.WatcherInTheDark(daoQ,
cfg["clan_regular_expressions"])
        self.shutdown = False
        self.dump = False
        threading.Thread.__init__(self)

    def run(self):
        print "Parser starting."
        while True:

            if self.dump:
                self.parser.sendDat()
                self.dump = False

            try:
                data = self.Q.get(False)
                self.parser.check(data)

            except Empty:

                if self.shutdown:
                    return

                continue

The variable Q being passed in is a Queue.Queue, which is used to send
data into another thread, which holds the DAO.


The sendData() method is called by a timer thread setting
Parsethread.dump to True. I was looking to avoid any asynchrous
problems via these convoluted method.

What I'm finding is that if a lot more sets of zeroed data are being
sent to the DAO than should occur.

If the resetStats() call is commented out, data is sent correctly. I
need to reset the variables after each send so as to not try and
co-ordinate state with a database, otherwise I'd be away laughing.

My speculation is that because the Queue is shared between two
threads, one of which is looping on it, that a data write to the Queue
may actually occur after the next method call, the resetStats()
method, has occurred.

So, the call to Queue.put() is made, but the actual data is accessedin
memory by the Queue after resetStats has changed it.

Am I on a right path here? I may have to do a bit of a rewrite to get
around this, as I've made some some assumptions early in the
architecture, so before I devote an hour to finding out I'm making
another wrong assumption, any hints on multi-threading in general are
welcomed and requested, and any confirmation or refutation of my
hypothesis also gladly welcomed.

I've spent about eight hours so far trying to debug this; I've never
been this frustrated in a Python project before to be honest... I've
reached my next skill level bump, so to speak.

Much thanks,

Liam Clarke