Newbie queue question

Jure Erznožnik jure.erznoznik at gmail.com
Wed Jun 17 16:00:02 CEST 2009


Hi,
I'm pretty new to Python (2.6) and I've run into a problem I just
can't seem to solve.
I'm using dbfpy to access DBF tables as part of a little test project.
I've programmed two separate functions, one that reads the DBF in main
thread and the other which reads the DBF asynchronously in a separate
thread.
Here's the code:

def demo_01():
    '''DBF read speed only'''

    dbf1 = Dbf('D:\\python\\testdbf\\promet.dbf', readOnly=1)
    for i1 in xrange(len(dbf1)):
        rec = dbf1[i1]
    dbf1.close()

def demo_03():
    '''DBF read speed into a FIFO queue'''

    class mt(threading.Thread):

        q = Queue.Queue(64)
        def run(self):
            dbf1 = Dbf('D:\\python\\testdbf\\promet.dbf', readOnly=1)
            for i1 in xrange(len(dbf1)):
                self.q.put(dbf1[i1])
            dbf1.close()
            del dbf1
            self.q.join()

    t = mt()
    t.start()
    while t.isAlive():
        try:
            rec = t.q.get(False, 0.2)
            t.q.task_done();
        except:
            pass

    del t


However I'm having serious issues with the second method. It seems
that as soon as I start accessing the queue from both threads, the
reading speed effectively halves.

I have tried the following:
1. using deque instead of queue (same speed)
2. reading 10 records at a time and inserting them in a separate loop
(hoped the congestion would help)
3. Increasing queue size to infinite and waiting 10 seconds in main
thread before I started reading - this one yielded full reading speed,
but the waiting took away all the threading benefits

I'm sure I'm doing something very wrong here, I just can't figure out
what.

Can anyone help me with this?

Thanks,
Jure



More information about the Python-list mailing list