threading a 10 lines out of a file

alex goretoy aleksandr.goretoy at gmail.com
Mon Jan 5 01:00:50 EST 2009


Hello All,

I have a question. I'm not sure exactly as how to explain it in any other
way then the way I will explain it. So I'm sorry if it's hard to understand
exactly what it is I'm trying to do. Maybe not. Anyway. Here goes.

Lets say I have a file that looks like this.

id,name,desc,test
123,abc,testing is fun,yeah baby
456,qwe,python makes if funner, yeah baby
789,zxc,this is another line in this file, yeah baby
...
...

This file can have an unknown amount of lines in it. It can be 700,400 or
even 7,000.
I do this to read the file:

        try:
            reader = csv.reader(file(filename, "rb"))
            try:
                header = reader.next()
                self.buffer = list(reader) # total = len(self.buffer)
                self.bufferp= [dict(zip(header,line)) for line in
self.buffer]
                self.header = header
                #for row in reader:
                    #self.buffer.append(row)
                    #s,a=[],{}

                    #for j in range(len(self.buffer[0])):
                        #a[self.buffer[0][j]]=row[j]
                    #self.bufferp.append(a)
                    #i+=1
                #self.total = i-1
            except csv.Error, e:
                sys.exit('file %s, line %d: %s' % (filename,
reader.line_num, e))
        except IOError, e:
            sys.exit('file %s, IOError: %s' % (filename, e))

What I am currently doing is looping over this file, performing a function
for every line. This function isbeing called from this class __init__(),

def loop_lines(self):

        for k in range(len(self.buffer)): #for every line in csv file
            self.line=self.buffer[k]
            self.some_function(self.line)


Now, for my question. Is it possible for me to thread this scenario somehow?

So that I can set a variable that says how many lines to work on at the same
time?

lets say 10 lines at a time, once it finishes some it moves on to the next
ones that are not in the thread pool or something to that nature. Always
making sure that it works on 10 at the same time.

How would I achieve something like this in my program? Can someone please
recommend something for me. I would greatly appreciate. My program would
appreciate it too, seeing as it will be multi-threaded :) Thank you for your
help.

-Alex Goretoy
http://www.alexgoretoy.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20090105/cb992e65/attachment.html>


More information about the Python-list mailing list