[BangPypers] Co-operating threads

[Anand Balachandran Pillai]

I am not sure how many of you have had a chance to
use generators to implement "co-operating" aka "micro"
threads in Python.

Dr. David Mertz gives a good introduction to the topic in his
charming Python article here.

http://www.ibm.com/developerworks/library/l-pythrd.html

The other day I had a problem of creating a text file containing
all the paths of C/C++ source code inside a directory tree
containing rather a large number of source files (around
10 million). The normal approach I take in these cases is
to write an os.walk function which keeps writing to a list
and finally dump the list to a file. However since there could be
around a million or more entries in the list, I thought I will
whip up a solution using micro-threads with a "writer" and
a "traverser" light weight thread.

The traverser willl traverse a folder, write contents to the list
and yield wherein the scheduler will automatically call the writer.
The writer flushes the list to the file and yields wherein the
scheduler switches back to the traverser.

The code is here. It essentially makes use of the same skeleton
as the code sample presented by David in his article.

http://dpaste.com/hold/260673/

My questions are aimed at those who have some experience
working with light-weight threads/generators in Python or with
Stackless.

1. Can this be done better - Say in Python 3.0 ?
2. How does one do this in Stackless ?
3. If you have any experience with countless other Python concurrency
libraries for solving problems like this ?
4. How will you do this using "multiprocess" module ?

I hope this makes up for an interesting discussion.

Thanks,

-- 
--Anand