[Tutor] Newbie Wondering About Threads

[Lie Ryan]

On Sat, 06 Dec 2008 21:43:11 -0500, Damon Timm wrote:

> On Sat, Dec 6, 2008 at 6:25 PM, Python Nutter <pythonnutter at gmail.com>
> wrote:
>> I'm on my phone so excuse the simple reply. From what I skimmed you are
>> wrapping shell commands which is what I do all the time. Some hints. 1)
>> look into popen or subprocess in place of execute for more flexibility.
>> I use popen a lot and assigning a popen call to an object name let's
>> you parse the output and make informed decisions depending on what the
>> shell program outputs.
> 
> So I took a peak at subprocess.Popen --> looks like that's the direction
> I would be headed for parallel processes ... a real simple way to see it
> work for me was:
> 
> p2 = subprocess.Popen(["lame","--silent","test.wav","test.mp3"]) 
> p3 = subprocess.Popen(["lame","--silent","test2.wav","test2.mp3"]) 
> p2.wait()
> p3.wait()

I think when you do that (p2.wait() then p3.wait() ), if p3 finishes 
first, you wouldn't start another p3 until p2 have finished (i.e. until 
p2.wait() returns) and if p2 finishes first, you wouldn't start another 
p2 until p3 finishes (i.e. until p3.wait() returns ).

The solution would be to start and wait() the subprocessess in two 
threads. Use threading module or -- if you use python2.6 -- the new 
multiprocessing module.

Alternatively, you could do a "non-blocking wait", i.e. poll the thread.

while True:
    if p1.poll(): # start another p1
    if p2.poll(): # start another p2

> 
> top showed that both cores get busy and it takes half the time!  So
> that's great -- when I tried to add the flac decoding through stdout I
> was able to accomplish it as well ... I was mimicing the command of
> "flac --decode --stdout test.flac | lame - test.mp3" ... see:
> 
> p = subprocess.Popen(["flac","--decode","--stdout","test.flac"],
> stdout=subprocess.PIPE)
> p2 = subprocess.Popen(["lame","-","test.mp3"], stdin=subprocess.PIPE)
> p2.communicate(p.communicate()[0])
> 
> That did the trick - it worked!  However, it was *very* slow!  The
> python script has a "real" time of 2m22.504s whereas if I run it from
> the command line it is only 0m18.594s.  Not sure why this is ...
> 
> The last piece of my puzzle though, I am having trouble wrapping my head
> around ... I will have a list of files
> ["file1.flac","file2.flac","file3.flac","etc"] and I want the program to
> tackle compressing two at a time ... but not more than two at a time (or
> four, or eight, or whatever) because that's not going to help me at all
> (I have dual cores right now) ... I am having trouble thinking how I can
> create the algorithm that would do this for me ...
> 
> Thanks everyone.  Maybe after a good night's sleep it will come to me.
>  If you have any ideas - would love to hear them.