<div dir="ltr">I agree with Mr. Griffin here.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Mar 6, 2013 at 5:35 PM, Daniel Griffin <span dir="ltr"><<a href="mailto:dgriff1@gmail.com" target="_blank">dgriff1@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">What sort of speed are you looking for here? Does the ordering matter? If not then you can just do a multiprocessing Pool and call map with a chunk of the million int pairs. </div>
<div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br>
<br><div class="gmail_quote">On Wed, Mar 6, 2013 at 4:31 PM, Oren Livne <span dir="ltr"><<a href="mailto:livne@uchicago.edu" target="_blank">livne@uchicago.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Thanks so much for all your answers!<br>
<br>
I have a text file with a million int pairs, each of which can be
processed immediately. I would like to set up a queue to read
lines from the file and feed a thread pool that will process it in
parallel and output into (say) another queue, to be processed by
another thread that prints the results.<div><div><br>
<br>
On 3/6/2013 5:19 PM, Brantley Harris wrote:<br>
</div></div></div><div><div>
<blockquote type="cite">
<div dir="ltr">Whoa, back up. What are you trying to do with
threads?</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Mar 6, 2013 at 5:05 PM, Daniel
Griffin <span dir="ltr"><<a href="mailto:dgriff1@gmail.com" target="_blank">dgriff1@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Python has a GIL so threads mostly sort of
suck. Use multiprocessing, twisted or celery. </div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Mar 6, 2013 at 3:29
PM, Oren Livne <span dir="ltr"><<a href="mailto:livne@uchicago.edu" target="_blank">livne@uchicago.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear
All,<br>
<br>
I am new to python multithreading. It seems that
using threading causes a slow down with more
threads rather than a speedup. should I be using
the multiprocessing module instead? Any good
examples for threads reading from a queue with
multiprocessing?<br>
<br>
Thanks so much,<br>
Oren<br>
<br>
#!/usr/bin/env python<br>
'''Sum up the first 100000000 numbers. Time the
speed-up of using multithreading.'''<br>
import threading, time, numpy as np<br>
<br>
class SumThread(threading.Thread):<br>
def __init__(self, a, b):<br>
threading.Thread.__init__(self)<br>
self.a = a<br>
self.b = b<br>
self.s = 0<br>
<br>
def run(self):<br>
self.s = sum(i for i in xrange(self.a,
self.b))<br>
<br>
def main(num_threads):<br>
start = time.time()<br>
a = map(int, np.core.function_base.linspace(0,
100000000, num_threads + 1, True))<br>
# spawn a pool of threads, and pass them queue
instance<br>
threads = []<br>
for i in xrange(num_threads):<br>
t = SumThread(a[i], a[i + 1])<br>
t.setDaemon(True)<br>
t.start()<br>
threads.append(t)<br>
<br>
# Wait for all threads to complete<br>
for t in threads:<br>
t.join()<br>
<br>
# Fetch results<br>
s = sum(t.s for t in threads)<br>
print '#threads = %d, result = %10d, elapsed
Time: %s' % (num_threads, s, time.time() - start)<br>
<br>
for n in 2 ** np.arange(4):<br>
main(n)<br>
<br>
Output:<br>
#threads = 1, result = 4999999950000000, elapsed
Time: 12.3320000172<br>
#threads = 2, result = 4999999950000000, elapsed
Time: 16.5600001812 ???<br>
#threads = 4, result = 4999999950000000, elapsed
Time: 16.7489998341 ???<br>
#threads = 8, result = 4999999950000000, elapsed
Time: 16.6720001698 ???<br>
<br>
_______________________________________________<br>
Chicago mailing list<br>
<a href="mailto:Chicago@python.org" target="_blank">Chicago@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/chicago" target="_blank">http://mail.python.org/mailman/listinfo/chicago</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Chicago mailing list<br>
<a href="mailto:Chicago@python.org" target="_blank">Chicago@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/chicago" target="_blank">http://mail.python.org/mailman/listinfo/chicago</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Chicago mailing list
<a href="mailto:Chicago@python.org" target="_blank">Chicago@python.org</a>
<a href="http://mail.python.org/mailman/listinfo/chicago" target="_blank">http://mail.python.org/mailman/listinfo/chicago</a>
</pre>
</blockquote>
<br>
<br>
</div></div><span><font color="#888888"><pre cols="72">--
A person is just about as big as the things that make him angry.</pre>
</font></span></div>
<br>_______________________________________________<br>
Chicago mailing list<br>
<a href="mailto:Chicago@python.org" target="_blank">Chicago@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/chicago" target="_blank">http://mail.python.org/mailman/listinfo/chicago</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Chicago mailing list<br>
<a href="mailto:Chicago@python.org">Chicago@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/chicago" target="_blank">http://mail.python.org/mailman/listinfo/chicago</a><br>
<br></blockquote></div><br></div>