<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
grocery_stocker a écrit :
<blockquote
cite="mid:d0d48afe-0f38-4dd5-b3dd-2827b57844e4@i28g2000prd.googlegroups.com"
type="cite">
<pre wrap="">On May 9, 8:36 am, Piet van Oostrum <a class="moz-txt-link-rfc2396E" href="mailto:p...@cs.uu.nl"><p...@cs.uu.nl></a> wrote:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">grocery_stocker <a class="moz-txt-link-rfc2396E" href="mailto:cdal...@gmail.com"><cdal...@gmail.com></a> (gs) wrote:
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">gs> The following code gets data from 5 different websites at the "same
gs> time".
gs> #!/usr/bin/python
gs> import Queue
gs> import threading
gs> import urllib2
gs> import time
gs> hosts = [<a class="moz-txt-link-rfc2396E" href="http://yahoo.com">"http://yahoo.com"</a>, <a class="moz-txt-link-rfc2396E" href="http://google.com">"http://google.com"</a>, <a class="moz-txt-link-rfc2396E" href="http://amazon.com">"http://amazon.com"</a>,
gs> <a class="moz-txt-link-rfc2396E" href="http://ibm.com">"http://ibm.com"</a>, <a class="moz-txt-link-rfc2396E" href="http://apple.com">"http://apple.com"</a>]
gs> queue = Queue.Queue()
gs> class MyUrl(threading.Thread):
gs> def __init__(self, queue):
gs> threading.Thread.__init__(self)
gs> self.queue = queue
gs> def run(self):
gs> while True:
gs> host = self.queue.get()
gs> if host is None:
gs> break
gs> url = urllib2.urlopen(host)
gs> print url.read(1024)
gs> #self.queue.task_done()
gs> start = time.time()
gs> def main():
gs> for i in range(5):
gs> t = MyUrl(queue)
gs> t.setDaemon(True)
gs> t.start()
gs> for host in hosts:
gs> print "pushing", host
gs> queue.put(host)
gs> for i in range(5):
gs> queue.put(None)
gs> t.join()
gs> if __name__ == "__main__":
gs> main()
gs> print "Elapsed Time: %s" % (time.time() - start)
gs> How does the parallel download work if each thread has a lock? When
gs> the program openswww.yahoo.com, it places a lock on the thread,
gs> right? If so, then doesn't that mean the other 4 sites have to wait
gs> for the thread to release the lock?
</pre>
</blockquote>
<pre wrap="">No. Where does it set a lock? There is only a short lock period in the queue
when an item is put in the queue or got from the queue. And of course we
have the GIL, but this is released as soon as a long during operation is
started - in this case when the Internet communication is done.
--
</pre>
</blockquote>
<pre wrap=""><!---->
Maybe I'm being a bit daft, but what prevents the data from <a class="moz-txt-link-abbreviated" href="http://www.yahoo.com">www.yahoo.com</a>
from being mixed up with the data from <a class="moz-txt-link-abbreviated" href="http://www.google.com">www.google.com</a>? Doesn't using
queue() prevent the data from being mixed up?
--
<a class="moz-txt-link-freetext" href="http://mail.python.org/mailman/listinfo/python-list">http://mail.python.org/mailman/listinfo/python-list</a>
</pre>
</blockquote>
Hello<br>
<br>
Each thread has a separate access to internet (its own TCP/IP
connection, port number etc.), so the incoming data will never get
mixed between the thread on the input. <br>
<br>
The only problem is when you explicitly use shared data structures be
the threads - like the queue here that they all access.<br>
But the queue is protected against multithreading so there is no
problem there (another data structure might give bugs, if not
explicitly locked before use).<br>
<br>
On the contarry, there will be mixes on the console (stdout), since
each thread can write to it at any moment. It's likely that the sources
of all the pages will get mixed, on your screen, yep. ^^<br>
<br>
Regards, <br>
pascal<br>
</body>
</html>