threaded urllib.urlretrieve() confusion
Michael Mell
mike at nthwave.net
Thu May 9 12:54:34 EDT 2002
I'm new to threads and am confounded by what seems to be happening.
The code below wants to spawn X threads per second, each thread
containing a call to doRequest(). Immediately after each threaded call
to doRequest(), the request should begin. However, what seems to be
happening is that all the spawning happens, then all the threads begin.
It seems that all the threads are spawned in # SPAWN THREADS before and
request in # DO REQUEST initiates.
How do I get each doRequest() to begin its process as soon as I call
thread.start_new_thread (doRequest, (idStr, ))
TIA
mike
Here is the code snippet. If you have threads, it should run on your
system exactly as-is.
==================
import time, thread, urllib
base = "http://192.168.1.3/index.html"
query = {'dataID':'27346'}
data = urllib.urlencode(query)
totalRequests = 6
requestsPerSecond = 2
active = [] # each item in active represents one active thread
results = [] # we'll store the results for later
def storeResult(r):
global active, results
active.pop() # one less thread active
results.append(r)
def doRequest(id): # DO REQUEST
result = {'startTime': time.time() }
print 'start',id, result['startTime']
result['response'] = urllib.urlretrieve(base, '', '', data) # URL
RETRIEVE
storeResult(result)
return 1
processStart = time.time()
print 'processStart', processStart
for i in range(totalRequests/requestsPerSecond): # SPAWN THREADS
for j in range(requestsPerSecond):
active.append(i) # a new thread is active
idStr = 'thread second '+`i` +' id '+ `j`
print 'spawn '+ idStr
thread.start_new_thread (doRequest, (idStr, )) # START NEW
THREAD
print 'sleep', i
time.sleep(1.0) # SLEEP pause the main process before starting the
next batch
while active: # WAIT until the last thread has finished
pass
processFinish = time.time()
print 'processFinish', processFinish
tElapse = processFinish - processStart
print 'tElapse', tElapse
==========================
--
mike[at]nthwave.net
llemekim YahooIM
415.455.8812 voice
419.735.1167 fax
More information about the Python-list
mailing list