[medusa] Re: tweaks to asyn{core,chat}.py

[Skip Montanaro]

Thanks for the profiling code.

>> I've experienced cases where is sits for the timeout period
>> (typically 30 seconds), then adds a fd that was in the read set to
>> the write set as well (or was it the other way 'round?), after
>> which the next select returns immediately.

Sam> You're saying there are two trips through poll() when there should
Sam> be one? Or did a socket not 'fire' when it should have?

I dunno. I've only experienced it in test situations, where I feed a server
log file back to the server as fast as possible, so I've not paid much
attention to it. (Also, note that I'm using a somewhat oldish version of
ZServer. YMMV.) 

I imagine you have something similar, but just in case, here's my
beat-server.py script. It takes a medusa/ZServer log file on stdin and
throws requests at the server given by the -h argument (default, localhost).
 by default, it runs 5 simultaneous threads but this can be adjusted up or
down with the -t flag. Note the wierd time.sleep call at the bottom of the
main loop. That was my feeble attempt to keep the server from getting into
this catatonic state where it would only wake up and process one request
every 30 seconds.

Skip

--[[application/octet-stream
Content-Disposition: attachment; filename="beat-server.py"][quoted-printable]]
#!/usr/bin/env python

import sys, thread, getopt, string, httplib, urlparse, time, re, random
from threading import *

def get_page(url, sema, verbose):
now =3D time.asctime(time.localtime(time.time()))
if verbose =3D=3D 2:
sys.stderr.write("%s: %s -> " % (now, url)); sys.stderr.flush()
try:
t =3D time.time()
scheme, loc, path, params, query, frag =3D urlparse.urlparse(url)=

pfx =3D scheme + "://" + loc
h =3D httplib.HTTP(loc)
server =3D string.split(loc, ":")[0]
h.putrequest('GET', url[len(pfx):])
h.putheader('Accept', '*/*')
h.putheader('Host', server)
h.endheaders()
reply, msg, hdrs =3D h.getreply()
bytes =3D len(h.getfile().read())
t =3D time.time() - t
except:
if not verbose:
sys.stderr.write("%s -> " % (url)); sys.stderr.flush()
sys.stderr.write("failed\n"); sys.stderr.flush()
time.sleep(2)
else:
if verbose:
sys.stderr.write("%d (%d bytes, %.2f seconds)" % (reply, byte=
s, t))
if verbose < 2 and (reply >=3D 400 or t > 10):
sys.stderr.write(" %s" % (url)); sys.stderr.flush()
sys.stderr.write("\n")
sys.stderr.flush()
sema.release()

def build_url_from_log(loc, pat, f):
line =3D f.readline()
while line:
m =3D pat.search(line)
if m:
url =3D "http://%s%s" % (loc, m.group(1))
return url
line =3D f.readline()
return line

def main():

optlist, args =3D getopt.getopt(sys.argv[1:], "t:h:qv")

maxthreads =3D 5
proxy =3D ""
loc =3D "localhost"
verbose =3D 1
for opt, arg in optlist:
if opt =3D=3D "-t":
maxthreads =3D string.atoi(arg)
elif opt =3D=3D "-h":
loc =3D arg
elif opt =3D=3D "-q":
verbose =3D 0
elif opt =3D=3D "-v":
verbose =3D 2
=

sema =3D Semaphore(maxthreads)
pathpat =3D re.compile('GET\s([^\s]*)\s')
try:
while 1:
url =3D build_url_from_log(loc, pathpat, sys.stdin)
if not url: break
sema.acquire()
Thread(target=3Dget_page, args=3D(url, sema, verbose)).start(=
)
time.sleep(random.uniform(0, 1.3))
except KeyboardInterrupt:
return

if __name__ =3D=3D "__main__": main()