[Catalog-sig] threads and xmlrpc?

[Lukasz Szybalski]

Hello,
I'm running a threaded app using some calls via xmlrpc to pypi. What
I'm trying to get is a to get a littler more responses in a shorter
time, as I see that the bandwidth used by xmlrpc calls are minimal
(<kb). The problem I run into is that connection is reset by peer
after about 10min (~500 calls). I use a single connection and a queue
of 8 threads to get the data. Would anybody have an example on how to
run xmlrpc in a thread? Do I set multiple connections, or is there a
setting to keep the connection live or reconnect if disconnected?

I've tried to catch the error but that is not working well. I don't
want to connect connect for each request, It would be best if somehow
I keep the connection live and issue multiple calls via threads.

Also, please advice if you think that somehow I am overloading your
servers. I've tasted some downloads speeds and I am sure you web
browser can accept 100+ requests per second, but what about xmlrpc?
Without threads I get <5 requests per second.

Thanks,
Lucas


Traceback (most recent call last):
 File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
   self.run()
 File "pypi.py", line 29, in run
   version=pypi.package_releases(package)[0]
 File "/usr/lib/python2.5/xmlrpclib.py", line 1147, in __call__
   return self.__send(self.__name, args)
 File "/usr/lib/python2.5/xmlrpclib.py", line 1437, in __request
   verbose=self.__verbose
 File "/usr/lib/python2.5/xmlrpclib.py", line 1185, in request
   errcode, errmsg, headers = h.getreply()
 File "/usr/lib/python2.5/httplib.py", line 1199, in getreply
   response = self._conn.getresponse()
 File "/usr/lib/python2.5/httplib.py", line 928, in getresponse
   response.begin()
 File "/usr/lib/python2.5/httplib.py", line 385, in begin
   version, status, reason = self._read_status()
 File "/usr/lib/python2.5/httplib.py", line 343, in _read_status
   line = self.fp.readline()
 File "/usr/lib/python2.5/socket.py", line 372, in readline
   data = recv(1)
error: (104, 'Connection reset by peer')


--
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
Bazaar and Launchpad
http://lucasmanual.com/mywiki/Bazaar



-- 
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
Bazaar and Launchpad
http://lucasmanual.com/mywiki/Bazaar