[Twisted-Python] Help requested for twisted
Hi, I am having a requirement on to send async some 300,000 or more rest api url calls.. When I work with the code for 150,000 it works fine .It takes around 8 minutes.Is it possible to improve the performance When the count increased to some 200,000 ,I am getting the error. ------------------------------------------------------------------------------------------------- Traceback (most recent call last): Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure. Unhandled error in Deferred: Unhandled Error Traceback (most recent call last): ----------------------------------------------------------------------------------------------- As my server is 8 CPU quadcore box, 2.40 Ghz, with 96G of RAM *Can you please suggest me how can I overcome this error * * * *Make the performance better* This is my piece of code where I have implemented twisted ------------------------------------------------------------------------------------------------ from twisted.internet import defer, reactor, task from twisted.web.client import getPage import sys, argparse, csv, collections, time, datetime maxRun =32 urls = "" def pageCallback(result): print len(result) return result def doWork(): for url in urls.split(','): d = getPage(url) d.addCallback(pageCallback) yield d def finish(ign): reactor.stop() def test(): deferreds = [] coop = task.Cooperator() work = doWork() for i in xrange(maxRun): d = coop.coiterate(work) deferreds.append(d) dl = defer.DeferredList(deferreds) dl.addCallback(finish) if __name__ == '__main__': filename = sys.argv[1] #I open the file ,read the files and create the url's test() reactor.run()
On 08:35 pm, ashapattani@gmail.com wrote:
Are your HTTP requests all the same server? Or are at least some of the calls to the same server? If so, if you use persistent connections (`twisted.web.client.Agent` supports these) you'll remove a bunch of the latency and overhead of those requests. Jean-Paul
Hi Jean, Thanks a lot for the quick response. All my requests are on the same server. or to the same domain. In that case do you suggest to use `twisted.web.client.Agent` I really don't require any responses from url that I send, it is fire and forget. thanks and regards, Asha On Mon, Jul 1, 2013 at 11:42 PM, <exarkun@twistedmatrix.com> wrote:
Hi , In my case the rest api url calls looks like this http://example.com/event/user_id=12345 So here how to I implement the agent for me every url has different different user_id's to be passes Thanks and regards, Asha On Tue, Jul 2, 2013 at 8:59 AM, Asha Joseph <ashapattani@gmail.com> wrote:
Hi , In my case the rest api url calls looks like this http://example.com/event/user_id=12345 So here how to I implement the agent for me every url has different different user_id's to be passes Thanks and regards, Asha On Mon, Jul 1, 2013 at 11:42 PM, <exarkun@twistedmatrix.com> wrote:
On 06:55 pm, ashapattani@gmail.com wrote:
Please don't post repeatedly. Your first message arrived at the list. You might need to wait a while before anyone volunteers to help. If enough time passes with no replies, consider posting again with more information - not just the same email. Chances are if no one has tried to help after a day or two, it's because the question wasn't clear. Jean-Paul
On 08:35 pm, ashapattani@gmail.com wrote:
Are your HTTP requests all the same server? Or are at least some of the calls to the same server? If so, if you use persistent connections (`twisted.web.client.Agent` supports these) you'll remove a bunch of the latency and overhead of those requests. Jean-Paul
Hi Jean, Thanks a lot for the quick response. All my requests are on the same server. or to the same domain. In that case do you suggest to use `twisted.web.client.Agent` I really don't require any responses from url that I send, it is fire and forget. thanks and regards, Asha On Mon, Jul 1, 2013 at 11:42 PM, <exarkun@twistedmatrix.com> wrote:
Hi , In my case the rest api url calls looks like this http://example.com/event/user_id=12345 So here how to I implement the agent for me every url has different different user_id's to be passes Thanks and regards, Asha On Tue, Jul 2, 2013 at 8:59 AM, Asha Joseph <ashapattani@gmail.com> wrote:
Hi , In my case the rest api url calls looks like this http://example.com/event/user_id=12345 So here how to I implement the agent for me every url has different different user_id's to be passes Thanks and regards, Asha On Mon, Jul 1, 2013 at 11:42 PM, <exarkun@twistedmatrix.com> wrote:
On 06:55 pm, ashapattani@gmail.com wrote:
Please don't post repeatedly. Your first message arrived at the list. You might need to wait a while before anyone volunteers to help. If enough time passes with no replies, consider posting again with more information - not just the same email. Chances are if no one has tried to help after a day or two, it's because the question wasn't clear. Jean-Paul
participants (2)
-
Asha Joseph -
exarkun＠twistedmatrix.com