Re: [Twisted-Python] Multicast XMLRPC
radix@twistedmatrix.com wrote:
If I use TCP and stick to the serial, synchronized semantics of RPC, doing one call at a time, I have only a few ways to solve the problem. Do one call at a time, repeat N times, and that could take quite a while. I could do M spawnProcesses and have each do N/M RPC calls. Or I could use M
do it that way. Granted I have M sockets open at a time, it is
On 06:43 pm, eprparadocs@gmail.com wrote: threads and possible for
this to take quite a while to execute. Performance would be terrible (and yes I want an approach that has good to very good performance. After all who would want poor to terrible performance?)
Let's just focus on this one thing, ignoring other resource issues for now, because I think it needs to be clarified. Maybe this isn't the case, but it looks like you're totally misunderstanding how asynchronous I/O works.
Here is an important thing: You can deal with multiple requests at the same time even with TCP. You don't need to wait for one result before you can send the next request. Send a bunch of requests at once, asynchronously, and then deal with each response as it comes in. This Just Works with multiple TCP connections in Twisted. No need for threads, no need for Broadcast or Multicast.
I understand about the state machine of Twisted and how it can do the requests. The problem is that I still get down to issuing 1000s of TCP requests (and on top of that XML-RPC, SOAP, PB or whatever). It does beg the question has anyone really used a twisted server (or client) to invoke thousands of simultaneous RPC requests? Chaz
On Fri, 25 Aug 2006 15:36:28 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
I understand about the state machine of Twisted and how it can do the requests. The problem is that I still get down to issuing 1000s of TCP requests (and on top of that XML-RPC, SOAP, PB or whatever). It does beg the question has anyone really used a twisted server (or client) to invoke thousands of simultaneous RPC requests?
Yes. I have done exactly that, while load-testing PB. Currently the only restriction I'm aware of that causes a real problem is python bug #1494314 (you can't use more than 1024 sockets at all in Python right now) but that will be fixed again in 2.4.4 and 2.5.0. My results are meaningless in your context though; you should take some sample of your application and that load and measure it. Maybe you'll find something entirely surprising is the bottleneck.
glyph@divmod.com wrote:
On Fri, 25 Aug 2006 15:36:28 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
I understand about the state machine of Twisted and how it can do the requests. The problem is that I still get down to issuing 1000s of TCP requests (and on top of that XML-RPC, SOAP, PB or whatever). It does beg the question has anyone really used a twisted server (or client) to invoke thousands of simultaneous RPC requests?
Yes. I have done exactly that, while load-testing PB. Currently the only restriction I'm aware of that causes a real problem is python bug #1494314 (you can't use more than 1024 sockets at all in Python right now) but that will be fixed again in 2.4.4 and 2.5.0.
My results are meaningless in your context though; you should take some sample of your application and that load and measure it. Maybe you'll find something entirely surprising is the bottleneck.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I guess I will have to switch to PB from XML-RPC. XML-RPC was convenient since I could write client apps in just about any language and access the system. I know it is difficult in passing things between Python and anything else. Do you have any words of wisdom on how to limit the problems so that non-Python apps could talk to a PB enabled server?
On Fri, 25 Aug 2006 16:37:30 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
glyph@divmod.com wrote:
I guess I will have to switch to PB from XML-RPC.
Not necessarily. PB is more bandwidth-efficient but there are fewer tools to work with it, so marshalling complex data structures can be quite slow. If you have small messages and a non-HTTP transport the differences may be negligible. I was mainly using it as an example of a different possible vector of optimization.
XML-RPC was convenient since I could write client apps in just about any language and access the system.
This convenience may well outweigh any real performance gain from using PB.
I know it is difficult in passing things between Python and anything else. Do you have any words of wisdom on how to limit the problems so that non- Python apps could talk to a PB enabled server?
Nope. You'd have to re-implement it. If multilanguage access is a significant concern than PB is unlikely to be a good choice.
participants (2)
-
Chaz. -
glyph＠divmod.com