Mailman 3 September 2009 - Twisted

Re: [Twisted-Python] I found an interisting comment about Twisted vs. Erlang
by Michael Handverger 01 Oct '09

01 Oct '09

> The biggest advantage of Twisted is a shorter development time, and the > availability of Python libraries. But I wonder if running something of the > scale of Facebook chat would be feasible at all, or if there are people on > this list that have run many parallelized twisted servers that saturated > many multi-core machines? In other words, are there practical examples of > large-scale Twisted deployment? My previous employer isn't quite up to Facebook's size, but they do over 1/2 billion monthly pageviews using Twisted for the data access layer on around 80 cores. I have no doubt that a Twisted app could scale a site to Facebook's size. You certainly wouldn't be without tough scaling problems to solve, but you'll have those no matter what technology you use if you want to scale to that level. Mike

1 0

Re: [Twisted-Python] Not understanding buildProtocol(), any clarification?
by exarkun＠twistedmatrix.com 01 Oct '09

01 Oct '09

On 08:16 pm, cyclops(a)speakeasy.net wrote: >Hello, I'm trying to understand and use buildProtocol(), which isn't >working as I expected, in two ways :) Question 1. If I have, on the >Server side: > >class MyFactory(Factory): > protocol = MyTestProtocol > def buildProtocol(self, addr): > p = Factory.buildProtocol(self, addr) > p.transport.write('connected.\n') > return p > >Should this work? Would the write() function execute? I thought that >'p' was a fully usable Protocol object at that point, but Python >disagrees with me :) Tells me it's a 'Nonetype' with no write() >method. If I take out the write(), everything works fine, I can >communicate between client/server, so no problems in the rest of the >code. Nope. There's no transport in buildProtocol. That happens later. You have to wait until "connectionMade" is called on the protocol before you trying to use the transport. > >Question 2. If I use buildProtocol() on the Server side to *not* make >a connection, as: > >class MyFactory(Factory): > protocol = MyTestProtocol > def buildProtocol(self, addr): > if (some condition): > return Factory.buildProtocol(self, addr) > else: > return None > >What is supposed to happen on the Client Side when I return None? I >was thinking that my factory's clientConnectionFailed() would fire, >indicating that the Server refused a connection. > >Instead, what I get is my Client's connectionMade() firing, >immediately followed by a clientConnectionLost() firing, as if the >connection was made then dropped. As you observed, doing this on the server accepts and then immediately closes the connection. There is no way to not accept the connection. This is a limitation of the underlying platform APIs. Jean-Paul

1 0

[Twisted-Python] Not understanding buildProtocol(), any clarification?
by John Crawford 01 Oct '09

01 Oct '09

Hello, I'm trying to understand and use buildProtocol(), which isn't working as I expected, in two ways :) Question 1. If I have, on the Server side: class MyFactory(Factory): protocol = MyTestProtocol def buildProtocol(self, addr): p = Factory.buildProtocol(self, addr) p.transport.write('connected.\n') return p Should this work? Would the write() function execute? I thought that 'p' was a fully usable Protocol object at that point, but Python disagrees with me :) Tells me it's a 'Nonetype' with no write() method. If I take out the write(), everything works fine, I can communicate between client/server, so no problems in the rest of the code. Question 2. If I use buildProtocol() on the Server side to *not* make a connection, as: class MyFactory(Factory): protocol = MyTestProtocol def buildProtocol(self, addr): if (some condition): return Factory.buildProtocol(self, addr) else: return None What is supposed to happen on the Client Side when I return None? I was thinking that my factory's clientConnectionFailed() would fire, indicating that the Server refused a connection. Instead, what I get is my Client's connectionMade() firing, immediately followed by a clientConnectionLost() firing, as if the connection was made then dropped. Just to check, I changed it to: buildProtocol(self, addr): return None which behaved the same way on the Client side. So what is really happening in buildProtocol? Is it making a connection to the Client side, and firing connectionMade(), no matter what? Or is something else going that I don't understand :) Thanks in advance for any clarification. John C>

1 0

[Twisted-Python] strange exception on Python 2.6 + Snow Leopard
by Phil Christensen 30 Sep '09

30 Sep '09

hey all, I just upgraded to Mac OS X 10.6 which includes Python 2.6.1. I'm running the latest SVN HEAD of Twisted, but when I run twistd I get an exception before it continues working (apparently) normally. Here's what I'm seeing: thrawn:~ phil$ twistd Traceback (most recent call last): File "/Users/phil/Workspace/Twisted/twisted/python/usage.py", line 413, in __str__ return self.getSynopsis() + '\n' + self.getUsage(width=None) File "/Users/phil/Workspace/Twisted/twisted/python/usage.py", line 449, in getUsage for (cmd, short, parser, desc) in self.subCommands: File "/Users/phil/Workspace/Twisted/twisted/application/app.py", line 675, in subCommands for plug in plugins: File "/Users/phil/Workspace/Twisted/twisted/plugin.py", line 200, in getPlugins allDropins = getCache(package) --- <exception caught here> --- File "/Users/phil/Workspace/Twisted/twisted/plugin.py", line 165, in getCache provider = pluginModule.load() File "/Users/phil/Workspace/Twisted/twisted/python/modules.py", line 381, in load return self.pathEntry.pythonPath.moduleLoader(self.name) File "/Users/phil/Workspace/Twisted/twisted/python/reflect.py", line 473, in namedAny obj = getattr(obj, n) exceptions.AttributeError: 'module' object has no attribute 'ssl_test' [continues to print usage info] I've deleted all my dropin.cache files, and grepped through my source and Twisted's for a reference to 'ssl_test', but to no avail. Like I said, it doesn't appear to be a fatal exception, but I'd like to make it go away. Any ideas? -phil

3 8

[Twisted-Python] AlreadyCalledError trying more than a server
by Antonio Beamud Montero 30 Sep '09

30 Sep '09

Hi all: I've implemented my client as comes in twisted documentation (FingerProxy example). I want to connect with several machines at the same time, send and receive data from this servers and start again (monitoring in one word). I've created a ClientFactory and a loop like this: conf['hosts'] = [list of hostnames] commands = [list of string commands, like 'status' or 'temp'] def runall(data): nodes = [] for h in conf['hosts']: mcf = manager.ClientFactory(commands) reactor.connectTCP(h, SERVER_PORT, mcf) nodes.append(mcf.deferred) dlist = defer.DeferredList(nodes) dlist.addCallback(runall) dlist.addErrback(lambda _: None) Only when the ClientFactory receives a 'bye' response, call's the d.callback('ok'). With a only host, works perfect, but trying to run against two o more hosts, the next exception is raised: File "/usr/lib/python2.6/site-packages/twisted/internet/defer.py", line 293, in _startRunCallbacks raise AlreadyCalledError twisted.internet.defer.AlreadyCalledError: In my debug console: This is the deferredList [<Deferred at 0x84c3d6c>, <Deferred at 0x852d1ac>] ... IPv4Address(TCP, 'ira', 10162) <Deferred at 0x84c3d6c> IPv4Address(TCP, 'lujuria', 10162) <Deferred at 0x84c3d6c current result: 'ok'> Twisted is invoking the same deferred, and chaining them, the 'ok' result is produced by the 'ira' callback invokation instead using everly deferred separately. What I'm doing wrong...? A lot of thanks.

3 3

[Twisted-Python] I found an interisting comment about Twisted vs. Erlang
by Alec Matusis 30 Sep '09

30 Sep '09

PS. >From my experience, the hardest problem in running large-scale Twisted servers is finding memory leaks, and to a lesser extent debugging intermittent silent server crashes. It's not easy to attach a debugger to a production sever if you are looking for sources of intermittent crashes for example. Looking for memory leaks, esp Python garbage collection problems/circular references is not easy as well, the manhole does not always help here. Again, attaching a debugger that allows to list *all* objects in the memory, is not often feasible for a production server under high load. And memory leaks often appear only under CPU utilization, due to problems with scheduling garbage collection. I believe E has a way to dynamically and meaningfully inspect ALL objects in the memory of any running process. > I think we are accounting way too much time to the unfounded > unaccountable random blathering of a troll :-) As you can see from the %CPU column, I have my reasons for concern ;) This is current copy and paste from a node with 2x quad core xeon L5420 @ 2.50GHz - 1 twistd process per core. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24448 nobody 20 0 991m 607m 2452 R 99 4.3 24764:37 twistd 28553 nobody 20 0 909m 453m 2412 R 95 3.2 1346:51 twistd 24640 nobody 20 0 1092m 676m 2452 R 93 4.8 32750:14 twistd 29900 nobody 20 0 802m 362m 2412 R 93 2.6 1180:53 twistd 24279 nobody 20 0 761m 277m 2424 R 42 2.0 13891:58 twistd 32422 nobody 20 0 1381m 962m 2372 R 10 6.9 13241:54 twistd 24210 nobody 20 0 599m 236m 2396 S 4 1.7 9063:29 twistd 29862 nobody 20 0 323m 14m 2384 S 2 0.1 71:53.50 twistd > -----Original Message----- > From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python- > bounces(a)twistedmatrix.com] On Behalf Of Laurens Van Houtven > Sent: Tuesday, September 29, 2009 12:28 PM > To: Twisted general discussion > Subject: Re: [Twisted-Python] I found an interisting comment about > Twisted vs. Erlang > > I think we are accounting way too much time to the unfounded > unaccountable random blathering of a troll :-) > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

1 0

[Twisted-Python] reactor blocked
by Jacopo Pecci 29 Sep '09

29 Sep '09

How could I get the first result as soon as ready and therefore parallelize things? I don’t want to block the reactor with further processing if It has some results ready to be redurned. thanks, Jacopo SERVER: fibo=Fibonacci() fact=pb.PBServerFactory(fibo) reactor.listenTCP(port, fact) reactor.run() CLIENT: fact=pb.PBClientFactory() reactor.connectTCP(host, port, fact) d=fact.getRootObject() n1=10000 d.addCallback(lambda obj: obj.callRemote("compute1", n1)) d.addCallback(elaborateResult1) d2=fact.getRootObject() n2=10000 d2.addCallback(lambda obj: obj.callRemote("compute2", n2)) d2.addCallback(elaborateResult2) reactor.run()

1 0

[Twisted-Python] PostgreSQL LISTEN NOTIFY example?
by Steve 29 Sep '09

29 Sep '09

My first message was sent as an HTML attachment - sorry! Hello, this is my first post to this list. I only started exploring Twisted less than a week ago. One thing that I would like to do is trigger a function via the PostgreSQL NOTIFY mechanism. Does anyone have an example or link to an example of how to correctly do this? I have something working but I am not sure if it is the best way.I would post the code but I am at work right now.

1 0

[Twisted-Python] PostgreSQL LISTEN NOTIFY Example?
by Steve 29 Sep '09

29 Sep '09

1 0

[Twisted-Python] Having touble with spread.pb un-serializing an object
by Benjamin Ash 28 Sep '09

28 Sep '09

Hi, I am trying to send new style classes that implement __slots__ with spread.pb. Things seem to go awry when attempting to un-serialize the objects on the client side. I am not sure if this is a known issue. How can I work around this issue? Thanks, -ben Stack trace: """ Unhandled error in Deferred: Traceback (most recent call last): File "/usr/lib/python-2.6/lib/python2.6/site-packages/twisted/spread/pb.py", line 514, in expressionReceived method(*sexp[1:]) File "/usr/lib/python-2.6/lib/python2.6/site-packages/twisted/spread/pb.py", line 884, in proto_answer d.callback(self.unserialize(netResult)) File "/usr/lib/python-2.6/lib/python2.6/site-packages/twisted/internet/defer.py", line 243, in callback self._startRunCallbacks(result) File "/usr/lib/python-2.6/lib/python2.6/site-packages/twisted/internet/defer.py", line 312, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/usr/lib/python-2.6/lib/python2.6/site-packages/twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "./postRelease.in", line 36, in checkMissingRpms print obj.name exceptions.AttributeError: name """ Intelerad Medical Systems Central Station 895, de la Gauchetiere W., Suite 400 Montreal, QC Canada H3B 4G1 www.intelerad.com This email or any attachments may contain confidential or legally privileged information intended for the sole use of the addressees. Any use, redistribution, disclosure, or reproduction of this information, except as intended, is prohibited. If you received this email in error, please notify the sender and remove all copies of the message, including any attachments.

2 3