Mailman 3 December 2007 - Twisted

[Twisted-Python] Strange behaviour for epoll reactor
by Mustafa Sakalsiz 13 Dec '07

13 Dec '07

We have a twisted application uses the epoll reactor. Sometimes it wastes %100 cpu. It has also some threads for single isolated functions (using deferToThred). First we suspected that threads don't end and goes to an infinite loop. Because application still accepts connections and executes the requested functions. Then I ran the following ps command ps h -Leo pid,pcpu,args,lwp Looking at the output, the main thread seemed to be wasting the whole cpu time. Later on I attached the process to the strace and saw the output below. I don't how does it happen or when does it happen. It is not regular nor repeatable, but this is the strace output when it gets into loop. I send to the list, because it may be useful for someone. Now we switched back to the select reactor. P.S.: We use perspective broker on the server side and qt4reactor for the client. futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 15) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 796897}, NULL) = 0 gettimeofday({1197377318, 796940}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 15) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797127}, NULL) = 0 gettimeofday({1197377318, 797169}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 15) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797388}, NULL) = 0 gettimeofday({1197377318, 797430}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 14) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797617}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797692}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 14) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797879}, NULL) = 0 gettimeofday({1197377318, 797921}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 14) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 798139}, NULL) = 0 gettimeofday({1197377318, 798181}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 14) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 798368}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 798442}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 13) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 798627}, NULL) = 0 gettimeofday({1197377318, 798669}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 13) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0

4 4

[Twisted-Python] MultiService puzzler
by Don Smith 13 Dec '07

13 Dec '07

Hi all, I'm trying to build an app using MultiService. I need to have 2 services, one that listens for UDP datagrams and another that can send those datagrams to an ActiveMQ server via a Stomp client. I can set up the two services just fine, but I'm stuck with how to call the MQ service send method from the UDPServer instance. Some code to illustrate: class GetUDP(DatagramProtocol): def datagramReceived(self, datagram, address): print datagram,address #Send datagram to the queue server # I don't know how to access the send method for the Stomp client service! testq = internet.TCPClient(APPVARIABLES['queue_server'], int(APPVARIABLES['queue_port']), connections.StompClientFactory (username='',password='',queue='/queue/test')) testq.setName('Testq') testq.setServiceParent(MQService) #create a UDP listener #TODO: this should be driven by the config file #reactor.listenUDP(42000, GetUDP()) internet.UDPServer(42000,GetUDP()).setServiceParent(MQService) # Create an application as normal application = service.Application("MQExample") # Connect our MultiService to the application, just like a normal service. MQService.setServiceParent(application) Note: this is won't run exactly as is, but should give an idea of my problem. I've setup the ActiveMQ Stomp client as 'testq'. I've setup the UDPServer service to listen for UDP packets. I can get the UDP packets and print them, but I can't figure out how to hand them to my Stomp client's send() method. This is possible, right? Thanks, Don

2 3

[Twisted-Python] multiple mysql results in function
by Ryan McGuire 13 Dec '07

13 Dec '07

I am using xmlrpc to send and recieve messages to a "thought board" kind of thing, and being COMPLETELY new to python and twisted, I am kind of stuck on something. I have everything working except for the fact that I want to return multiple rows when I run the mysql query. Here is the code I have: from twisted.web import xmlrpc, server import MySQLdb from twisted.enterprise import adbapi class itr(xmlrpc.XMLRPC): def xmlrpc_thought(self, uname, pwrd, arg, msg): """Return the most recent thought""" if arg == "get": db=adbapi.ConnectionPool("MySQLdb",host='192.168.0.220',user='redbeardmcg',passwd='xxxxx',db='itr') db.start() return db.runQuery(""" SELECT * FROM thoughts WHERE tid > (SELECT curtid FROM users WHERE uid = (SELECT uid FROM users WHERE username='%s' AND password='%s')) """ % (uname, pwrd)).addCallback(lambda results: results[0][2]) db.stop() if arg == "post": db=adbapi.ConnectionPool("MySQLdb",host='192.168.0.220',user='redbeardmcg',passwd='xxxxx',db='itr') db.start() db.runQuery(""" INSERT INTO thoughts(uid,thought) VALUES((SELECT uid FROM users WHERE username='%s' AND password='%s'), '%s') """ % (uname, pwrd, msg)) x='Thought added' return x if __name__ == '__main__': from twisted.internet import reactor r = itr() reactor.listenTCP(8899, server.Site(r)) reactor.run() I believe something needs to be changed in the callback, but I very well may be wrong. Thanks in advance! RDMcGuire01(a)indianatech.net

3 2

[Twisted-Python] PyAMF 0.1.0 alpha
by Arnar Birgisson 13 Dec '07

13 Dec '07

Hello there, For people communicating with Flash or Flex/AIR frontends, this might be of use. cheers, Arnar We are pleased to announce the first alpha version of PyAMF. PyAMF [1] is a lightweight library that allows Flash and Python applications to communicate via Adobe's ActionScript Message Format. A summary of features in this release: * AMF0 and AMF3 encoders/decoders * Support for IExternalizable, ArrayCollection, ObjectProxy, ByteArray, RecordSet and RemoteObject * Remoting gateways for Twisted, WSGI, and Django * Authentication/setCredentials support (AMF0 only) * zlib compression support for ByteArray * Remoting client with httplib Check out the download page [2] and installation instructions [3]. Got questions? First stop is the mailing list [4]. Cheers, The PyAMF Team [1] - http://pyamf.org [2] - http://pyamf.org/wiki/Download [3] - http://pyamf.org/wiki/Install [4] - http://pyamf.org/wiki/MailingList

1 0

[Twisted-Python] (no subject)
by Raja 12 Dec '07

12 Dec '07

-Raja

1 0

[Twisted-Python] How send information (constructor data?) to a derived protocol
by Doug Farrell 12 Dec '07

12 Dec '07

Hi all, I'm trying to develop a system to handle Python logging messages via the logging.handlers.sockethandler system. What I want is a to be able to have multiple Python processes logging to the same log file, as well as a global log file. I want to do this by having a local logging server that handles all of the log messages from all of the Python processes per server. I also want to have a global logging server that handles all logging messages from all processes from all servers. I've written a class that derives from Protocol and does the right thing with the log messages generated by the logging.handler.sockethandler, and I'd like to use this class for both the local and global servers. On the global server machine there will be both services running; the local and global server. So here's where I finally get to my question. I'd like to use my derived Protocol class, but I want each instance of it (in the local server and in the global server) to use a separate logging configuration file. How do I pass information to my Protocol class so that it configures itself for the correct logging configuration file? I don't see how to pass information via the constructor, and I don't see how else to get information in there. I'm fairly new to Twisted, so maybe I'm missing something obvious. Any help would be greatly appreciated! Thanks, Doug

2 1

[Twisted-Python] number of Hack Fests reaches an all-time high
by zooko 12 Dec '07

12 Dec '07

Folks: There will be a Hack Fest this coming Friday, December 14th, at 17:00 hours Pacific Time, at the offices of allmydata.com in Potrero Hill, San Francisco. This will be the third Hack Fest, making its cardinal number one larger than the cardinal number of any previous Hack Fest! [1, 2] We already have two people lined up to show off their projects, making the number of presenters with prior planning one larger than any previous number of prepared presentations. Those two are Pavel with a Python decompiler ("WATCH as he translates .pyc files into .py files before your very eyes!") and Nejucomo with "Conway's Second Life", which is mattress of cells floating above the ground which spawn new cells and reconfigure themselves in order to play Conway's Life. Since one of these demos will be taking place in Second Life (projected onto the wall of our office in First Life), we thought it would be fun to invite people who can't physically come to San Francisco to virtually join the Hack Fest in Second Life. Please e- mail me zooko(a)zooko.com and nejucomo(a)gmail.com to coordinate. By the way, there will probably be more projects at play at the Hack Fest -- these two are just the only two that are already planned. You are welcome to bring yours! As usual, allmydata.com will generously provide drinks, eats, and funky beats. Please RSVP to zooko(a)zooko.com and secorp(a)allmydata.com. Regards, Zooko [1] http://lists.zooko.com/pipermail/p2p-hackers/2007-November/ 001412.html [2] http://lists.zooko.com/pipermail/p2p-hackers/2007-November/ 001444.html

1 0

[Twisted-Python] stopping MultiService child services
by Don Smith 11 Dec '07

11 Dec '07

Hi all, I'm still a bit of a novice but I'm trying to implement MultiService. The trick is that I want my program to be able to change it's config while it is running. Once the app realizes it has a new configuration it needs to stop all services, load the new config and start services accordingly. However, I'm stuck right now with how to stop all services. I'm thinking I need to loop over the children of MultiService and call stopService on them, but I'm not sure how to do that. Or should I just call stopServivce on the MultiService object? Thanks, Don

2 2

[Twisted-Python] Emulating urllib.read()-behaviour with twisted.web.client
by Franz Buchinger 10 Dec '07

10 Dec '07

Hello, I'm currently switching from the urllib library to twisted.web.client in my HTTP Download Class. How can I emulate urllib.read(bytes=None) behaviour with twisted.web.client? (only reading n bytes of the ressource, if the bytes parameter is given). The second difficulty (at least from my twisted newbie perspective) is to return the downloaded data to the method caller outside the class. Any hints are appreciated.. Thanks, Franz

1 0

[Twisted-Python] Please help: having problems with the simplest Twisted Server/Client Demo Example from the Tutorial
by Sajit Rao 10 Dec '07

10 Dec '07

Hello all, I've been trying the most basic (only a few lines) server/client code example in the Twisted docs and can't get even that to work (code included below). It's very frustrating, because I did get a simple server/client example working without using Twisted. Any help will be greatly appreciated. The problem is that the client never connects with the server. it prints "Started to Connect" and hangs, then times out with an error. I'm trying this on my laptop running WinXP - I first start the server program (from the command line python prompt) and then the client program in another window on the same machine. Please let me know what I'm doing wrong. Server Code: from twisted.internet.protocol import Factory, Protocol from twisted.internet import reactor class QOTD(Protocol): def connectionMade(self): self.transport.write("An apple a day keeps the doctor away\r\n") self.transport.loseConnection() factory = Factory() factory.protocol = QOTD reactor.listenTCP(8007, factory) reactor.run() Client Code: from twisted.internet.protocol import Protocol, ClientFactory from sys import stdout class Echo(Protocol): def dataReceived(self, data): stdout.write(data) class EchoClientFactory(ClientFactory): def startedConnecting(self, connector): print 'Started to connect.' def buildProtocol(self, addr): print 'Connected.' return Echo() def clientConnectionLost(self, connector, reason): print 'Lost connection. Reason:', reason def clientConnectionFailed(self, connector, reason): print 'Connection failed. Reason:', reason from twisted.internet import reactor reactor.connectTCP('localhost', 8007, EchoClientFactory()) reactor.run() Result Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import QOTDServer ..then I start the client in a separate window... Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import EchoClient Started to connect. Connection failed. Reason: [Failure instance: Traceback (failure with no frames) : <class 'twisted.internet.error.TimeoutError'>: User timeout caused connection failure. ][/b]

3 3