Mailman 3 October 2007 - Twisted

[Twisted-Python] Should I use asynchronous programming in my own modules?
by Jürgen Strass 19 Oct '07

19 Oct '07

Hello, I'm rather new to twisted and asynchronous programming in general. Overall, I think I've understood the asynchronous programming model and its implications quite well. Nevertheless, there are some remaining questions. To give some example, I'd like to develop my own simplified document format in XML and a corresponding parser. The output of the parser (a specialized document object model) will be traversed and translated into HTML afterwards. This module could be useful outside any twisted application, of course. Instead of generating HTML one could develop a generator that produces LaTeX, for example. But it could also be used to render HTML pages in a twisted web application. The question is this: since parsing and generating large documents could block the reactor in a twisted app, should I use any of twisted's asynchronous programming features in this module (for better integration with twisted) or should I rather develop it in a traditional way and run it in a thread? The question came to my mind, because somewhere I read that long lasting operations in third party modules should be called in a thread. This is clear. I also read that if one has the opportunity to develop an application from scratch, one should rather go for using twisted's asynchronous programming features and divide long lasting operations into small chunks. In principal, this approach is clear to me, but does it also apply for modules which are entirely independent from twisted networking code? And if so, is there any way to decouple them from the twisted library for reuse in other applications? The last question is what criteria I could use to divide long lasting operations into chunks. In almost all books about asynchronous programming I only read that if they're too big, they could block the event loop. Of course, but how big is too big? And what's the measure for it? Milliseconds, number of operations, number of code lines - or what? Doesn't it depend entirely on the application at hand and how reactive it has to be? Moreover, depending on the hardware used, on a Pentium II less chunks can be processed at the same time than on a Athlon 64, for example. And couldn't chunks also be too small, spending more time than necessary in putting them into the reactor's queue, then maybe sorting them and then calling them? In case the overhead involved in scheduling some chunk is bigger than the processing time of the chunk itself, the chunks are too small, aren't they? Thanks in advance for any answers, Jürgen

6 10

[Twisted-Python] Swapping a set of deferreds to diskOrganization: One Piece
by Vincent Bernat 19 Oct '07

19 Oct '07

Hi ! I have a function which returns a deferred but will take many time to really process an item. This function will be feed with a lot of items. For example, the function could be called 1000 times per second and will take 30 minutes to really process each item. However, I don't know the delay. The function could also process the items in real time. When the function takes 30 minutes to really process the item, the reactor will keep a lot of deferreds and huge portions of memory. Therefore, I would like to store myself those items (in a database for example) and wait the function to be ready to process all the items (because the function knows when it will be ready). I could store the items directly in the database and asks the function an unique deferred whose callback will be handled when the function is ready to process my items. But since, the function could be a function that process items in real time (and more faster than a database), I would like to use this "fallback" method only if the reactor has already a lot of deferred for this function. Ideally, in this case, current deferred should be converted to database. In short, I would like to be able to store/flush deferreds in database when I have a lot of them and keep them in memory if I have only a few of them. For example, a naive implementation could be to implement subclass of Deferred and each Deferred instance can store itself to disk if a lot of instances of this subclass already exists. This is not exactly what I want since I need a mechanism that will also flush all instances to disk instead of keeping them in memory. So an instance should be able to replace a lot of instances and recreate them when the function becomes ready. Does such a thing already exists? If it helps, I can guarantee that function will process items in FIFO order. Thanks. -- I WILL NOT AIM FOR THE HEAD I WILL NOT AIM FOR THE HEAD I WILL NOT AIM FOR THE HEAD -+- Bart Simpson on chalkboard in episode 8F13

1 0

[Twisted-Python] Content Encoding : gzip ?
by anurag uniyal 18 Oct '07

18 Oct '07

Is it possible to send and recieve compressed content using twisted.web? I have set Accept-encoding to gzip but it doesn't make any difference. Will I have to cater for this myself? rgds Anurag __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

3 2

Re: [Twisted-Python] [newbie] server script with client functionality
by Evert Rol 16 Oct '07

16 Oct '07

Hi Johann, Thanks for your reply. >> I'm new to Twisted, so I'm not sure where to find information on >> this. > http://twistedmatrix.com/projects/core/documentation/howto/tutorial/ > client.html I had looked at the tutorial, but admittedly not gone all the way, as it didn't seem to address my problem. But it looks like this part of the tutorial actually does. Then again, by that time, I felt a bit daunted by the Twisted framework (Protocols, Services, Factories and what not, all hooked into each other). So I've switched back to the socket module. While less neat, it does have the additional advantage that when exporting my script to other people, I don't have to ask them to install Twisted as well. But perhaps, in time, I will give Twisted another try. I'll certainly play with a few times more, to see if I can get the hang of it. Anyway, thanks again, Evert ps: ah yes, I did mean reactor.run(). I had already gotten somewhat confused with the terminology. > Evert Rol wrote: >> Hi all, >> >> I'm trying to write a server script that can pass the received >> data on >> (acting as a client) to a next server. >> Say, server A receives data from client B, and then acts as client A >> (while keeping in contact with client B) for server C. >> I'm using it to pass data through an extra machine before reaching >> the >> actual machine, where the extra machine serves as a security measure >> (eg, if that gets hacked, that won't bring too may other things >> down). >> >> >> I've been looking at creating a client at the moment that the server >> receives data (in the dataReceived() method), but then I get two >> factory.run() methods > do you mean reactor.run()? Factories usually just implement > buildProtocol, and can be used to keep references to protocols and > other > stuff (see tutorial). >> , and would also need to find a way to stop the client; that doesn't >> seem to be correct. Or perhaps using threads, but then I don't know >> how to pass the data from the server to client. > you don't need threads. >> Best to me would to have both connections active, each on their own >> port, and when data is received, 'something' in the client part gets >> called that sends these data on. >> Alternatively, I could simply call an external program with the data >> as arguments from the server script, but I'd like a all-in-one >> solution if possible. Or perhaps I'm looking at this the wrong way? >> (Although I'd prefer not to try and turn 'server C' into 'client C'.) > what do you mean by turning 'server C' into 'client C'? >> >> Any suggestions or pointers? >> > How many connections from A to C do you need? as many as for B to A? > Since you're in control of both servers it might make sense to keep > some > connections open between A and C, and save some handshake-overhead. > > The finger example comes close to what you need, you just have to make > serverprotocolA and clientprotocolA aware of each other. > > Johann >> Thanks in advance, >> >> Evert >>

1 0

[Twisted-Python] using Twisted with postgresql
by Doris Lam 16 Oct '07

16 Oct '07

Hi, I'm new to Twisted and is currently looking for a way to asynchronously do queries to a postgres database. From what I've seen it looks like there's a project called pgasync (http://jamwt.com/pgasync/) that does this, or I can use Twisted's adbapi and some other synchronous module. I'm wondering if anyone has used pgasync and how it compares to using adbapi and threads? From the pgasync site it seems the last update was in 2005, so I'm not sure if anyone's still maintaining it. If using adbapi's better, can I get some suggestions as to which postgres module I should use? So far I've seen pyPgSQL and psychopg that're both DB API 2 compliant. Any help will be appreciated! Thanks, Doris

4 3

[Twisted-Python] [newbie] server script with client functionality
by Evert Rol 16 Oct '07

16 Oct '07

Hi all, I'm trying to write a server script that can pass the received data on (acting as a client) to a next server. Say, server A receives data from client B, and then acts as client A (while keeping in contact with client B) for server C. I'm using it to pass data through an extra machine before reaching the actual machine, where the extra machine serves as a security measure (eg, if that gets hacked, that won't bring too may other things down). I'm new to Twisted, so I'm not sure where to find information on this. I've been looking at creating a client at the moment that the server receives data (in the dataReceived() method), but then I get two factory.run() methods, and would also need to find a way to stop the client; that doesn't seem to be correct. Or perhaps using threads, but then I don't know how to pass the data from the server to client. Best to me would to have both connections active, each on their own port, and when data is received, 'something' in the client part gets called that sends these data on. Alternatively, I could simply call an external program with the data as arguments from the server script, but I'd like a all-in-one solution if possible. Or perhaps I'm looking at this the wrong way? (Although I'd prefer not to try and turn 'server C' into 'client C'.) Any suggestions or pointers? Thanks in advance, Evert

2 1

[Twisted-Python] trouble with missing exceptions
by markus espenhain 15 Oct '07

15 Oct '07

hi i've written a tcp client - using a derived NetstringReceiver, a ReconnectingClientFactory and just reactor.connectTCP + reactor.run. the problem is that after the connection is made - every exception that occurred seems to be lost and the execution at this point discontinues - also with a try/catch block around it class MyProtocol(basic.NetstringReceiver): ... def onData(self, data): print data try: print 'before' # printed raise Exception('err') except Exception, e: print e # nothing print 'after' # nothing after print 'before' nothing more in onData happens - anyway the app runs and on new data the onData is entered again i poke around 2h or so and i can't remember to had this problem before ... so anybody can give me a hint or maybe something that i miss? thanks markus using twisted 2.4.0

2 2

[Twisted-Python] 2006, 'MySQL server has gone away'
by Werner Thie 15 Oct '07

15 Oct '07

Hi all While using mySQL V 5.0.33 and MySQL-python-1.2.2 with twisted/adbapi with the following connection params DB_DRIVER = "MySQLdb" DB_ARGS = { 'host': 'xxx.xxx.xxx.xxx', 'db': 'db', 'user': 'dbuser', 'passwd': 'dbpasswd' } I noticed (2006, 'MySQL server has gone away') errors, which seem to be not recoverable from an adbapi standpoint. Further analysis and reading the mySQL manual showed, that the mySQL server in versions > 5 seems to close open connections after (default) 8 hours. The reconnect feature of the db is disabled by default and there is no parameter to preset it via MySQL-python. Further digging brought up this link http://sourceforge.net/tracker/index.php?func=detail&aid=1483074&group_id=2… which patches the (also 1.2.2) version such, that a reconnect keyword is introduced in the DB_ARGS. Questions: - are there any adverse effects in applying this patch and setting reconnect: 1 in DB_ARGS? - is there a better, safer way to avoid this nasty error? Thxs, Werner

3 3

[Twisted-Python] Http PUT
by Voznesensky Vladimir 15 Oct '07

15 Oct '07

Dear Twisted authors, I need to realise an HTTP server capable of serving PUT request. There seems to be no standard twisted implementation for it. Is it possible to realise and include it into the standard twisted contribution? I have no solid idea were to start with it. Thanks, VV

8 7

[Twisted-Python] returning strings with twisted.web.xmlrpc
by Gerardo 15 Oct '07

15 Oct '07

How can I return NON-ASCII strings via xmlrpc( p.e. latin1)? Is it possible to read them using python's xmlrpclib? Thanks.

1 0