Mailman 3 April 2008 - Twisted

[Twisted-Python] documentation / kqueue / feedback
by Dr aio 17 Apr '08

17 Apr '08

Hi out there Let me tell you about my experience with TM so far ... Seeing there is doc. available, I digged through it. After all TM seems worth the effort. From what I see, I love TM. Links are often broken, though. Often seemingly useful doc. is old, very old. Even current doc. isn't up to date. Anyway, I try and I mean I try hard to get on my feet following some examples and snippets. The problem isn't the deferreds. I understand them pretty quickly and like them. The problem is that the docs are like a maze, irritating, frustrating and misleading rather than helping. Trying to write a rpy. Of course, it doesn't work. Looking at all those different docs, I'm bewildered. Somehow hacking my way through it. They talk about "resource trees", yet I don't find them in the doc. I find putchild() but all examples and docs indicate, that you can do it only within the root resource. Again browsing, again looking desperately for some docs, that don't mislead and make things even more fuzzy. Falling over an article with some halfway useful tap doc. OK, a little later, I have my first web server running an my first rpy works, spitting out "test". Wow. I'm wondering, why of all things "rpy" ? python won't generate sth. like "rpyc" for that, which translates in lost speed, I assume. No explanation. I'm on FreeBSD. Of course, I want to use kqueue. Now, another "funny" journey begins ... the py-kqueue in FreeBSD 6.3 and 7.0 is version 1.4. I remember, however, having read something by Isthmar Trauring Shtull (forgive me, if I didn't spell that correctly) that I need version 1.3 in order to apply his patch, which, of course (sorry, that's my disappointment carrying me away ...), doesn't work. No prob, I do it by hand. To no avail. That code is old, very old and it doesn't work. I remember havin seen some version 2 of pykqueue. Setup works nicely but Python/twisted coughs. The module has another name now. "(%"%§$( !!! So I fall back to the version 1.4 that comes with FreeBSD ports. It works. But in no-deamon mode only. As soon as I tell twistd to run web.tap deamonized it vomitts and breaks. Intense googleing, and I mean "intense". Reading through years of this mailing list. No positive result. What I say is said, because I love python an I consider twisted brilliant and immensely useful - theoretically. So, don't get me wrong and accept it as constructive, albeit somewhat pissed off remarks, OK :) I have the clear impression that twisted is something in between a toy and a brilliant product. It's hackers, however, leave much to be desired in terms of usefulness. As it is, it's a great and promising hobby but not a useful product. Sorry, but doc strings don't replcae a halfway decent documentation and a reasonable tutorial, considering the highly complex matter. Sorry, but documentation that is often enough outdated and sometimes offers broken links is next to useless. Sorry, but the most brilliant code is a lot less attractive and useful, if even "stable versions" are broken, in flux, etc ... Kqueue seems to be vital to an event driven approach like TM, yet there are multiple versions of pykqueue floating around, none of them properly working (and I didn't fumble around. I plain simply used the FreeBSD port), some of them with some mor and some with some less "annotations" (I refuse to call that doc.) - How about getting 1 version of pykqueue properly running and into TM ? - How about freezing some TM version (like 8.0) and updating/matching docs? (Of course, experiments are funny and intriguing, but quite some of us out here need sth. stable for everydays work) - How about writing some complete docs and tutorials? Sth. along the lines of "My first web server wit TM" (instead all those - sometimes seeming to contradict each other - snippets) I'm still fascinated by TM and still willingto invest time, efforts and brains. But I'd love to have some properly working base and docs that expßlain rather than confuse. Looking forward to learn and enjoy. (Yes, Should someone need sth. tested on FreeBSD, I'll gladly be of service)

11 20

[Twisted-Python] Is there a simple Producer/Consumer example or tutorial?
by Gabriel Rossetti 17 Apr '08

17 Apr '08

Hello all, I was wondering if there is a simple example or a tutorial for the Producer-Consumer stuff in twisted? Thanks, Gabriel

3 4

[Twisted-Python] documentation / kqueue / feedback
by Dr aio 17 Apr '08

17 Apr '08

First: THANKS for reacting so friendly and civilized :) Cliff > Have you tried the finger tutorial. Sure. When I say, I DID read what's available, then I did. Otherwise I wouldn't dare to complain. Thomas Hervé <therve(a)free.fr> wrote: > ... > > - How about getting 1 version of pykqueue properly running and into TM ? > > The current one is grossly working, even though there are some > problems, and that the setup is not triavial. But a new reactor has > been developed, using Python 2.6 KQueue wrapper, and should be > available some day. See http://twistedmatrix.com/trac/ticket/1918 for > the progress on this field. Any tests and feedback would help this > going further. Great. Thanks. PLEASE test it properly and prepare a reasonably conmplete readme at least. And kindly see below for my current problems. mike(a)mkp.ca wrote: > Feel better now? ;-) > > I understand your pain. Twisted is very much written by hackers for > hackers, with the old school practices of source and oral tradition in > place of documentation. It took me many attempts before I felt I was > competent, and I still can't do anything without constantly consulting > the source code. I confess, I haven't bothered to look at the docs > for years, and I suspect it is the same with the core Twisted > developers. > > Life is made easier by having a mentor who can look over your work and > point out misunderstandings and antipatterns. I find the mailing list > to be a mixed bag as a replacement. Sometimes you get real gems from > Glyph and company, and sometimes you just get a hard time. I'd be > happy to help if I can, but any deep thinking will probably have to > wait for weekends. I'm also a long time FreeBSD user and Python > developer. > > Hope you make it over the curve! Thanks, Mike, for your friendly welcome. Yes, I do understand that attitude. I love to have it, too. BUT: twisted is having and looking for sponsors, that is, twisted is - gladly ! - meant to be a product and not some hackers private delight (only). You see, I argue for using twisted. Possibly some of us argue against other solutions and, worst case, they risk their ass doing so. Again, I share your attitude and i understand it so well. But, PLEASE, understand that I need to get sth. working, that some colleagues put a lot of trust, valuable trust, into twisted, even taking a risk. That should be seen and properly respnonded to, too. I'm not talking about script kiddies. I'm talking about seasoned colleages with a (couple) dozend years in the job. We don't need pampering, not even mentoring, practical and welcome as that might be. We need realiabilty and a _fair_ chance to find our ways in twisted. A reasonable starting point. -------- Here's what I tried today: Using py-kqueue (1.4) in FreeBSDs ports. 'twistd -r kqueue -no -f web.tap' works fine. 'twistd -r kqueue -f web tap' breaks. Next I checked, wether the changes proposed by IST where in version 1.4. Nope. So I edited them into the patch that comes with the port anyway (patch-kqsyscallmodule). If anyone is interested, I'll post it. Same as without, i.e. it works but it breaks if run as daemon. Here's what I find in the log: Log opened. twistd 2.5.0 (/usr/local/bin/python2.5 2.5.2) starting up reactor class: <class 'twisted.internet.kqreactor.KQueueReactor'> Loading web.tap... Loaded. Changing process name to test Log opened. twistd 2.5.0 (/home/www/tm/ 2.5.2) starting up reactor class: <class 'twisted.internet.kqreactor.KQueueReactor'> Loading web.tap... Loaded. twisted.web.server.Site starting on 80 Starting factory <twisted.web.server.Site instance at 0x85935ec> Traceback (most recent call last): File "/usr/local/bin/twistd", line 21, in <module> run() File "/usr/local/lib/python2.5/site-packages/twisted/scripts/twistd.py", line 27, in run app.run(runApp, ServerOptions) File "/usr/local/lib/python2.5/site-packages/twisted/application/app.py", line 379, in run runApp(config) File "/usr/local/lib/python2.5/site-packages/twisted/scripts/twistd.py", line 23, in runApp _SomeApplicationRunner(config).run() File "/usr/local/lib/python2.5/site-packages/twisted/application/app.py", line 158, in run self.postApplication() File "/usr/local/lib/python2.5/site-packages/twisted/scripts/_twistd_unix.py", line 213, in postApplication startApplication(self.config, self.application) File "/usr/local/lib/python2.5/site-packages/twisted/scripts/_twistd_unix.py", line 174, in startApplication service.IService(application).privilegedStartService() File "/usr/local/lib/python2.5/site-packages/twisted/application/service.py", line 228, in privilegedStartService service.privilegedStartService() File "/usr/local/lib/python2.5/site-packages/twisted/application/service.py", line 228, in privilegedStartService service.privilegedStartService() File "/usr/local/lib/python2.5/site-packages/twisted/application/internet.py", line 68, in privilegedStartService self._port = self._getPort() File "/usr/local/lib/python2.5/site-packages/twisted/application/internet.py", line 86, in _getPort return getattr(reactor, 'listen'+self.method)(*self.args, **self.kwargs) File "/usr/local/lib/python2.5/site-packages/twisted/internet/posixbase.py", line 467, in listenTCP p.startListening() File "/usr/local/lib/python2.5/site-packages/twisted/internet/tcp.py", line 750, in startListening self.startReading() File "/usr/local/lib/python2.5/site-packages/twisted/internet/abstract.py", line 265, in startReading self.reactor.addReader(self) File "/usr/local/lib/python2.5/site-packages/twisted/internet/kqreactor.py", line 98, in addReader self._updateRegistration(fd, EVFILT_READ, EV_ADD) File "/usr/local/lib/python2.5/site-packages/twisted/internet/kqreactor.py", line 89, in _updateRegistration kq.kevent([kevent(*args)], 0, 0) OSError: [Errno 9] Bad file descriptor

3 3

[Twisted-Python] My memory leak, diagnosed
by Matthew Glubb 17 Apr '08

17 Apr '08

Hi All, I just wanted to put your minds at rest regarding what I though might have been a problem with the epoll reactor. As always, inferring anything from a production machine is not desirable and it seems that my assumptions about epoll were based on coincidence. I have since found the leak: <http://twistedmatrix.com/trac/ticket/3188> Unfortunately, my workaround involved rewriting my proxy code without using twistd.web. I'll happily contribute a patch back to the t.w.proxy code in time but for the moment it doesn't support persistent connections (I wanted to keep it simple). I also feel that t.w.proxy would be better served by making use of a reciprocal IProducer / IConsumer arrangement between the ProxyServer protocol and the ProxyClient protocol which is something that my workaround does not implement. Thanks again for Twisted, Matt Matthew Glubb Technical Partner email: matthew.glubb(a)madebykite.com phone: 44 (0) 7715 754017 skype: mglubb Kite http://madebykite.com -- GPG: 96FF DE0E 0B7B 37F0 7F8D C54C E285 3D8F 5625 9244

2 1

Re: [Twisted-Python] Twisted Advice
by Paul Wilson 17 Apr '08

17 Apr '08

> > > >I would like to provide an environment where developers can write call > >servicing applications with no knowledge of the network. I had rested > >on a coroutine approach, whereby a developer could write something > >like this: > > > > response, events = (yield getKeyPress(SomePromptAnnouncement)) > > > > Just to clarify - this looks like a generator rather than a coroutine. > > >With the response, the developer can do an undefined number of things, > >taking an undefined amount of time. During this time, management > >events can arrive on a separate port, to which the developer gets > >knowledge of through the yield statement. > > > >On the other side of this generator is a scheduler which takes > >instructions such as "getKeyPress" and passes it through twisted to > >the remote client, such that it can play the announcement and get a > >keypress. The scheduler gets the response and sends to back to the > >generator, along with any events that have arrived on this separate > >port. > > > >When a call/connection arrives, the client will send an opening > >line(s) to which I was specify the correct LineRecieved method. This > >will trigger some scheduler code defined elsewhere via a deferred, > >which will prompt the developer's code for some instructions, such as > >"PlayPrompt". Am I correct in thinking that while a developer's code > >is executing, all other connections are paused, and that the twisted > >server will not accept new connections until it returns? > > Yes, that's correct. Twisted is single-threaded - it invokes event > handlers in the same thread it uses to monitor event sources. As long > as an event handler is running, event sources are not monitored for > events and no other handlers will be invoked. > > > > >My original assumption was that Twisted would spawn a new thread > >within which the scheduler would be set to run to manage the > >communication for the duration of the customer call/interaction. > >However, the approach taken by twisted is that if the developer's code > >got itself in a muddle or infinite loop or took a very long time > >accessing a database, the rest of the application would just be > >frozen. > > Generally speaking, Twisted will only create a thread when application > code asks it to. > > > > >I wrote a quick test app that defers to a function that sleeps then > >returns its response line. During the 20 seconds that the test server > >took to return a line, the server would not accept any other requests > >until the sleep function ended. > > The "defers to a function" language is a bit confusing. However, your > conclusion sounds right. If you block the reactor thread, nothing else > will happen. > > >Clearly, I must be missing something; I have designed my application > >upon the wrong axis. If so, I have some misunderstanding as to how to > >properly structure an application like this upon Twisted. Or, twisted > >is a framework whereby the response to a network event is expect to > >arrive immediately. > > It's not necessary for responses to arrive immediately. Responses just > need to be obtained without blocking. For example, if getting a response > involves talking to someone else over the network, then Twisted has some > networking APIs that don't block. ;) Likewise, twisted.enterprise lets > you talk to a rdbm (it actually uses threads, but it mostly doesn't show > that to you). > >I have seen "deferToThread" mentioned but I cannot find enough > >documentation to decide whether it's what I need or not. Or perhaps > >callInThread() is what I need? > > One of these may be appropriate. You could decide that in your application > framework, user code is always run in a non-reactor thread. This is easily > accomplished: your event handler just needs to invoke user code in a thread > instead of directly. deferToThread lets you run a function in a thread and > gives you a Deferred which will be called back with the return value of the > function when it returns or which will errback if it raises an exception. > callInThread is lower level, not exposing the result of the function. Ahh. Okay; thanks for this. deferToThread looks like a good option here and seems to fit what I'm looking for. From experience, can you point me towards some good code that uses deferToThread? > > > >Any suggestions about this would be very much appreciated. Even > >better, if anybody knows of good documentation/tutorials they can > >point me to, that would be appreciated also. > > It sounds like you're looking for deferToThread or one of its friends. > Note however that just throwing user code into a thread doesn't mean > programmers can be oblivious to their environment. It simply trades the > need to pay attention to when you block and for how long for the more > complex task of paying attention to thread safety. If each task is > isolated from all others, this may be a good trade-off to make, but if > there's shared state it may not be. Thanks for your reply; there are some subtle and difficult questions to answer here. I'd like each instance of the developer's code to be isolated with it's own state using threading.local, but ultimately I'll use deferToThread to set it all up. I assume that I can use some threadsafe implementation (if the GIL doesn't enforce that anyway) of a dictionary to store the global information between each call handling thread. Thanks again for your time. > Jean-Paul

2 1

[Twisted-Python] Explicitly closing an AMP connection?
by Nathan 16 Apr '08

16 Apr '08

What's the "twisted" way of cleanly closing an AMP connection? The closest thing I can find on http://twistedmatrix.com/documents/current/api/twisted.protocols.amp.AMP.ht… is stopProducing() -- but it's undocumented. Will that kill the connection? ~ Nathan

3 3

Re:[Twisted-Python] Twisted Advice
by Andrew Francis 16 Apr '08

16 Apr '08

Hi Paul: >I would like to provide an environment where >developers can write call servicing applications with >no knowledge of the network. I had rested on a >coroutine approach, whereby a developer could write >something like this: I posted some code a little while ago that has WS-BPEL logic riding on top of Stackless and Twisted. >Am I correct in thinking that while a developer's code >is executing, all other connections are paused, and >that the twisted server will not accept new >connections until it returns? [From the Schmitt Reactor Pattern Paper] >Non-preemptive. In a single-threaded application >process, event handlers are not preempted while they >are executing. This implies that an event handler >should not perform blocking I/O on an individual >handle since this will block the entire process and >impede the responsiveness for clients connected to >other handles. If you accept the Schmitt explanation, I would suggest that it is convenient to think of Twisted as a non-preemptive threading system where callbacks and server protocol instances are user space "threads of execution" and the reactor is the scheduler. Since a "thread" cannot be involuntarily be pre-empted, the reactor will not have a chance to trigger callbacks. Moral of the story - you don't want to do long CPU intensive work in a callback. >My original assumption was that Twisted would spawn a >new thread within which the scheduler would be set to >run to manage the communication for the duration of >the customer call/interaction. I think you want reactor.callInThread(). I remember Philip Mayers telling me about this in a post about a year ago. Here is an example - forget the Stackless stuff http://stacklessexamples.googlecode.com/svn/trunk/examples/twisted/TwistedW… (Actually I would rather like to forget that piece of code altogether) However you may want to think over your design if you find you need to extensively use OS threads. I would be tempted to ask myself questions like: 1)Is my application CPU or I/O bound"? 2)Can I encapsulate the message exchange pattern between the client and the server be done in a single protocol"? 3)Can I conveniently yield the thread at predictable points? >Or, twisted is a framework whereby the response to a >network event is expect to arrive immediately. I think the exact opposite is true, that is why the deferred object exists. I would suggest Twisted is ideal for reactive, I/O bound applications particularly when you can represent the message exchange pattern in a single protocol. Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0

[Twisted-Python] Twisted Advice
by Paul Wilson 16 Apr '08

16 Apr '08

Hi! I am researching technologies for building an application development framework with Python, and have been interested in Twisted to manage the network communication side of things. However, I read something yesterday that appears to undermine the applicability of twisted for my work. I've done quite a bit of reading around the subject since, and have ended up confusing myself! A remote client accesses my server to get instructions on how to progress a customer "interaction." The remote client is itself a server driven by a telephone call and keytone input. The remote client could be servicing N calls (channels) simultaneously, and thus my server needs to support N simultaneous TCP connections. Pretty standard stuff. I would like to provide an environment where developers can write call servicing applications with no knowledge of the network. I had rested on a coroutine approach, whereby a developer could write something like this: response, events = (yield getKeyPress(SomePromptAnnouncement)) With the response, the developer can do an undefined number of things, taking an undefined amount of time. During this time, management events can arrive on a separate port, to which the developer gets knowledge of through the yield statement. On the other side of this generator is a scheduler which takes instructions such as "getKeyPress" and passes it through twisted to the remote client, such that it can play the announcement and get a keypress. The scheduler gets the response and sends to back to the generator, along with any events that have arrived on this separate port. When a call/connection arrives, the client will send an opening line(s) to which I was specify the correct LineRecieved method. This will trigger some scheduler code defined elsewhere via a deferred, which will prompt the developer's code for some instructions, such as "PlayPrompt". Am I correct in thinking that while a developer's code is executing, all other connections are paused, and that the twisted server will not accept new connections until it returns? My original assumption was that Twisted would spawn a new thread within which the scheduler would be set to run to manage the communication for the duration of the customer call/interaction. However, the approach taken by twisted is that if the developer's code got itself in a muddle or infinite loop or took a very long time accessing a database, the rest of the application would just be frozen. I wrote a quick test app that defers to a function that sleeps then returns its response line. During the 20 seconds that the test server took to return a line, the server would not accept any other requests until the sleep function ended. Clearly, I must be missing something; I have designed my application upon the wrong axis. If so, I have some misunderstanding as to how to properly structure an application like this upon Twisted. Or, twisted is a framework whereby the response to a network event is expect to arrive immediately. I have seen "deferToThread" mentioned but I cannot find enough documentation to decide whether it's what I need or not. Or perhaps callInThread() is what I need? Any suggestions about this would be very much appreciated. Even better, if anybody knows of good documentation/tutorials they can point me to, that would be appreciated also. Regards, Paul

2 1

[Twisted-Python] Re: about smtp client auth problem.
by Chris 16 Apr '08

16 Apr '08

1 0

[Twisted-Python] Is there twisted.enterprise.adbapi support for .executemany()?
by Nathan 15 Apr '08

15 Apr '08

Does anyone know of any (possibly experimental) twisted.enterprise.adbapi support for .executemany() ? (see http://www.python.org/dev/peps/pep-0249/ -- search for ".executemany") If not, I'll see if I can't add a runOperationMany() to twisted.enterprise.adbapi.ConnectionPool. The functionality I want is simple in theory. .executemany runs many identical insert/update/delete operations using a different dictionary in a supplied list of dictionaries for the input each time, and returns None on success of all of the queries, or rolls-back the whole transaction and returns a failure on an error. Passing it in any query that returns results would also result in returning a failure. ~ Nathan

2 1

Twisted April 2008