Hi out there
Let me tell you about my experience with TM so far ...
Seeing there is doc. available, I digged through it. After all TM seems
worth the effort. From what I see, I love TM.
Links are often broken, though. Often seemingly useful doc. is old, very
old. Even current doc. isn't up to date.
Anyway, I try and I mean I try hard to get on my feet following some
examples and snippets. The problem isn't the
deferreds. I understand them pretty quickly and like them. The problem is
that the docs are like a maze, irritating, frustrating and misleading rather
Trying to write a rpy. Of course, it doesn't
work. Looking at all those different docs, I'm bewildered. Somehow
hacking my way through it.
They talk about "resource trees", yet I don't find them in the doc. I find
putchild() but all examples and docs indicate, that you can do it only
within the root resource.
Again browsing, again looking desperately for some docs, that don't mislead
and make things even more fuzzy. Falling over an article with some halfway
useful tap doc.
OK, a little later, I
have my first web server running an my first rpy works, spitting out
I'm wondering, why of all things "rpy" ? python won't generate sth. like
"rpyc" for that, which translates in lost speed, I assume. No explanation.
I'm on FreeBSD. Of course, I want to use kqueue. Now, another "funny"
journey begins ...
the py-kqueue in FreeBSD 6.3 and 7.0 is version 1.4. I remember, however,
having read something by Isthmar Trauring Shtull (forgive me, if I didn't
spell that correctly) that I need version 1.3 in order to apply his patch,
which, of course (sorry, that's my disappointment carrying me away ...),
doesn't work. No prob, I do it by
hand. To no avail. That code is old, very old and it doesn't work.
I remember havin seen some version 2 of pykqueue. Setup works nicely but
Python/twisted coughs. The module has another name now. "(%"%§$( !!!
So I fall back to the version 1.4 that comes with FreeBSD ports. It works.
But in no-deamon mode only. As soon as I tell twistd to run web.tap
deamonized it vomitts and breaks.
Intense googleing, and I mean "intense". Reading through years of this
mailing list. No positive result.
What I say is said, because I love python an I consider twisted brilliant
and immensely useful - theoretically. So, don't get me wrong and accept it
as constructive, albeit somewhat pissed off remarks, OK :)
I have the clear impression that twisted is something in between a toy and a
brilliant product. It's hackers, however, leave much to be desired in terms
of usefulness. As it is, it's a great and promising hobby but not a useful
Sorry, but doc strings don't replcae a halfway decent documentation and a
reasonable tutorial, considering the highly complex matter.
Sorry, but documentation that is often enough outdated and sometimes offers
broken links is next to useless.
Sorry, but the most brilliant code is a lot less attractive and useful, if
even "stable versions" are broken, in flux, etc ...
Kqueue seems to be vital to an event driven approach like TM, yet there are
multiple versions of pykqueue floating
around, none of them properly working (and I didn't fumble around. I
plain simply used the FreeBSD port), some of
them with some mor and some with some less "annotations" (I refuse to
call that doc.)
- How about getting 1 version of pykqueue properly running and into TM ?
- How about freezing some TM version (like 8.0) and updating/matching docs?
(Of course, experiments are funny and intriguing, but quite some of us out
here need sth. stable for everydays work)
- How about writing some complete docs and tutorials? Sth. along the lines
of "My first web server wit TM" (instead all those - sometimes seeming to
contradict each other - snippets)
I'm still fascinated by TM and still willingto invest time, efforts and
brains. But I'd love to have some properly working base and docs that
expßlain rather than confuse.
Looking forward to learn and enjoy.
(Yes, Should someone need sth. tested on FreeBSD, I'll gladly be of service)
First: THANKS for reacting so friendly and civilized :)
> Have you tried the finger tutorial.
Sure. When I say, I DID read what's available, then I did. Otherwise I
wouldn't dare to complain.
Thomas Hervé <therve(a)free.fr> wrote:
> > - How about getting 1 version of pykqueue properly running and into TM ?
> The current one is grossly working, even though there are some
> problems, and that the setup is not triavial. But a new reactor has
> been developed, using Python 2.6 KQueue wrapper, and should be
> available some day. See http://twistedmatrix.com/trac/ticket/1918 for
> the progress on this field. Any tests and feedback would help this
> going further.
Great. Thanks. PLEASE test it properly and prepare a reasonably conmplete
readme at least.
And kindly see below for my current problems.
> Feel better now? ;-)
> I understand your pain. Twisted is very much written by hackers for
> hackers, with the old school practices of source and oral tradition in
> place of documentation. It took me many attempts before I felt I was
> competent, and I still can't do anything without constantly consulting
> the source code. I confess, I haven't bothered to look at the docs
> for years, and I suspect it is the same with the core Twisted
> Life is made easier by having a mentor who can look over your work and
> point out misunderstandings and antipatterns. I find the mailing list
> to be a mixed bag as a replacement. Sometimes you get real gems from
> Glyph and company, and sometimes you just get a hard time. I'd be
> happy to help if I can, but any deep thinking will probably have to
> wait for weekends. I'm also a long time FreeBSD user and Python
> Hope you make it over the curve!
Thanks, Mike, for your friendly welcome.
Yes, I do understand that attitude. I love to have it, too.
BUT: twisted is having and looking for sponsors, that is, twisted is -
gladly ! - meant to be a product and not some hackers private delight
You see, I argue for using twisted. Possibly some of us argue against other
solutions and, worst case, they risk their ass doing so.
Again, I share your attitude and i understand it so well. But, PLEASE,
understand that I need to get sth. working, that some colleagues put a lot
of trust, valuable trust, into twisted, even taking a risk. That should be
seen and properly respnonded to, too.
I'm not talking about script kiddies. I'm talking about seasoned colleages
with a (couple) dozend years in the job.
We don't need pampering, not even mentoring, practical and welcome as that
might be. We need realiabilty and a _fair_ chance to find our ways in
twisted. A reasonable starting point.
Here's what I tried today:
Using py-kqueue (1.4) in FreeBSDs ports.
'twistd -r kqueue -no -f web.tap' works fine.
'twistd -r kqueue -f web tap' breaks.
Next I checked, wether the changes proposed by IST where in version 1.4.
So I edited them into the patch that comes with the port anyway
If anyone is interested, I'll post it.
Same as without, i.e. it works but it breaks if run as daemon.
Here's what I find in the log:
twistd 2.5.0 (/usr/local/bin/python2.5 2.5.2) starting up
reactor class: <class 'twisted.internet.kqreactor.KQueueReactor'>
Changing process name to test
twistd 2.5.0 (/home/www/tm/ 2.5.2) starting up
reactor class: <class 'twisted.internet.kqreactor.KQueueReactor'>
twisted.web.server.Site starting on 80
Starting factory <twisted.web.server.Site instance at 0x85935ec>
Traceback (most recent call last):
File "/usr/local/bin/twistd", line 21, in <module>
line 27, in run
line 379, in run
line 23, in runApp
line 158, in run
line 213, in postApplication
line 174, in startApplication
line 228, in privilegedStartService
line 228, in privilegedStartService
line 68, in privilegedStartService
self._port = self._getPort()
line 86, in _getPort
return getattr(reactor, 'listen'+self.method)(*self.args, **self.kwargs)
line 467, in listenTCP
File "/usr/local/lib/python2.5/site-packages/twisted/internet/tcp.py", line
750, in startListening
line 265, in startReading
line 98, in addReader
self._updateRegistration(fd, EVFILT_READ, EV_ADD)
line 89, in _updateRegistration
kq.kevent([kevent(*args)], 0, 0)
OSError: [Errno 9] Bad file descriptor
I just wanted to put your minds at rest regarding what I though might
have been a problem with the epoll reactor. As always, inferring
anything from a production machine is not desirable and it seems that
my assumptions about epoll were based on coincidence.
I have since found the leak:
Unfortunately, my workaround involved rewriting my proxy code without
using twistd.web. I'll happily contribute a patch back to the
t.w.proxy code in time but for the moment it doesn't support
persistent connections (I wanted to keep it simple). I also feel that
t.w.proxy would be better served by making use of a reciprocal
IProducer / IConsumer arrangement between the ProxyServer protocol
and the ProxyClient protocol which is something that my workaround
does not implement.
Thanks again for Twisted,
phone: 44 (0) 7715 754017
GPG: 96FF DE0E 0B7B 37F0 7F8D C54C E285 3D8F 5625 9244
> >I would like to provide an environment where developers can write call
> >servicing applications with no knowledge of the network. I had rested
> >on a coroutine approach, whereby a developer could write something
> >like this:
> > response, events = (yield getKeyPress(SomePromptAnnouncement))
> Just to clarify - this looks like a generator rather than a coroutine.
> >With the response, the developer can do an undefined number of things,
> >taking an undefined amount of time. During this time, management
> >events can arrive on a separate port, to which the developer gets
> >knowledge of through the yield statement.
> >On the other side of this generator is a scheduler which takes
> >instructions such as "getKeyPress" and passes it through twisted to
> >the remote client, such that it can play the announcement and get a
> >keypress. The scheduler gets the response and sends to back to the
> >generator, along with any events that have arrived on this separate
> >When a call/connection arrives, the client will send an opening
> >line(s) to which I was specify the correct LineRecieved method. This
> >will trigger some scheduler code defined elsewhere via a deferred,
> >which will prompt the developer's code for some instructions, such as
> >"PlayPrompt". Am I correct in thinking that while a developer's code
> >is executing, all other connections are paused, and that the twisted
> >server will not accept new connections until it returns?
> Yes, that's correct. Twisted is single-threaded - it invokes event
> handlers in the same thread it uses to monitor event sources. As long
> as an event handler is running, event sources are not monitored for
> events and no other handlers will be invoked.
> >My original assumption was that Twisted would spawn a new thread
> >within which the scheduler would be set to run to manage the
> >communication for the duration of the customer call/interaction.
> >However, the approach taken by twisted is that if the developer's code
> >got itself in a muddle or infinite loop or took a very long time
> >accessing a database, the rest of the application would just be
> Generally speaking, Twisted will only create a thread when application
> code asks it to.
> >I wrote a quick test app that defers to a function that sleeps then
> >returns its response line. During the 20 seconds that the test server
> >took to return a line, the server would not accept any other requests
> >until the sleep function ended.
> The "defers to a function" language is a bit confusing. However, your
> conclusion sounds right. If you block the reactor thread, nothing else
> will happen.
> >Clearly, I must be missing something; I have designed my application
> >upon the wrong axis. If so, I have some misunderstanding as to how to
> >properly structure an application like this upon Twisted. Or, twisted
> >is a framework whereby the response to a network event is expect to
> >arrive immediately.
> It's not necessary for responses to arrive immediately. Responses just
> need to be obtained without blocking. For example, if getting a response
> involves talking to someone else over the network, then Twisted has some
> networking APIs that don't block. ;) Likewise, twisted.enterprise lets
> you talk to a rdbm (it actually uses threads, but it mostly doesn't show
> that to you).
> >I have seen "deferToThread" mentioned but I cannot find enough
> >documentation to decide whether it's what I need or not. Or perhaps
> >callInThread() is what I need?
> One of these may be appropriate. You could decide that in your application
> framework, user code is always run in a non-reactor thread. This is easily
> accomplished: your event handler just needs to invoke user code in a thread
> instead of directly. deferToThread lets you run a function in a thread and
> gives you a Deferred which will be called back with the return value of the
> function when it returns or which will errback if it raises an exception.
> callInThread is lower level, not exposing the result of the function.
Ahh. Okay; thanks for this. deferToThread looks like a good option
here and seems to fit what I'm looking for. From experience, can you
point me towards some good code that uses deferToThread?
> >Any suggestions about this would be very much appreciated. Even
> >better, if anybody knows of good documentation/tutorials they can
> >point me to, that would be appreciated also.
> It sounds like you're looking for deferToThread or one of its friends.
> Note however that just throwing user code into a thread doesn't mean
> programmers can be oblivious to their environment. It simply trades the
> need to pay attention to when you block and for how long for the more
> complex task of paying attention to thread safety. If each task is
> isolated from all others, this may be a good trade-off to make, but if
> there's shared state it may not be.
Thanks for your reply; there are some subtle and difficult questions
to answer here. I'd like each instance of the developer's code to be
isolated with it's own state using threading.local, but ultimately
I'll use deferToThread to set it all up.
I assume that I can use some threadsafe implementation (if the GIL
doesn't enforce that anyway) of a dictionary to store the global
information between each call handling thread.
Thanks again for your time.
>I would like to provide an environment where
>developers can write call servicing applications with
>no knowledge of the network. I had rested on a
>coroutine approach, whereby a developer could write
>something like this:
I posted some code a little while ago that has WS-BPEL
logic riding on top of Stackless and Twisted.
>Am I correct in thinking that while a developer's
>is executing, all other connections are paused, and
>that the twisted server will not accept new
>connections until it returns?
[From the Schmitt Reactor Pattern Paper]
>Non-preemptive. In a single-threaded application
>process, event handlers are not preempted while they
>are executing. This implies that an event handler
>should not perform blocking I/O on an individual
>handle since this will block the entire process and
>impede the responsiveness for clients connected to
If you accept the Schmitt explanation, I would suggest
that it is convenient to think of Twisted as a
non-preemptive threading system where callbacks and
server protocol instances are user space "threads of
execution" and the reactor is the scheduler. Since a
"thread" cannot be involuntarily be pre-empted, the
reactor will not have a chance to trigger callbacks.
Moral of the story - you don't want to do long CPU
intensive work in a callback.
>My original assumption was that Twisted would spawn a
>new thread within which the scheduler would be set to
>run to manage the communication for the duration of
>the customer call/interaction.
I think you want reactor.callInThread(). I remember
Philip Mayers telling me about this in a post about a
Here is an example - forget the Stackless stuff
(Actually I would rather like to forget that piece of
However you may want to think over your design if you
find you need to extensively use OS threads.
I would be tempted to ask myself questions like:
1)Is my application CPU or I/O bound"?
2)Can I encapsulate the message exchange pattern
between the client and the server be done in a single
3)Can I conveniently yield the thread at predictable
>Or, twisted is a framework whereby the response to a
>network event is expect to arrive immediately.
I think the exact opposite is true, that is why the
deferred object exists.
I would suggest Twisted is ideal for reactive, I/O
bound applications particularly when you can represent
the message exchange pattern in a single protocol.
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
I am researching technologies for building an application development
framework with Python, and have been interested in Twisted to manage
the network communication side of things. However, I read something
yesterday that appears to undermine the applicability of twisted for
my work. I've done quite a bit of reading around the subject since,
and have ended up confusing myself!
A remote client accesses my server to get instructions on how to
progress a customer "interaction." The remote client is itself a
server driven by a telephone call and keytone input. The remote client
could be servicing N calls (channels) simultaneously, and thus my
server needs to support N simultaneous TCP connections. Pretty
I would like to provide an environment where developers can write call
servicing applications with no knowledge of the network. I had rested
on a coroutine approach, whereby a developer could write something
response, events = (yield getKeyPress(SomePromptAnnouncement))
With the response, the developer can do an undefined number of things,
taking an undefined amount of time. During this time, management
events can arrive on a separate port, to which the developer gets
knowledge of through the yield statement.
On the other side of this generator is a scheduler which takes
instructions such as "getKeyPress" and passes it through twisted to
the remote client, such that it can play the announcement and get a
keypress. The scheduler gets the response and sends to back to the
generator, along with any events that have arrived on this separate
When a call/connection arrives, the client will send an opening
line(s) to which I was specify the correct LineRecieved method. This
will trigger some scheduler code defined elsewhere via a deferred,
which will prompt the developer's code for some instructions, such as
"PlayPrompt". Am I correct in thinking that while a developer's code
is executing, all other connections are paused, and that the twisted
server will not accept new connections until it returns?
My original assumption was that Twisted would spawn a new thread
within which the scheduler would be set to run to manage the
communication for the duration of the customer call/interaction.
However, the approach taken by twisted is that if the developer's code
got itself in a muddle or infinite loop or took a very long time
accessing a database, the rest of the application would just be
I wrote a quick test app that defers to a function that sleeps then
returns its response line. During the 20 seconds that the test server
took to return a line, the server would not accept any other requests
until the sleep function ended.
Clearly, I must be missing something; I have designed my application
upon the wrong axis. If so, I have some misunderstanding as to how to
properly structure an application like this upon Twisted. Or, twisted
is a framework whereby the response to a network event is expect to
I have seen "deferToThread" mentioned but I cannot find enough
documentation to decide whether it's what I need or not. Or perhaps
callInThread() is what I need?
Any suggestions about this would be very much appreciated. Even
better, if anybody knows of good documentation/tutorials they can
point me to, that would be appreciated also.
Does anyone know of any (possibly experimental)
twisted.enterprise.adbapi support for .executemany() ?
(see http://www.python.org/dev/peps/pep-0249/ -- search for ".executemany")
If not, I'll see if I can't add a runOperationMany() to
twisted.enterprise.adbapi.ConnectionPool. The functionality I want is
simple in theory. .executemany runs many identical
insert/update/delete operations using a different dictionary in a
supplied list of dictionaries for the input each time, and returns
None on success of all of the queries, or rolls-back the whole
transaction and returns a failure on an error. Passing it in any
query that returns results would also result in returning a failure.