I wrote web application on twisted and my application read from ulogd daemon
and push the output to the client browser using AJAX .also I have another
application displaying ethernets traffic using AJAX.
but after some minutes the twisted process in the server eat the memory and
the OOM-killer of the linux will kill the twisted process.
are there any solution to make the twisted release the memory .
note: my ram is 256mb
Muaaz Osman Hussain
> I just started to learn twisted. I tried to look after a working FTP example
> on the net for twisted, but unfortunately I could not really found any. This
> one seemed to be the most completed, but it says "Unknown error code" when I
> try to connect to it.
I'm sorry, but I don't have the chance of looking it further right
now, but I took a look at your code, and I say that instead of making
FTPServerFactory inherit from FtpFactory, you opted to instantiate it
inside a object inherited class. I never tried this approach, and that
may be the root of your problem. I'm redirecting you e-mail to twisted
mailing list, so people there can help you better. I advice you to
subscribe to that mailing list to get help (if you aren't subscribed
yet). You may also try the IRC channel for some help. Twisted folks
are very responsive and willing to help.
> In my opinion the code is fine. May I ask you to take a look on it please,
> and give me any suggestions, if you can see, why it is not working?
> I would really appreciate your help.
Adriano Monteiro Marques
"Don't stay in bed, unless you can make money in bed." - George Burns
I want to build a MVC with the Model based on twisted with a custom
protocol. At the moment all my attemps terrible failed. So far the client
build a valid ssl-connection to the server and send the lines he is supposed
to send but don't recognise any incomming lines from the server. So I
decided to build the model from scratch again. The controller inits the
model-class (implementation of the custom chat-protocol) to be able to call
methods of the instance and send messages to the server. So how do I connect
the model to he server? In former implementations I were using the
Clientfactory but this didn't work very well.
Thanks for your help
I have an application that is doing some web spidering. Right now I'm
using urllib to retrieve the URLs, but it is painfully slow. I was
wondering if it's feasible to swap out urllib with a twisted client
that uses deferds so I can process urls in a more "parallel" fashion?
I've done a bunch of Googleing, but I haven't come across anything
that I can use as a drop in replacement. If you can point me in the
right direction I'd really appreciate it!
Thanks for your help!
The problem is that: the UDP listening ports increase rapidly and eventually
come to the end: too many open file in selector.
As many as the hosts you lookup, those UDP ports are still alive. It is very
different from running this script in Linux (gentoo 2.6).
In Linux, the listening UDP port keep only 1 instance no matter how many
Try the script below:
from twisted.mail import relaymanager
from twisted.internet import reactor,defer
MXC = relaymanager.MXCalculator()
print 'Got mxRecord: %s' % mxRecord.name
d = getMailExchange('gmail.com')
d = getMailExchange('yahoo.com')
#d.addCallback(next) ## uncomment the line to increase your UDP
listening ports as many as you want
I came across this livejournal entry today:
Which has this link:
.. which is a paper by Rob von Behren, Jeremy Condit and Eric Brewer. I
personally raised an eyebrow when I saw that because Eric Brewer was the CTO
of Inktomi, who back the had a very highly scaleable network system called
Traffic Server amongst other things (I worked at Inktomi between 2000 & 2002,
which is why I recognised the name).
Traffic Server's architecture was in many respects reactor based, and in many
respects it's architecture and twisted's architecture weren't *massively*
different. OK, TS was written in C++, and twisted is python, but a reactor
approach is still a reactor approach.
Specifically I found this part particularly surprising:
Event-based programming has been highly touted in recent years as the best
way to write highly concurrent applications. Having worked on several of
these systems, we now believe this approach to be a mistake. Specifically,
we believe that threads can achieve all of the strengths of events,
including support for high concurrency, low overhead, and a simple
The paper appears to be 5 years old already, so the impact of this paper
appears minimal, but I thought it may be of interest to others. (The reason it
raised such a surprise for me is because it would be as surprising to me as
it would be to read of any of the core twisted devs to suddenly come out in
favour of a threading approach rather than non-threading!)
Looking slightly below the surface of things, it appears to be advocating
something more like stackless's tasklets, rather than OS level threads (since
it makes reference near the end to Erlang's threading model).
Either why, hopefully interesting :)
Hi Michael and Colleagues:
> .. which is a paper by Rob von Behren, Jeremy Condit and
> Eric Brewer.
As a part of writing "Adventures in Stackless Python/Twisted Integration," I have read "The Why Events are" paper as well as the Ousterhout "Why Threads Are A Bad Idea (for most purposes).
The "Why Events" paper led me to the Adya et al, "Cooperative task management without manual stack management" and the original Lauer and Needham "On the duality of operating system structures." I also visited the Schmitt papers on the Reactor pattern.
>From what I recalled, I preferred the latter three papers because they provide a much better analytic framework for describing the issues. In particular, the Adya provides terminology.
I have two basic sets of take-aways from the paper survey
1) "Events are great papers" state that threads are bad because of A,B, and C. "Threads are great papers" state that events are bad because of X,Y, and Z. As a programmer using these systems, perhaps I should pay special attention to issues A,B,C,X,Y and Z whenever applicable (and by the way papers thanks for telling me C and Y were issues since I didn't know).
Perhaps out of all this, we can come up with appropriate guidelines for lockless programming?
2) The "Duality" paper argues that messaging passing and procedural systems can be described in terms of each other (bi-simulation?). I found this to be a very powerful way to look at Stackless Python and Twisted's threading models.
A simple example.
At the time I presented "Adventures," to avoid deadlock, I ran a reactor protocol instance in its own Stackless thread. This bugged me. It turned out that by replacing channels with deferreds (and doing a little buffering), I could replace the extraneous tasklet. My way of looking at it was, after all, protocol instances/deferreds are just another way of doing continuations... Use them, don't replace them....
> Looking slightly below the surface of things, it appears to
> be advocating something more like stackless's tasklets, rather than
> OS level threads (since it makes reference near the end to Erlang's >threading model).
A quote from the paper:
"To evaluate the ability of threads to support high
concurrency, we designed and implemented a simple
(5000 line) user-level cooperative threading package for
Linux. Our thread package uses the coro coroutine
library  for minimalist context switching, and it
translates blocking I/O requests to asynchronous requests
> The paper appears to be 5 years old already, so the impact
> of this paper appears minimal, but I thought it may be of interest to
> others. (The reason it raised such a surprise for me is because it >would be as surprising to me as it would be to read of any of the core >twisted devs to suddenly come out in favour of a threading approach >rather than non-threading!)
"The Why Events" is essentially the approach I took in regards to Stackless and Twisted. Right now, I am analyzing Stackless's scheduling model to see if improvements can be made. I feel I have much to learn...
That said, I feel inline callbacks are a step in this direction. As Stackless/PyPy concepts get into mainstream Python, I can see Twisted following in this direction. Already one can see that with stuff like Christopher Armstrong's Corotwine.
Again, I think the key to incorporate threads and keep to the spirit of Twisted, simplicity and safety, is to come up with some guidelines for lockless programming.
I have to run commands and (later in time) collect logs from many machines,
connecting through SSH.
I was _genuinely_ wondering what an SSH client written in Twisted could give
me more than executing plain old shell scripts in a for loop.
>From what I saw, given the fact that my run/collect commands remain still
shell commands (cp, tar, etc...), the only thing that may change is that
using Twisted I could have a better utilization of resources rather than a
simple shell for loop in which I iterate in order over all the machines on
which I have to issue commands.
Am I missing something? I would like to stress the fact that I'm asking this
as a very specific question for Twisted usage in the context of
running/retrieving results on remote machines (as indeed I already happily
use Twisted for running custom network protocols) and I would right to
choose a simple but effective solution for this case.
Thanks in advance,
I'm making a simple app that will run a local proxy and redirect me away
from distracting websites. For example, if I go to facebook.com I'd like it
to instead send me to google.com
I'd like to use python and in my research I think a twisted reverse proxy is
what I want. I've done the examples for setting up a proxy, including one
that counts the words on the pages you visit, but i cannot find anything on
fetching the destination and re-directing. I've been sifting through the
many classes of the proxy examples and am pretty lost. Does anyone have
some example code for something like this?