I just have started to study twistedmatrix in my spare time couple of weeks
ago. I am satisfied with the books, but as they are very terse, and they are
not really FTP related, I could not figure out how the FTP is working.
I would like to be able to filter/modify the LIST(ing) process. Eg after
list(ing) the result should not list any .py files even if they are there.
I have tried to look after IFTPShell, but no success sofar. Here is a code
snipplet from the "official" testcases - or even from twisted.ftp)
def requestAvatar(self, avatarId, mind, *interfaces):
for iface in interfaces:
if iface is ftp.IFTPShell:
avatar = ftp.FTPShell(filepath.
return ftp.IFTPShell, avatar, getattr(avatar, 'logout',
Could anybody give me any advice, where to start, please?
Any help would be appreciated.
My problem is that PyAmf uses the standard Python logging module and twisted
uses it's own logging module. I would like to have a single logging
mechanism that can be specified via twistd. I would also like to have more
control of the output of the twisted log. After googling and reading
Glyph's essay on the logging module (
http://twistedmatrix.com/trac/wiki/TwistedLogging) I have came up with two
1) Simply configure the logging module with a logging.Handler subclass that
outputs to twisted.python.log.msg. I can then add a custom LoggingObserver
to control the output.
2) I can use log.PythonLoggingObserver to redirect to the standard logging
module and use the standard logging formatting options and logging levels to
control output (taking into account the caveats pointed out by Glyph).
My questions are as follows:
1) Are the above reasonable or am I missing something simpler?
2) When adding a LoggingObserver I am left with the LoggingObserver
initialized by twistd. I can supress this with -l /dev/nul but it seems
better to eliminate it completely. I guess I could just clear the
,observers property of the theLogPublisher singleton put that seems like an
implementation detail. Again, am I missing something? Is there a way to
tell twistd to let me set up my logging manually?
Note, I am using version 8.1.
As a Twisted newbie, I read the Twisted documentation, and found that:
1. ITransport.write() doesn't return a deferred.
2. IProtocol doesn't have a dataSent() callback or something alike.
Since ITransport.write() is a non-blocking call, how can I be informed
when data sent by ITransport.write() is actually sent successfully?
I've managed to come up with a minimal sftp server implementation
which able to auth users using rsa_key
the code is available at: http://gist.github.com/37446
The point is that I am missing the knowledge needed in order to add
That is, controlling which path(s) a user can access.
it appears to be undocumented
Would love to get some references, samples, etc.
Thanks in advance,
I am trying to migrate to web2. And find the "OldRequestAdapter".
What I did is callling:
## in a resource of web2
request = twisted.web2.compat.makeOldRequestAdapter(web2_request)
File "c:\python25\Lib\site-packages\twisted\web2\compat.py", line 336, in
cookiename = string.join(['TWISTED_SESSION'] + self.sitepath, "_")
AttributeError: 'OldRequestAdapter' object has no attribute 'sitepath'
Thanks for the detailed reply.
>>>>> "glyph" == glyph <glyph(a)divmod.com> writes:
glyph> On 27 Nov, 05:16 pm, terry(a)jon.es wrote:
glyph> For me, baroque and elaborate start-up dances are a code smell.
glyph> Services should be as independent as possible. Of course, sometimes
glyph> some kind of initialization conversation is unavoidable, but I do
glyph> like to try to keep it as short as possible.
I do too. Sometimes it takes (me at least) a few iterations before you see
how best to do that.
glyph> I think you're misunderstanding what a "service" is. The word is,
glyph> perhaps, a bit to lofty for its humble job. A service is just an
glyph> event notification mechanism that tells you when it's time to start
glyph> up, and when it's time to shut down.
glyph> I can understand why it would be attractive to misunderstand in this
glyph> way, though: IService doesn't do very much, you have requirements
glyph> that it doesn't cover, and if it were the thing you understand it to
glyph> be then it would cover those requirements. I'm sure that would be
glyph> nicer for you :).
glyph> This might seem a bit inconsistent, since stopService uses the
glyph> return of a Deferred. However, this is for a very specific reason,
glyph> not a generalized error-handling case: you may need to prevent the
glyph> *rest* of the system (specifically, the reactor) from completely
glyph> shutting down until you've managed to cleanly shut down whatever
glyph> you're trying to shut down on potentially remote systems.
glyph> startService has no such problem though; the service subsystem has
glyph> told you "It's time to start up!" - its job is done, and the reactor
glyph> isn't going away as part of service startup, so it's your
glyph> responsibility as an application author to make sure your other
glyph> dependencies are properly initialized.
OK, this is helpful - I have been looking at it from a different point of
view, as you've guessed.
>> But if something does go wrong, you've got a failure propagating its way
>> down a errback chain, eventually (unless an errback switches you back to
>> the callback chain) popping out the end and causing the reactor to issue
>> an Unhandled Error message. So you can't indicate that the service has
>> failed to start by throwing, because the exception is going to pop
>> harmlessly out the end of the deferred chain as a generic unhandled
>> error and will not cause Twisted to know that the service couldn't
glyph> The key question here is: indicate to whom? If you want to indicate
glyph> it to some other object, well, try:except: or addErrback and call a
glyph> method on that object. Nothing magic about it.
I have code written as a Twisted plugin. So I have a class implementing
IServiceMaker and IPlugin, and I create an instance of that class which
gets found when I invoke twistd from the command line.
So in my case I want to indicate to twistd that the service that my class
creates a makeService method to create, but which I do not set in motion,
has failed to start and that twistd should exit, or do something other than
cheerfully tell me that there's been an Unhandled Error.
Does that make more sense? Sorry, I should have said I was using twistd.
glyph> In what way would you expect the service mechanism to "deal with"
glyph> returning a Deferred? Stop starting other services? Print out some
glyph> different log message?
I'm not sure what should happen. I'm sitting at the command line, I've
asked twistd to start something for me, there's clearly been a problem
doing so (and this doesn't have to be baroque, maybe I just couldn't listen
on a specific port I wanted, or maybe my code somehow raised an Exception),
but I don't seem to have a mechanism for having twistd take any notice at
I'm just talking about the case where startService calls something that
returns a deferred and there's an Exception that comes back down the
Deferred chain as a failure. I suppose if startService raises an Exception
itself directly, something else happens - maybe twistd exits.
glyph> IService is a very, very simple interface. If you want to respond
glyph> to failures from startService (deferred failures, exceptions, or
glyph> whatever else) in a useful way, then you can write your own
glyph> implementation of it which manages startup order, keeps track of
glyph> dependencies, and maintains a state machine that handles stopService
glyph> appropriately if called in mid- startup.
glyph> I don't think that having to implement an interface with 6 methods
glyph> on it could be considered "cruel and unusual". If you think so you
glyph> may want to investigate options other than Twisted: you will
glyph> frequently be expected to implement interfaces with methods on them
glyph> There's no need to "track down and subclass" lots of things. Your
glyph> IService wants the things that it contains to have a richer
glyph> interface which allows for error handling, dependencies, and
glyph> propagation, so simply write a single wrapper for simpler IService
glyph> objects that expands the interface to do the other things that
glyph> you're interested in.
In the case of a service being started by twistd, it doesn't seem as simple
as you describe, but maybe that's my lack of understanding again. I can
easily subclass IService, but something else is calling the startService
method of that subclass. And that thing, whatever it is, is not expecting
me to return a deferred. So if my startService has for some reason got its
hands on a deferred, it can't simply hand it back to its caller and have
something (twistd in my case) see that an error occurred.
It does feel like I have to track down what this something else might
be. Either working from my IServiceMaker implementation or working from
/usr/bin/twistd to find where startService is not trivial (you guys wrote
it, I'm sure the logic is all much clearer to you). After looking through a
few files I wind up at twisted/application/app.py, which has a
startApplication function that calls
service.IService(application).startService(). So I guess that's what is
calling my startService. So I could make my own startApplication function,
but I then have the same problem, I wind up with a deferred on my hands and
my caller is not expecting me to return it. Plus, the startApplication
function sits at the top level of twisted/application/app.py, so I have to
find whatever is calling that. That seems to be
twisted/scripts/_twistd_unix.py, which imports twisted.application.app and
has a top-level startApplication that calls app.startApplication. But who
is calling that? Looks like twisted/scripts/twistd.py is, and that's called
So should I write my own twistd? All this doesn't seem to be a matter of
simple subclassing. Plus, I can't just go in and start editing the
top-level functions in twisted/application/app.py and
twisted/scripts/_twistd_unix.py or code that imports app, etc.
Sorry for so many questions - I really don't know if I'm missing something
simple here. I do enjoy digging into all this, and I appreciate your
apparently limitless patience. I wish I knew it all better. Twisted is
complex and it's a pretty good bet that anything you think of or run into
as a n00b has been thought of or encountered before, and that whatever way
you think of to solve it will probably be non-optimal, or plain wrong, or
in ignorance of a solution someone much more experienced has already
implemented, or... etc. Hence my many questions.
glyph> This all strikes me as totally straightforward and easy, and I don't
glyph> think I'm any kind of super-genius for being able to write a few
glyph> Python classes that call a few simple start/stop methods in the
glyph> order that I want them to run in :).
I should have mentioned that I want to use twistd.
In fact I have something like a process pool running in one service and I
talk to it from another machine. I say "hey, process pool, start me up the
following service (a twistd service)" and I would then like to know if that
service started, and if not then why not. So having twistd fail or report
an error if it can't start a service would be useful.
glyph> Doing either of those things would definitely be wrong. There's no
glyph> reason to sys.exit or reactor.stop if your application can't start
glyph> up, unless your management system specifically calls for such a
Maybe this is a case where it's (semi-)justified. At least if I called
sys.exit the twistd process would go away, instead of sitting there acting
as though nothing's wrong :-) I can also, of course, try interacting with
the service I think I just started on the remote machine, and if I can't
then I can tell the original process pool to kill the twistd process. But
that seems a pretty roundabout alternative to just having twistd notice
that something went awry when calling startService.
glyph> In the future, even the Twisted plugin code might be starting some
glyph> things in addition to your application. As I mentioned above, a
glyph> good reason to do that is to perform diagnostics on failed startups
I assume you really mean "startups" and not "services". In which case, I'm
100% sure there's something funny here, but I can't figure it out. I'd love
to know, and I'm smiling broadly in any case. Too bad email loses so much
humor..... we can always try though.
Date: Sat, 20 Dec 2008 08:28:20 -0700
From: Joe Strout <joe(a)strout.net>
Subject: Re: [Twisted-Python] how to get an idle callback while
running a reactor?
To: Twisted general discussion <twisted-python(a)twistedmatrix.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>Well, the thing is that there will actually be a great many things that
>could cause it to need to send a message, such as the time, a change in
>the outside temperature, a certain change in the value of a stock fund,
>etc. And the AIM interface isn't the only interface to this bot.
I would argue all these things can be viewed external events - even time (imagine a clock source is calling your system every N time units). In turn, your programme reacts to the external events.
At a more concrete level, you have to ask yourself questions like "is my programme a client, a server (or both)? That is, is my programme making connections? Or is it waiting for connections? This will affect how you structure and start-up your application. From I see SkippyTalkBot is a client.
>Now, that does suggest an alternative approach, which is to ignore
>twisted's event/idle mechanisms and use something external to it. But
>I'm also fairly newish to Python, so I'm not sure exactly how to do that
>either -- reactor.run() is a blocking call, so I supposed I'd have to
Yes under the hood reactor.run() blocks. However the reactor is what effectively starts a Twisted application. I would argue that under most circumstances, you probably don't care that the reactor blocks because your application isn't doing anything until it receives an event from the reactor. As for threads. Off hand, I suspect you don't need threads .....
>But if there's a simple way to do this within Twisted, I'd like to learn
>what that is
Let us pretend your programme polls various network sources at regular intervals. So perhaps your code could look like:
if __name__ == "__main__":
# it may be helpful to keep track of the task objects
source['source-a] = task.LoopingCall(callback-for-stockTicker, ...)
source['source-b] = task.loopingCall(callback-for-thermostat, ...)
source['source-c] = task.loopingCall(callback-for-timer, ....)
for source in sources:
source.start(someInterval, True | False)
and each callback-for-X sets up the appropriate call to the remove source and its callback chain.
# assume we are using SOAP
ticker = soap.Proxy(....)