some questions about twisted.web
I just wrote a simple tool to make my AJAX app development cycle better. It's a tiny web server that serves up the current directory along side some reverse proxies. You can find the current code here: http://github.com/metajack/tape/tree/master In writing this I ran into a number of issues with twisted.web. I'm pretty sure some of these are bugs, but I wanted to get some feedback here before I filed them. * HTTPFactory seems only able to log to files. It takes a path name, not a file like object. This makes it very difficult to log to stdout, as I have to override log() in my own class. This in turns means cherry picking a global out of twisted.web - fun! * log() never gets called when a ReverseProxyResource is used. It appears as though request.finish() is never called, though I wasn't able to fully track this down. I find this behavior very odd. * static.File, when given a directory, creates instances of itself to handle children. This code completely fails for me with when I subclassed static.File. Note that my class only has a few construction parameters, while static.File has 5. This is not documented. It would be nice if the code could detect this case and tell me that I have to overload createSimilarFile. * The only way to stick something in the tree at an arbitrary location seems to be to walk the tree to that spot, creating dummy locations as you go. I find it extremely weird coming from several other web frameworks. There is nothing else but the concept of "hey you! get me child X". This makes it pretty hard to implement anything better than walking the tree since state would have to be collected over the traversal. * Related to the above, the handling of foo vs. foo/ is pretty confusing. foo/ is considered the '' child of foo. This is pretty yuck to me. Unfortunately, I don't have any creative suggestions about how to do it better right now. * ReverseProxyResource returns nothing if you don't give it a '/' at the end. I had to work around this by doing a redirect in the subclass and then returning a new ReverseProxyResource when the '' child is accessed. It seems that if ReverseProxyResource's path is '', it just keels over with no error. These combined to make what should have been a few lines of code and an hour into 3-4 hours and almost 150+ lines. Most of the time I spent tracing through the Twisted source trying out figure out where the weird stuff was coming from. And after all that logging still doesn't work right. jack.
Jack Moffitt pisze:
I just wrote a simple tool to make my AJAX app development cycle better. It's a tiny web server that serves up the current directory along side some reverse proxies. You can find the current code here:
http://github.com/metajack/tape/tree/master
In writing this I ran into a number of issues with twisted.web. I'm pretty sure some of these are bugs, but I wanted to get some feedback here before I filed them.
* HTTPFactory seems only able to log to files. It takes a path name, not a file like object. This makes it very difficult to log to stdout, as I have to override log() in my own class. This in turns means cherry picking a global out of twisted.web - fun!
When you start your application: from twisted.python import log log.startLogging(sys.stdout) Or, even better, create an Application object and use twistd(1) to start your service. Using twistd(1), you have a choice to daemonize your process or not.
* static.File, when given a directory, creates instances of itself to handle children. This code completely fails for me with when I subclassed static.File. Note that my class only has a few construction parameters, while static.File has 5. This is not documented.
http://glyph.twistedmatrix.com/2009/04/my-time-at-pycon.html """ These sessions helped to reinforce for me the need to repeat this frequently for our users: if Twisted is doing something which confuses you but seems wrong, please go ahead and file a bug on twistedmatrix.com <http://twistedmatrix.com/>. If you're wrong, and Twisted is working properly, there's still a bug, it's just a bug in the documentation. If the documentation were perfect presumably you would understand why it's doing what it's doing. Even if we ultimately decide that the documentation is sufficient, by adding a bug in a tracker, you've told Google that people with your question should find the document we refer to in our comments, so there's still value. """
* The only way to stick something in the tree at an arbitrary location seems to be to walk the tree to that spot, creating dummy locations as you go.
Override locateChild
I find it extremely weird coming from several other web frameworks.
Twisted initially seems very different, no matter what framework you came from.
* Related to the above, the handling of foo vs. foo/ is pretty confusing. foo/ is considered the '' child of foo. This is pretty yuck to me. Unfortunately, I don't have any creative suggestions about how to do it better right now.
addSlash = True -- m
When you start your application:
from twisted.python import log log.startLogging(sys.stdout)
Now I've just deluged the poor user with lots of factory start and stops. All I want is normal HTTP style logs. Twisted's logging infrastructure has its place, but it is not quite ready to produce user friendly output for a simple utility.
Or, even better, create an Application object and use twistd(1) to start your service. Using twistd(1), you have a choice to daemonize your process or not.
Great, now the user has to edit a tac file just to configure the thing. I'd rather them be able to type "tape" and not have to worry about twistd -ny some.tac. I use tac files a lot, but they don't make much sense for simple utilities. It sucks that it's either/or with Twisted applications.
These sessions helped to reinforce for me the need to repeat this frequently for our users: if Twisted is doing something which confuses you but seems wrong, please go ahead and file a bug on twistedmatrix.com.
This mailing list should be indexed by Google as well. I do plan to file the bugs we find here as well.
* The only way to stick something in the tree at an arbitrary location seems to be to walk the tree to that spot, creating dummy locations as you go.
Override locateChild
What is that? I see no such function: Not in IResource: http://twistedmatrix.com/documents/current/api/twisted.web.resource.IResourc... Not in the server.Site: http://twistedmatrix.com/documents/current/api/twisted.web.server.Site.html grep -r doesn't show it in the twisted/web or any child directories. Not even in SVN trunk.
I find it extremely weird coming from several other web frameworks.
Twisted initially seems very different, no matter what framework you came from.
I grok Twisted just fine. This may be my first post to twisted-web, but I've been hacking on and around Twisted for some time. Even Apache doesn't work this way. I can tell apache that my proxy is at /some/url/over/here and it will work. It will return 404s for the intermediate URLs. Because Resources only have immediate children and paths are only dealt with in single element pieces, there doesn't seem to be a way to do this in Twisted Web. It feels weird that the tree must be complete. It feels weird that paths are dealt with in tiny sections.
* Related to the above, the handling of foo vs. foo/ is pretty confusing. foo/ is considered the '' child of foo. This is pretty yuck to me. Unfortunately, I don't have any creative suggestions about how to do it better right now.
addSlash = True
This does not exist in Resource or in server.Site. What am I missing? It's not mentioned in the docs either. I see that static.File has an addSlash method, but this doesn't seem to be hooked up to anything. jack.
Jack Moffitt wrote:
When you start your application:
from twisted.python import log log.startLogging(sys.stdout)
Now I've just deluged the poor user with lots of factory start and stops. All I want is normal HTTP style logs. Twisted's logging infrastructure has its place, but it is not quite ready to produce user friendly output for a simple utility.
That's probably a bug/feature request.
Or, even better, create an Application object and use twistd(1) to start your service. Using twistd(1), you have a choice to daemonize your process or not.
Great, now the user has to edit a tac file just to configure the thing. I'd rather them be able to type "tape" and not have to worry about twistd -ny some.tac. I use tac files a lot, but they don't make much sense for simple utilities. It sucks that it's either/or with Twisted applications.
I'm assuming "tape" is your application?
These sessions helped to reinforce for me the need to repeat this frequently for our users: if Twisted is doing something which confuses you but seems wrong, please go ahead and file a bug on twistedmatrix.com.
This mailing list should be indexed by Google as well. I do plan to file the bugs we find here as well.
* The only way to stick something in the tree at an arbitrary location seems to be to walk the tree to that spot, creating dummy locations as you go.
Override locateChild
What is that? I see no such function: Not in IResource: http://twistedmatrix.com/documents/current/api/twisted.web.resource.IResourc...
Not in the server.Site: http://twistedmatrix.com/documents/current/api/twisted.web.server.Site.html
grep -r doesn't show it in the twisted/web or any child directories. Not even in SVN trunk.
It's in nevow's appserver/resource sub-classes, not in t.web; not that this helps you of course. Basically, you're right - the resource dispatch in t.web is, to say the least, difficult. Sadly, you need to change both the http server, channel and request code to alter it (which is what Nevow does)
I find it extremely weird coming from several other web frameworks.
Twisted initially seems very different, no matter what framework you came from.
I grok Twisted just fine. This may be my first post to twisted-web, but I've been hacking on and around Twisted for some time. Even Apache doesn't work this way. I can tell apache that my proxy is at /some/url/over/here and it will work. It will return 404s for the intermediate URLs. Because Resources only have immediate children and paths are only dealt with in single element pieces, there doesn't seem to be a way to do this in Twisted Web. It feels weird that the tree must be complete. It feels weird that paths are dealt with in tiny sections.
I believe the eventual plan is to have t.web use the nevow appserver & resource model, with backwards compatibility stubs.
* Related to the above, the handling of foo vs. foo/ is pretty confusing. foo/ is considered the '' child of foo. This is pretty yuck to me. Unfortunately, I don't have any creative suggestions about how to do it better right now.
addSlash = True
This does not exist in Resource or in server.Site. What am I missing? It's not mentioned in the docs either.
Again, that's a nevow thing. This does have an equivalent in t.web - set isLeaf = True Basically, you want the nevow resource model. Sadly, that loses you the easy render_METHOD helpers, though you can easily re-add them by hooking renderHTTP and dispatching on a base class. To the devs: If there are a list of tickets for bringing the nevow stuff into t.web, I'd probably be able to spend some time working on them.
Now I've just deluged the poor user with lots of factory start and stops. All I want is normal HTTP style logs. Twisted's logging infrastructure has its place, but it is not quite ready to produce user friendly output for a simple utility.
That's probably a bug/feature request.
I think such bugs exist, but in case they don't, I'll file it. There just needs to be some control on what I want to see without having to subclass every bit of twisted to turn it off. For the most part I've not cared so much because we generate so much of our own logging in the daemons we build that the extra noise is negligible For end user logs though, anything more than the bare minimum is quickly overwhelming. ejabberd (and many other Erlang apps) have this same problem.
Great, now the user has to edit a tac file just to configure the thing. I'd rather them be able to type "tape" and not have to worry about twistd -ny some.tac. I use tac files a lot, but they don't make much sense for simple utilities. It sucks that it's either/or with Twisted applications.
I'm assuming "tape" is your application?
Yes. And by application, it's 150 lines in one file. Command line user oriented XMPP tools are another area where I hit this problem a lot. I essentially am reduced to bootstrapping all the infrastructure myself when twistd doesn't fit the problem. It makes simple examples very difficult to construct or blog about. All that dumb code gets in the way. Writing Twisted daemons is beautiful, and clients should be too.
Basically, you're right - the resource dispatch in t.web is, to say the least, difficult. Sadly, you need to change both the http server, channel and request code to alter it (which is what Nevow does) [and later...] I believe the eventual plan is to have t.web use the nevow appserver & resource model, with backwards compatibility stubs.
So we have Woven, Nevow, web, and web2. It's a giant mess. It's unclear to users which thing is the current thing, although there has been quite a bit of everyone pointing to twisted.web lately. They all have their own problems, and it's not clear to me that backwards compatibility is a great thing here. The problem space is well understood now, or at least better understood than several years back, and most of these were designed beforehand. Let legacy users add "from twisted.legacy import web" and let's make something actually useful in its place. What I love about twisted.web is that for the most part, it's extremely simple and easy to work with. I should note that it's awesome that I was able to write a small proxying web server in so few lines. It's just slightly frustrating that it is plagued by all these little issues. jack.
Jack Moffitt pisze:
It's a giant mess.
I don't think so.
Keep smoking whatever it is that you're smoking. It's better than my stuff. I bet you have unicorns for pets, too.
I am really sorry if you feel insulted; if so -- I didn't mean it; English is not my native language, maybe I did a mistake somewhere. I am using Twisted and Divmod software for some time. Being able to compare those with some other opensource projects out there, I definetley choose them over another -- also because of configurability (mentioned twistd(1) program is a wonderful example of pluggable architecture) and backward-compatibility. If you need any more help on twisted.web and nevow, feel free to post your questions here. I just don't think that dissing efforts of many developers to write backwards-compatible software by calling those efforts "a giant mess" is a sensible thing. Twisted is opensource. You can file bugs and send patches at http://twistedmatrix.com/ -- m
I just don't think that dissing efforts of many developers to write backwards-compatible software by calling those efforts "a giant mess" is a sensible thing.
Quite frankly, I think "dissing" is the wrong word here. It is a well know fact acknowledged by several twisted developers that the web situation is in fact "a giant mess", but as you probably know, there is significant effort going to remedy this. Twisted is opensource. You can file bugs and send patches at
http://twistedmatrix.com/ Not to come off as too much of a fanboy here, but Jack Moffitt does file bugs and send patches. Additionally, and just as important, he is an excellent evangelist of Twisted. Check out http://metajack.im/ and see his articles on Twisted. I for one look forward to reading them, and they are important for the community.
I myself have not (yet!) sent patches for Twisted, but I'm a grateful user that tries to give back in other ways. I personally evangelize Twisted, I have contribute a bit financially, and I currently work at University of California at San Diego on the Ocean Observatories Initiative (OOI [1]) project, and I have been pushing to make Twisted be a fundamental piece in the project's cyberinfracstructure, because I absolutely think it is the right tool for the job in many cases. So, in summary, Twisted FTW :-) thanks, Alex [1] http://www.oceanleadership.org/ocean_observing -- Alex Clemesha clemesha.org
So we have Woven, Nevow, web, and web2. It's a giant mess.
Totally agreed, FWIW. It's
unclear to users which thing is the current thing, I was this user a couple years ago. I knew that twisted had so much more greatness that just web, but others may not. I personally kept with it, which has paid off huge dividends.
although there has
been quite a bit of everyone pointing to twisted.web lately. They all have their own problems, and it's not clear to me that backwards compatibility is a great thing here. The problem space is well understood now, or at least better understood than several years back, and most of these were designed beforehand. Let legacy users add "from twisted.legacy import web" and let's make something actually useful in its place. Yes, please!
What I love about twisted.web is that for the most part, it's extremely simple and easy to work with. I should note that it's awesome that I was able to write a small proxying web server in so few lines. It's just slightly frustrating that it is plagued by all these little issues.
Agreed++ -- Alex Clemesha clemesha.org
Jack Moffitt pisze:
When you start your application:
from twisted.python import log log.startLogging(sys.stdout)
Now I've just deluged the poor user with lots of factory start and stops. All I want is normal HTTP style logs. Twisted's logging infrastructure has its place, but it is not quite ready to produce user friendly output for a simple utility.
twistd(1) starts logging automatically, giving you a command line option to redirect the logs to a file.
Or, even better, create an Application object and use twistd(1) to start your service. Using twistd(1), you have a choice to daemonize your process or not.
Great, now the user has to edit a tac file just to configure the thing.
tac files are standard Python files. twistd(1) is also a standard way to deploy Twisted applications. If you want the user just to run a file with Python, you will copy a lot of things that twistd(1) already does.
I'd rather them be able to type "tape" and not have to worry about twistd -ny some.tac.
$ echo "twistd -ny tape.tac" > tape; chmod +x ./tape; export PATH=.:$PATH
I use tac files a lot, but they don't make much sense for simple utilities.
Of course. That's why twistd(1) lets you start things right off the command line.
* The only way to stick something in the tree at an arbitrary location seems to be to walk the tree to that spot, creating dummy locations as you go.
Override locateChild
What is that? I see no such function: Not in IResource: http://twistedmatrix.com/documents/current/api/twisted.web.resource.IResourc...
Ahh. You're using twisted.web . I thought you're using Nevow. Nevow's rend.Page http://starship.python.net/crew/mwh/nevowapi/nevow.rend.Page.html
* Related to the above, the handling of foo vs. foo/ is pretty confusing. foo/ is considered the '' child of foo. This is pretty yuck to me. Unfortunately, I don't have any creative suggestions about how to do it better right now.
addSlash = True
This does not exist in Resource or in server.Site. What am I missing? It's not mentioned in the docs either.
See above. -- m
twistd(1) starts logging automatically, giving you a command line option to redirect the logs to a file.
Yes I know. And it has lots of information in those logs that I care nothing about, and no way to filter them without doing a lot of subclassing. I'm trying to make software users can use without getting lots of noise in their face.
Great, now the user has to edit a tac file just to configure the thing.
tac files are standard Python files. twistd(1) is also a standard way to deploy Twisted applications. If you want the user just to run a file with Python, you will copy a lot of things that twistd(1) already does.
Standard python files are not good user interfaces. Sorry. I realize I have to copy some stuff twistd does, but if my goal is to make friendly software, then that's what I have to do.
I'd rather them be able to type "tape" and not have to worry about twistd -ny some.tac.
$ echo "twistd -ny tape.tac" > tape; chmod +x ./tape; export PATH=.:$PATH
tac files are configuration files. If I have to load configuration files in the configuration file, isn't that defeating the point? I don't want the user to have to hand edit a tac file to change the listening port or which urls to reverse proxy. That's the whole problem I'm trying to solve here in the first place. twistd cannot take arbitrary command line options, and the Twisted core guys have told others that would e doing it wrong besides. jack.
Jack Moffitt pisze:
twistd(1) starts logging automatically, giving you a command line option to redirect the logs to a file.
Yes I know. And it has lots of information in those logs that I care nothing about, and no way to filter them without doing a lot of subclassing.
log.addObserver
Standard python files are not good user interfaces. Sorry. I realize I have to copy some stuff twistd does, but if my goal is to make friendly software, then that's what I have to do.
Why won't you write an opensource GUI for twistd(1) then? You could release a simplified version for your end-users then. At the moment, twistd(1) on my machine allows me to run 17 different services, including mail, web, ftp and ssh. If it's good for running those, I don't see why your code couldn't be made compatible with it.
tac files are configuration files. If I have to load configuration files in the configuration file, isn't that defeating the point?
I don't know, what platform are you running? chmod +x and shebang should do the trick on some, so you will have an executable configuration file, how about that?
That's the whole problem I'm trying to solve here in the first place. twistd cannot take arbitrary command line options,
twistd can do that, just add a plugin for it: http://twistedmatrix.com/projects/core/documentation/howto/tap.html -- m
OoO En ce début de soirée du mardi 07 avril 2009, vers 21:19, Jack Moffitt <jack@chesspark.com> disait :
tac files are configuration files. If I have to load configuration files in the configuration file, isn't that defeating the point? I don't want the user to have to hand edit a tac file to change the listening port or which urls to reverse proxy. That's the whole problem I'm trying to solve here in the first place. twistd cannot take arbitrary command line options, and the Twisted core guys have told others that would e doing it wrong besides.
You can pass arbitrary command line options by writing a plugin. http://twistedmatrix.com/projects/core/documentation/howto/tap.html -- panic("Foooooooood fight!"); 2.2.16 /usr/src/linux/drivers/scsi/aha1542.c
You can pass arbitrary command line options by writing a plugin. http://twistedmatrix.com/projects/core/documentation/howto/tap.html
Which, as I recall, Glyph and others have discourged others from doing this. Perhaps I am mistaken, but if twistd is intended for this purpose, why has this not become part of it all along? This is disregarding the whole fact that none of already existing twistd options are at all revelevant to my users, and only serve to provide more noise. jack.
Ugh. This thread is a mess. Can you guys quit the bullshit? On Tue, Apr 7, 2009 at 3:47 PM, Jack Moffitt <jack@chesspark.com> wrote:
You can pass arbitrary command line options by writing a plugin. http://twistedmatrix.com/projects/core/documentation/howto/tap.html
Which, as I recall, Glyph and others have discourged others from doing this. Perhaps I am mistaken, but if twistd is intended for this purpose, why has this not become part of it all along?
I don't think anyone's discouraging twistd plugins. They're nice for what they do. You may be thinking of mktap plugins. mktap is deprecated. I don't understand the grammar in "why has this not become part of it all along". Are you suggesting use of a time machine? I would love to, if you made one available to me.
This is disregarding the whole fact that none of already existing twistd options are at all revelevant to my users, and only serve to provide more noise.
If this is a command line non-daemon application, then you're right. twistd is not really geared towards those. There are some vague ideas for helping people write tools like that in a way that makes sense, but nothing very concrete. As for your initial list of things, I encourage you to file them as bugs (making sure you search for existing bugs first). They look pretty reasonable. Some of them are probably just documentation bugs. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ http://canonical.com/
Which, as I recall, Glyph and others have discourged others from doing this. Perhaps I am mistaken, but if twistd is intended for this purpose, why has this not become part of it all along?
I don't think anyone's discouraging twistd plugins. They're nice for what they do. You may be thinking of mktap plugins. mktap is deprecated.
I didn't mean that plugins were discouraged. I meant that contortions to allow twistd to take application specific command line options were actively discouraged. See the following threads: http://twistedmatrix.com/pipermail/twisted-python/2006-June/013331.html http://twistedmatrix.com/pipermail/twisted-python/2005-May/010541.html http://twistedmatrix.com/pipermail/twisted-python/2005-June/010857.html These are pretty clear I think. twistd is not intended for this use case. The alternative currently is to embed config file reading into the tac (only solves part of the problem), or redo all the start up application infrastructure that twistd makes so nice.
I don't understand the grammar in "why has this not become part of it all along". Are you suggesting use of a time machine? I would love to, if you made one available to me.
What I meant is if writing a plugin for adding your own options to twistd is the recommended solution, why doesn't twistd just do this for us? Surely someone would have done it by now. As we can see from the threads above, it's not done because they don't want it done. It makes sense to me why this is so, and so I didn't attempt to do it this way in Tape.
This is disregarding the whole fact that none of already existing twistd options are at all revelevant to my users, and only serve to provide more noise.
If this is a command line non-daemon application, then you're right. twistd is not really geared towards those. There are some vague ideas for helping people write tools like that in a way that makes sense, but nothing very concrete.
It's a command line non-daemon app, but even if it were a temporary helper daemon you launched now and then, it would still apply. Twisted needs better help for this case, and that's what I've been attempting to say here all along.
As for your initial list of things, I encourage you to file them as bugs (making sure you search for existing bugs first). They look pretty reasonable. Some of them are probably just documentation bugs.
Will do. Thanks for your feedback. jack.
On Tue, Apr 7, 2009 at 4:12 PM, Jack Moffitt <jack@chesspark.com> wrote:
Which, as I recall, Glyph and others have discourged others from doing this. Perhaps I am mistaken, but if twistd is intended for this purpose, why has this not become part of it all along?
I don't think anyone's discouraging twistd plugins. They're nice for what they do. You may be thinking of mktap plugins. mktap is deprecated.
I didn't mean that plugins were discouraged. I meant that contortions to allow twistd to take application specific command line options were actively discouraged. See the following threads:
http://twistedmatrix.com/pipermail/twisted-python/2006-June/013331.html http://twistedmatrix.com/pipermail/twisted-python/2005-May/010541.html http://twistedmatrix.com/pipermail/twisted-python/2005-June/010857.html
These are pretty clear I think. twistd is not intended for this use case. The alternative currently is to embed config file reading into the tac (only solves part of the problem), or redo all the start up application infrastructure that twistd makes so nice.
I believe that all of those threads are in the context of tac files, which are the things you use with the -y parameter to twistd. 'twistd plugins' are a different thing, and one of the main design goals for them was to take command line arguments. Also, those threads are very old, and certain opinions may have changed. Anyway, I'm sure I know about the intended usage for twistd plugins, because I implemented the plugin system for twistd, and I did it with Jean-Paul's blessing. You can trust me. :-) I don't want to sound like I think twistd plugins are perfect, though. They actually suck a lot, for various reasons. I hope you read my words carefully and realize I'm only trying to clear up the piles upon piles of miscommunication and bad advice in this thread by getting some facts straight. For the record: I agree that the web with Twisted situation is a mess, unlike Michal, and I think the TwistedWebPlan all but admits that directly (hey, we have to be a *little* bit cheerful, right?). A *lot* of work has gone into twisted.web-related stuff recently, and a lot more is going to be put into it. This is mostly thanks to people who have donated money to Twisted. If anyone wants this work to continue, I strongly encourage you to donate (or get your company to)! If you want to do a personal donation, then you can use paypal or Google Checkout by visiting http://twistedmatrix.com/ (The donation area in on the left side, right underneath the snakeball book). More information about sponsorship is available at: http://twistedmatrix.com/trac/wiki/TwistedSoftwareFoundation
I don't understand the grammar in "why has this not become part of it all along". Are you suggesting use of a time machine? I would love to, if you made one available to me.
What I meant is if writing a plugin for adding your own options to twistd is the recommended solution, why doesn't twistd just do this for us? Surely someone would have done it by now.
I really can't understand you. The way I'm reading your first sentence in this paragraph is so silly that I must be misreading the intention. Maybe the problem is that you don't understand what twistd plugins really are. If you care, I recommend reading the URL that Michal referred to earlier: http://twistedmatrix.com/projects/core/documentation/howto/tap.html
As we can see from the threads above, it's not done because they don't want it done. It makes sense to me why this is so, and so I didn't attempt to do it this way in Tape.
As I pointed out above, those threads are either about something else or are no longer representative of current opinion.
This is disregarding the whole fact that none of already existing twistd options are at all revelevant to my users, and only serve to provide more noise.
If this is a command line non-daemon application, then you're right. twistd is not really geared towards those. There are some vague ideas for helping people write tools like that in a way that makes sense, but nothing very concrete.
It's a command line non-daemon app, but even if it were a temporary helper daemon you launched now and then, it would still apply. Twisted needs better help for this case, and that's what I've been attempting to say here all along.
I can't speak for the others in this thread, but if anyone denied that Twisted needs help in the deployment or web areas, they're wrong. :-)
As for your initial list of things, I encourage you to file them as bugs (making sure you search for existing bugs first). They look pretty reasonable. Some of them are probably just documentation bugs.
Will do. Thanks for your feedback.
You're welcome. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ http://canonical.com/
On Tue, Apr 7, 2009 at 4:39 PM, Christopher Armstrong <radix@twistedmatrix.com> wrote:
recently, and a lot more is going to be put into it. This is mostly thanks to people who have donated money to Twisted. If anyone wants
Sorry, I wanted to add something to this, but it got lost in revision: This is mostly thanks to people who have donated money to Twisted (such as Chesspark. Thank you, Jack Moffit! Do you have any more money for us? :). -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ http://canonical.com/
I really can't understand you. The way I'm reading your first sentence in this paragraph is so silly that I must be misreading the intention. Maybe the problem is that you don't understand what twistd plugins really are. If you care, I recommend reading the URL that Michal referred to earlier:
http://twistedmatrix.com/projects/core/documentation/howto/tap.html
Thanks for the second nudge here. The big note about how this is not the same as tap files cleared it up for me. So to get what I want, I'd write a plugin for Tape, then wrap that with some user friendly python to hide the non-essential command line options or the fact that I need twistd. It sounds like a reasonable first step to making this better would be to have some way to hook into twistd from the code side. Perhaps this is already possible? jack.
On Tue, Apr 7, 2009 at 5:08 PM, Jack Moffitt <jack@chesspark.com> wrote:
I really can't understand you. The way I'm reading your first sentence in this paragraph is so silly that I must be misreading the intention. Maybe the problem is that you don't understand what twistd plugins really are. If you care, I recommend reading the URL that Michal referred to earlier:
http://twistedmatrix.com/projects/core/documentation/howto/tap.html
Thanks for the second nudge here. The big note about how this is not the same as tap files cleared it up for me.
So to get what I want, I'd write a plugin for Tape, then wrap that with some user friendly python to hide the non-essential command line options or the fact that I need twistd.
It sounds like a reasonable first step to making this better would be to have some way to hook into twistd from the code side. Perhaps this is already possible?
The twistd API is almost non-existent, unfortunately. However, you can get a long way by doing something like this in your "main script": from twisted.scripts.twistd import run sys.argv[1:] = ["pluginname", "-whatever"] run() It's decidedly a hack, but it works and relies only on public APIs, so it's tolerable until we get a better API. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ http://canonical.com/
from twisted.scripts.twistd import run sys.argv[1:] = ["pluginname", "-whatever"] run()
It's decidedly a hack, but it works and relies only on public APIs, so it's tolerable until we get a better API.
That sounds like the best bet for now. If I get some time perhaps I'll do this for Tape and then write it up so others in a similar situation can get a little guidance. It will make for a reasonable second plugin tutorial as well. jack.
On Apr 7, 2009, at 3:58 PM, Christopher Armstrong wrote:
If this is a command line non-daemon application, then you're right. twistd is not really geared towards those. There are some vague ideas for helping people write tools like that in a way that makes sense, but nothing very concrete.
On Apr 7, 2009, at 5:08 PM, Jack Moffitt wrote:
So to get what I want, I'd write a plugin for Tape, then wrap that with some user friendly python to hide the non-essential command line options or the fact that I need twistd.
It sounds like a reasonable first step to making this better would be to have some way to hook into twistd from the code side. Perhaps this is already possible?
I was wondering if I could get clarification on the anticipated problems with using twistd to run non-daemonizing CLI applications. It seems like you'd just need to return some kind of custom Service from makeService, which would call your app's main() from startService, and then call reactor.stop() when your app is finished doing whatever it needs to do. Then you could still use twistd's logging and other goodies, as well as parsing your own command line with t.p.usage. Add a quick one- or two-line shell script to hide the twistd part of the incantation, and it seems pretty good. Maybe there's another issue I'm not thinking of, though... -phil
So to get what I want, I'd write a plugin for Tape, then wrap that with some user friendly python to hide the non-essential command line options or the fact that I need twistd.
Just to correct myself for posterity, it looks like twistd plugins actually automatically hide the normal twistd command line options. I noticed this when playing around with twistd web. This is great! No my "wrapper" is down to just a shell script, which is what Michal was suggesting all along I'm guessing. I can save the fork by calling twisted.scripts.twistd.run as Christopher suggested as well. jack.
On 7 Apr, 07:47 pm, jack@chesspark.com wrote:
You can pass arbitrary command line options by writing a plugin. �http://twistedmatrix.com/projects/core/documentation/howto/tap.html
Which, as I recall, Glyph and others have discourged others from doing this. Perhaps I am mistaken, but if twistd is intended for this purpose, why has this not become part of it all along?
Chris has already basically set the record straight here, but for future readers I just wanted to have a direct reply to this message and clear up a very unfortunate choice of acronyms. Some of this is a summary of other things that have already been mentioned in this thread. I definitely do not discourage the use of twistd for this purpose :). In fact I strongly recommend it. Here's the history. ".tap" ("Twisted Application Pickle") files, and the associated "mktap" ("make twisted application pickle") command-line utility, are strongly discouraged. They've been deprecated for a long time, and they'll be gone soon. "TAPs" ("Twisted Application *Plugins*") were originally plugins for "mktap". That's why the naming is similar. However, *they are now plugins for twistd*, and have been for some time. Developers are strongly encouraged to use this interface, and to read the document linked above. Before "TAPs" were plugins for twistd, the suggested configuration mechanism for twisted applications was to write a '.tac' file and tell your users to edit it. Processing twistd's commandline arguments at this point was impossible without gratuitous hacks, so tac files were a better alternative. However, as Jack noted, editing tac files is inconvenient, so we added the twistd plugin interface. We now recommend that adding command-line arguments to your 'twistd' plugin is the way to go if you want run-time configuration for your twisted server application. The technique that Chris suggested elsewhere in this thread, to set up correct command-line arguments for your plugin and then call twistd's run() function, is the currently supported way to invoke twistd's functionality from your own scripts. This is what we use to support Divmod's "axiomatic" command, which is basically a database-configured version of "twistd". A more expressive API for a wider variety of use- cases is forthcoming, but this technique should still work even once such an API has been added. The document in question would really benefit from some updating; in particular, it would be good to move the mention of TAP files to some kind of endnote rather than being mentioned before the actual relevant features.
Thanks for the clarification! -martin -----Original Message----- From: twisted-web-bounces@twistedmatrix.com [mailto:twisted-web-bounces@twistedmatrix.com] On Behalf Of glyph@divmod.com Sent: Tuesday, April 07, 2009 10:04 PM To: Twisted Web World Subject: Re: [Twisted-web] some questions about twisted.web On 7 Apr, 07:47 pm, jack@chesspark.com wrote:
You can pass arbitrary command line options by writing a plugin. http://twistedmatrix.com/projects/core/documentation/howto/tap.html
Which, as I recall, Glyph and others have discourged others from doing this. Perhaps I am mistaken, but if twistd is intended for this purpose, why has this not become part of it all along?
Chris has already basically set the record straight here, but for future readers I just wanted to have a direct reply to this message and clear up a very unfortunate choice of acronyms. Some of this is a summary of other things that have already been mentioned in this thread. I definitely do not discourage the use of twistd for this purpose :). In fact I strongly recommend it. Here's the history. ".tap" ("Twisted Application Pickle") files, and the associated "mktap" ("make twisted application pickle") command-line utility, are strongly discouraged. They've been deprecated for a long time, and they'll be gone soon. "TAPs" ("Twisted Application *Plugins*") were originally plugins for "mktap". That's why the naming is similar. However, *they are now plugins for twistd*, and have been for some time. Developers are strongly encouraged to use this interface, and to read the document linked above. Before "TAPs" were plugins for twistd, the suggested configuration mechanism for twisted applications was to write a '.tac' file and tell your users to edit it. Processing twistd's commandline arguments at this point was impossible without gratuitous hacks, so tac files were a better alternative. However, as Jack noted, editing tac files is inconvenient, so we added the twistd plugin interface. We now recommend that adding command-line arguments to your 'twistd' plugin is the way to go if you want run-time configuration for your twisted server application. The technique that Chris suggested elsewhere in this thread, to set up correct command-line arguments for your plugin and then call twistd's run() function, is the currently supported way to invoke twistd's functionality from your own scripts. This is what we use to support Divmod's "axiomatic" command, which is basically a database-configured version of "twistd". A more expressive API for a wider variety of use- cases is forthcoming, but this technique should still work even once such an API has been added. The document in question would really benefit from some updating; in particular, it would be good to move the mention of TAP files to some kind of endnote rather than being mentioned before the actual relevant features. _______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Jack Moffitt <jack@chesspark.com> writes:
Now I've just deluged the poor user with lots of factory start and stops. All I want is normal HTTP style logs. Twisted's logging infrastructure has its place, but it is not quite ready to produce user friendly output for a simple utility.
I generally just use the default logging but then it's mostly for my own use, not end users. As an interim to any more extensive changes, perhaps you could just use a local HTTPFactory subclass for your site, overriding with your own log method that bypasses the twisted logging to do whatever you want? You could then use twisted logging (if preferred) for lower level logs and do some higher level stuff for user logging. If you're looking for
Great, now the user has to edit a tac file just to configure the thing. I'd rather them be able to type "tape" and not have to worry about twistd -ny some.tac. I use tac files a lot, but they don't make much sense for simple utilities. It sucks that it's either/or with Twisted applications.
For what it's worth, I've just never seemed to get into any of the services or applications stuff with twisted. It's certainly not a requirement to run a server. You can implement your own daemon support on a Unix platform, or run as a service under Windows or whatever. There's nothing stopping you from just instantiating the right factory classes, associating them with a port and running the reactor. That can then be a single python module that gets run directly, or if built into an application (such as py2exe on Windows), executed directly. I normally have more than just the web server running in a single process, but all you'd probably need is to instantiate your "Site" class, connect it with reactor.listenTCP, and then sit in reactor.run().
I grok Twisted just fine. This may be my first post to twisted-web, but I've been hacking on and around Twisted for some time. Even Apache doesn't work this way. I can tell apache that my proxy is at /some/url/over/here and it will work. It will return 404s for the intermediate URLs. Because Resources only have immediate children and paths are only dealt with in single element pieces, there doesn't seem to be a way to do this in Twisted Web. It feels weird that the tree must be complete. It feels weird that paths are dealt with in tiny sections.
As another view on this point, for me, I find the "work segment by segment" approach of twisted's to be very natural, albeit different than an arbitrary path mapping mechanism such as apache or many web frameworks. I actually missed the incremental lookup mechanism when first working with django on a project, for example. It can feel weird compared to other mechanisms, but just because it doesn't match how Apache works need not make it a problem. There's more than one way to navigate an object hierarchy within a server based on the URL in a request. Since it's inherently simple in terms of using the getChild* methods (or resource.getChildForRequest helper function), it's easy to override to make very dynamic sites, which I guess I find myself doing more often than not. It can be very flexible when it comes to dynamically picking the resource to handle a request. Yes, for a static (URL wise) portion of a site you do need to construct a "tree" of resources, so I just generally do that at initialization time. At worst you can stub out any piece of the tree with a plain old Resource object to which you can add further children. That's not so say supporting a more "mapper" based approach on top of this might not be useful, if only to assist people more used to that approach. But I definitely wouldn't want to lose the dynamic capabilities provided by the current system. One way to tie into the current framework to take more control is to define the resource at the root of your mapper-capable tree using "isLeaf". That resource's "render" method will then get called for any URLs at that point or below, have access to the full URL (already split into segments), and it can do whatever lookups/mapping to resource objects it might like. I've used that to strip out URL prefixes (such as client keys unique to them) and then re-perform the URL lookup to locate the resource, but it could just as easily do a mapper based URL lookup at that point to yield the right resource object to then call (instantiating it if needed) to render the request. -- David
I generally just use the default logging but then it's mostly for my own use, not end users.
We do as well. This app in particular is end user focused, so I wanted something nicer.
As an interim to any more extensive changes, perhaps you could just use a local HTTPFactory subclass for your site, overriding with your own log method that bypasses the twisted logging to do whatever you want?
That's essentially what I did in the code. It's not pretty though.
I normally have more than just the web server running in a single process, but all you'd probably need is to instantiate your "Site" class, connect it with reactor.listenTCP, and then sit in reactor.run().
Yep. Tape currently does this, and all my command line XMPP tools do as well. There's a lot of bootstrap though, and since most people I know starting out want to write command line type utilities, this makes learning Twisted significantly harder. At least making a tac file is easy. You just make some services, call setServiceParent and you're done. All that bootstrapping happens out of your view.
As another view on this point, for me, I find the "work segment by segment" approach of twisted's to be very natural, albeit different than an arbitrary path mapping mechanism such as apache or many web frameworks. I actually missed the incremental lookup mechanism when first working with django on a project, for example. It can feel weird compared to other mechanisms, but just because it doesn't match how Apache works need not make it a problem. There's more than one way to navigate an object hierarchy within a server based on the URL in a request.
I think it's fine to support both approaches. I think my original complaint was actually worded that way. The issue is that there may not be a path from root to a child. Ie, /some/path/to/proxy exists, but /some/path doesn't. It takes some jiggering to set that up right now, because you have to make dummy nodes along the path. Which works, but is not as optimal for this use case.
One way to tie into the current framework to take more control is to define the resource at the root of your mapper-capable tree using "isLeaf". That resource's "render" method will then get called for any URLs at that point or below, have access to the full URL (already split into segments), and it can do whatever lookups/mapping to resource objects it might like. I've used that to strip out URL prefixes (such as client keys unique to them) and then re-perform the URL lookup to locate the resource, but it could just as easily do a mapper based URL lookup at that point to yield the right resource object to then call (instantiating it if needed) to render the request.
When the first getChild is called how do I know it's the first getChild call so that I do the special logic? Perhaps that's what is confusing me. jack.
Jack Moffitt <jack@chesspark.com> writes:
When the first getChild is called how do I know it's the first getChild call so that I do the special logic? Perhaps that's what is confusing me.
Well, "first getChild" is ambiguous to me. If you mean the getChild on the instance that is supposed to do non-default mapping, I'd suggest you know simply because the Resource class that implements it has to get instantiated and reached somehow by the normal lookup mechanism, so once it gets control, by definition that's the right time. Or put another way, you only instantiate and install into the tree the object supporting the new lookup mechanism just where you need it, so as far as that object instance is concerned, if it's being called, it needs the special logic. This of course assumes that you aren't trying to implement a single class for both lookup types, but rather have a Resource subclass designed for use as the new lookup mechanism. If you made such a class your Root object, then essentially your whole site would be mapped using the new mechanism. If you stuck it further down (such as via a putChild on lower level resources, or by dynamically returning it in some othera custom getChild), then the new lookups start taking place beyond that point in the URL. An important point mentioned in some of the other responses is how the isLeaf Resource attribute gets used in traversal. When Twisted is working its way down a URL and traversing objects through getChild, if it hits one with isLeaf=True, it will stop at that point and let that object render the result, even if the URL hasn't been exhausted. During rendering the Resource will have access to the prepath and postpath attributes on the current request indicating the portion of the URL used to reach the current Resource, and any remaining portion. So two approaches you might follow: * Intercept the getChild lookup mechanism itself within the instance of your specific Resource object - use whatever mapping approach you want, and return an appropriate Resource. That Resource object, when initially instantiated for use by the site, might have its own configuration file, cache of instantiated Resource objects, or whatever. Assuming that the mapping process yields the final Resource for the matched URL you'll want to ensure the resource you return is used for the rendering, even though request.postpath at that point may not be empty. One way to handle this would be to ensure that isLeaf=True on any returned Resource (your mapping class can even set this if you don't want to require authors of the Resource objects themselves to care) so that traversal stops at that point and they are directly called to render the page. One drawback to using isLeaf on the returned resource might be that the request postpath won't be accurate during rendering as it will still reflect the location of your mapping Resource rather than the rendering resource. If that's an issue (say for computed links during rendering), you could alternatively leave isLeaf=False in the returned Resource but ensure that its getChild() always returns itself, which would let the normal Twisted mechanism eat through the URL but never leave the Resource. Or yet another approach - if you manipulate request.prepath/postpath during a getChild call, you'll control the traversal, so simply clearing postpath (presumably after appending it to prepath) should guarantee the traversal stops with the Resource object you're returning, while leaving prepath/postpath accurate during rendering. * Define the mapping Resource itself with isLeaf=True, and then during its render() operation, do whatever dynamic lookups you need to in order to locate the appropriate Resource, and return the result of its render() operation. This approach also allows the mapping operation itself to be deferrable since its occurring during the render operation rather than the initial object traversal. In this case, since you are in control of calling the final Resource object's render(), there's no worry about further object traversal, but you may still want to manipulate prepath/postpath in the active request object to have appropriate values for use by the final rendering Resource. As an example of the latter, I have a site that accepts URLs of the form: http://<site>/approval/<key>/x/y/z where <key> is a unique key handed out to clients via email to gate their access to their data. I need to use <key> to validate and identify the client, but the rest of the URL is a fairly static association of Resource objects per the normal Twisted "putChild" setup. However, once the client is validated, a job UUID for them is automatically appended to the URL for use by any eventual Resource. I'll grant that this scenario is not necessarily the greatest argument for the Twisted traversal mechanism versus an RE-based mapper (assuming the latter lets you save portions of the matched URL for the use of the rendering object), but it does show a dynamic (and deferred) processing mechanism within twisted.web. I have an ApprovalRoot Resource subclass (with isLeaf=True), that was tied in under my site's Root object (with putChild), as shown below. It uses the approval key to do some database lookups and validations, then re-uses the same getChildForRequest function Twisted itself uses for traversal on the remainder of the path (popping off the approval key first), via an internal Resource tree. In this case, since the second level lookup is processing the Request object, prepath/postpath end up correct for the render() call without further intervention. If you replace the lookup with your own traversal mapping object, it could just as easily map resources in other ways. Of course, if the mapping mechanism had no requirement for deferrable lookups, doing the processing during the getChild operations is probably a little more logical. -- David - - - - - - - - - - - - - - - - - - - - - - - - - During initialization, site setup includes (among other resources): # Main site URL entry points self.root = Root(self.options) msg_root = ApprovalRoot(db, self.options) self.root.putChild('approval', msg_root) which uses the following class: class ApprovalRoot(Resource): """Act as root of the approval tree, which is accessed from URLs in messages, and always include the message key as the first part of request.postpath. Strips off the key, validates it, and then passes control on to appropriate job or file based objects depending on the remainder of the URL. This is almost identical to normal child lookup by non-leaf objects, but handled at render time since the message key validation is a deferred operation.""" isLeaf = True def __init__(self, db, options): Resource.__init__(self) self.db = db self.loader = options['loader'] # Use a separate resource as the root of the remaining URL processing # since the isLeaf on ourselves would defeat any child search self.job_root = Resource() def putChild(self, path, child): """Permit simulated children, so that the overall structure of the web site can still be established in a higher level function""" self.job_root.putChild(path, child) def _db_retrieveJobUuid(self, key): sql = sa.select([schema.jobs.c.uuid, schema.messages.c.expiration], sa.and_(schema.jobs.c.uuid == schema.messages.c.job_uuid, schema.messages.c.key == key)) r = sql.execute().fetchone() if not r: raise _Unavailable elif (r.expiration and r.expiration < datetime.utcnow()): raise NoResource('The email approval key has expired') else: return r.uuid def _cb_render(self, job_uuid, request): # Transfer control to the appropriate child for rendering. In the # case of a top level render, modify the postpath to include the job # uuid as an argument. if request.postpath and not request.postpath[0]: request.postpath.append(job_uuid.hex) child = getChildForRequest(self.job_root, request) r = child.render(request) if r != NOT_DONE_YET: request.write(r) request.finish() def _cb_render_err(self, failure, request): if failure.check(NoResource): request.write(failure.value.render(request)) request.finish() return return failure def _finishRequest(self, value, request): request.finish() return value def render(self, request): if len(request.postpath) < 1: return ErrorPage(http.NOT_FOUND, 'Missing approval reference', '').render(request) # We only render message key failures, so if the URL has no further # segments beyond the key, add a trailing "/" to trigger the child # lookup for the default handler. if len(request.postpath) == 1: request.redirect(request.prePathURL() + '/' + request.postpath[0] + '/') request.finish() else: msg_key = request.postpath.pop(0) d = self.db.run(self._db_retrieveJobUuid, msg_key) d.addCallback(self._cb_render, request) d.addErrback(self._cb_render_err, request) d.addErrback(self._finishRequest, request) d.addErrback(log.err) return NOT_DONE_YET
Jack Moffitt <jack@chesspark.com> writes:
I think it's fine to support both approaches. I think my original complaint was actually worded that way. The issue is that there may not be a path from root to a child. Ie, /some/path/to/proxy exists, but /some/path doesn't. It takes some jiggering to set that up right now, because you have to make dummy nodes along the path. Which works, but is not as optimal for this use case.
Oh, one follow-on to my last response. While I've been referring to getChild while talking about the traversal, technically its getChildWithDefault that is being used with standard Resource instances. This gives precedence to explicit child resource instances (added with putChild), falling back to getChild if no matching static child has been added. So in terms of a custom mapping Resource, you could install this as the Root of your site, implement the custom getChild mapping, and still use putChild to install some static paths that didn't use the new mapping. In such a case, getChild would only be getting called if no static children were found, so again, the fact that it was being called at all would imply that you should fall back to your custom mapping. The one case you would probably have to check for would be a getChild lookup for "", which should render the actual site root page contents, unless you wanted to delegate that to some other resource by using putChild with a path of "". -- David
participants (11)
-
Alex Clemesha -
Christopher Armstrong -
Christopher Armstrong -
David Bolen -
glyph＠divmod.com -
Jack Moffitt -
Martin Bright -
Michał Pasternak -
Phil Christensen -
Phil Mayers -
Vincent Bernat