From ianb at colorstudy.com Wed Jun 1 05:56:38 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 31 May 2005 22:56:38 -0500 Subject: [Web-SIG] Google Summer Of Code Message-ID: <429D31F6.5030506@colorstudy.com> I've started a page for ideas for the Google Summer Of Code (http://code.google.com/summerofcode.html) specifically with ideas relating to Python and web programming. If you have any others, please add them on. Also, I think the deal involves some level of individual advisorship/mentorship -- the PSF has to approve the projects at some level, but obviously the PSF itself can't be a mentor. I think some relationship is required (though maybe Google is thinking that they'd hand off a bunch of proposals, and the sponsoring organizations would choose the ones they want to mentor?) -- either way it can't hurt for people to indicate an interest in mentorship. Anyway, the page is here: http://wiki.python.org/moin/PythonWebProgrammingIdeas -- my ideas are all focused on WSGI applications, which is kind of abstract and maybe a little on the boring/low-level side ;) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From sanxiyn at gmail.com Thu Jun 2 19:44:56 2005 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Fri, 3 Jun 2005 02:44:56 +0900 Subject: [Web-SIG] CSS selector parsing Message-ID: <5b024817050602104429fd8668@mail.gmail.com> Hello, I am new here. Web SIG charter says: "HTML and XML parsing are pretty solid, but a critical lack on the client side is the lack of a CSS parser." Is there any progress on a CSS parser? Any prior art? I wrote a quick one using SimpleParse. You can read it here: http://deadbeefbabe.org/paste/765 What do you think? Seo Sanghyeon From jjinux at gmail.com Fri Jun 3 00:10:22 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Thu, 2 Jun 2005 15:10:22 -0700 Subject: [Web-SIG] HTMLTemplate Message-ID: Anyone have any comments on HTMLTemplate? Thanks, -jj -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through. From ianb at colorstudy.com Fri Jun 3 00:25:45 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 02 Jun 2005 17:25:45 -0500 Subject: [Web-SIG] HTMLTemplate In-Reply-To: References: Message-ID: <429F8769.7000100@colorstudy.com> Shannon -jj Behrens wrote: > Anyone have any comments on HTMLTemplate? Which one? I think there's a few. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From mike_mp at zzzcomputing.com Fri Jun 3 00:42:47 2005 From: mike_mp at zzzcomputing.com (mike bayer) Date: Thu, 2 Jun 2005 18:42:47 -0400 (EDT) Subject: [Web-SIG] HTMLTemplate In-Reply-To: References: Message-ID: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> referring to this one: http://freespace.virgin.net/hamish.sanderson/htmltemplate.html it looks very cool and elegant, and clearly produces templates that are super-clean. however, I would wonder how convenient it really is to create 100% of all the programmatically generated content in your "controller" module, including even the most trivial concatenation of strings. I would think that for a complicated page design with alot of data embedded in it, this would lead to a much bigger mess of HTML mixed with code in the controller class than the one it seeks to prevent in the HTML template. > Anyone have any comments on HTMLTemplate? > > Thanks, > -jj > > -- > I have decided to switch to Gmail, but messages to my Yahoo account will > still get through. > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: > http://mail.python.org/mailman/options/web-sig/mike_mp%40zzzcomputing.com > From jonathan at carnageblender.com Fri Jun 3 00:56:43 2005 From: jonathan at carnageblender.com (Jonathan Ellis) Date: Thu, 02 Jun 2005 15:56:43 -0700 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> References: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> Message-ID: <1117753003.17379.235532492@webmail.messagingengine.com> On Thu, 2 Jun 2005 18:42:47 -0400 (EDT), "mike bayer" said: > > referring to this one: > > http://freespace.virgin.net/hamish.sanderson/htmltemplate.html > > it looks very cool and elegant, and clearly produces templates that are > super-clean. however, I would wonder how convenient it really is to > create 100% of all the programmatically generated content in your > "controller" module, including even the most trivial concatenation of > strings. I would think that for a complicated page design with alot of > data embedded in it, this would lead to a much bigger mess of HTML mixed > with code in the controller class than the one it seeks to prevent in the > HTML template. Hmm... The idea in a template system is, the model provides the data and the template interpolates it. So in this case, the template engine takes the template
  • LINK
  • and a Python list and combines them to give
  • Home
  • Products
  • About
  • No manual concatenation is involved. The callback organization is interesting but seems more cumbersome than the more-traditional approach you see in Cheetah et al. I guess if you are absolutely allergic to any sort of code whatsoever in your presentation layer, though, HTMLTemplate might be the way to go. :) -Jonathan From ianb at colorstudy.com Fri Jun 3 00:58:11 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 02 Jun 2005 17:58:11 -0500 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> References: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> Message-ID: <429F8F03.2030205@colorstudy.com> mike bayer wrote: > referring to this one: > > http://freespace.virgin.net/hamish.sanderson/htmltemplate.html > > it looks very cool and elegant, and clearly produces templates that are > super-clean. however, I would wonder how convenient it really is to > create 100% of all the programmatically generated content in your > "controller" module, including even the most trivial concatenation of > strings. I would think that for a complicated page design with alot of > data embedded in it, this would lead to a much bigger mess of HTML mixed > with code in the controller class than the one it seeks to prevent in the > HTML template. I imagine myself writing a mini-language in the form of complex id and class name conventions to automate this process. I am using something vaguely similar to this in a project. Clients write HTML using a WYSIWYG editor, and the "template" has conventions about what certain tags and classes mean. Then there's some transformations that the template itself can produce. E.g., we produce a table of contents by finding all the h3 tags, and in ZPT it looks like: In this case "index" is a special object (bound to that content), and index[tagname] (in TAL this is index/tagname) returns a list of all the tags found in that content. item/anchor return #id, and sets an id on the element if one doesn't exist. And so on. These little transforming objects can strip content out, change classes, etc. But I definitely wouldn't use this for the entire template. It's useful because the authors really can't be expected to know how to template anything, and the in-browser WYSIWYG editor isn't a good environment to compose abstract templates. It works well when contained inside a more self-contained templating system like ZPT. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Fri Jun 3 01:01:39 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 02 Jun 2005 18:01:39 -0500 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <1117753003.17379.235532492@webmail.messagingengine.com> References: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> <1117753003.17379.235532492@webmail.messagingengine.com> Message-ID: <429F8FD3.3040706@colorstudy.com> Jonathan Ellis wrote: > The callback organization is interesting but seems more cumbersome than > the more-traditional approach you see in Cheetah et al. I guess if you > are absolutely allergic to any sort of code whatsoever in your > presentation layer, though, HTMLTemplate might be the way to go. :) If there's no code then it's not a presentation layer. It's an HTML layer, nothing more, just dumb data. Presentation requires logic. From that perspective, I think doing proper model/presentation separation using something like HTMLTemplate requires discipline and the addition of another Python presentation layer. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ksenia at ksenia.nl Fri Jun 3 01:04:52 2005 From: ksenia at ksenia.nl (Ksenia Marasanova) Date: Fri, 3 Jun 2005 01:04:52 +0200 Subject: [Web-SIG] HTMLTemplate In-Reply-To: References: Message-ID: <489AB64A-B328-4698-AF5D-B9864C633456@ksenia.nl> Op 3-jun-2005, om 0:10 heeft Shannon -jj Behrens het volgende geschreven: > Anyone have any comments on HTMLTemplate? I used it quite a lot, and while it is fast, bugs free and simple to use, total separation of markup and code is IMHO contraproductive - every simple change mostly requires changing two files. --Ksenia From jjinux at gmail.com Fri Jun 3 01:30:14 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Thu, 2 Jun 2005 16:30:14 -0700 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <489AB64A-B328-4698-AF5D-B9864C633456@ksenia.nl> References: <489AB64A-B328-4698-AF5D-B9864C633456@ksenia.nl> Message-ID: Everyone, thanks for your comments! Ksenia, that's exactly the kind of experience-backed opinion I was looking for ;) Ian, I also especially liked your comment, "If there's no code then it's not a presentation layer. It's an HTML layer, nothing more, just dumb data. Presentation requires logic." Thanks Again, -jj On 6/2/05, Ksenia Marasanova wrote: > Op 3-jun-2005, om 0:10 heeft Shannon -jj Behrens het volgende > geschreven: > > > Anyone have any comments on HTMLTemplate? > > I used it quite a lot, and while it is fast, bugs free and simple to > use, total separation of markup and code is IMHO contraproductive - > every simple change mostly requires changing two files. -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through. From jjinux at gmail.com Fri Jun 3 01:53:50 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Thu, 2 Jun 2005 16:53:50 -0700 Subject: [Web-SIG] HTMLTemplate In-Reply-To: References: <489AB64A-B328-4698-AF5D-B9864C633456@ksenia.nl> Message-ID: I blogged on the subject: http://jjinux.blogspot.com/ Thanks, -jj On 6/2/05, Shannon -jj Behrens wrote: > Everyone, thanks for your comments! Ksenia, that's exactly the kind > of experience-backed opinion I was looking for ;) Ian, I also > especially liked your comment, "If there's no code then it's not a > presentation layer. It's an HTML layer, nothing more, just dumb data. > Presentation requires logic." > > Thanks Again, > -jj > > On 6/2/05, Ksenia Marasanova wrote: > > Op 3-jun-2005, om 0:10 heeft Shannon -jj Behrens het volgende > > geschreven: > > > > > Anyone have any comments on HTMLTemplate? > > > > I used it quite a lot, and while it is fast, bugs free and simple to > > use, total separation of markup and code is IMHO contraproductive - > > every simple change mostly requires changing two files. > > -- > I have decided to switch to Gmail, but messages to my Yahoo account will > still get through. > -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through. From tim at skreak.com Fri Jun 3 04:30:54 2005 From: tim at skreak.com (Tim Gilbert) Date: Thu, 02 Jun 2005 19:30:54 -0700 Subject: [Web-SIG] HTMLTemplate Message-ID: <429FC0DE.7020501@skreak.com> [Oops, just realized I sent this directly to Shannon instead of the list. Sorry if you see it twice, Shannon.] HTMLTemplate is an interesting approach, although the API (or maybe the API documentation) seems a little convoluted. Another template system which takes a similar tack is PyMeld (http://www.entrian.com/PyMeld/). I am very impressed by its simplicity and pythonicity; it's definitely worth a look if the HTMLTemplate approach appeals to you. I agree with Mike Bayer's earlier comments, although in fairness all that logic would properly belong in the view layer, not the controller (or model), and it could be abstracted away from the controller easily enough. On a slightly related note, I am gearing up to do a comparison of Python templating systems, similar to the PyWebOff or Ian's web framework shootout; with any luck it should be ready to be announced by next week sometime. Tim Shannon -jj Behrens wrote: > Anyone have any comments on HTMLTemplate? From sridharinfinity at gmail.com Fri Jun 3 04:43:36 2005 From: sridharinfinity at gmail.com (Sridhar Ratna) Date: Fri, 3 Jun 2005 08:13:36 +0530 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> References: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> Message-ID: <8816fcf805060219431de2740@mail.gmail.com> On 6/3/05, mike bayer wrote: > > referring to this one: > > http://freespace.virgin.net/hamish.sanderson/htmltemplate.html > > it looks very cool and elegant, and clearly produces templates that are > super-clean. however, I would wonder how convenient it really is to I see that as reduced and simplified version of Nevow templates - http://www.nevow.com -- Sridhar Ratna - http://www.livejournal.com/users/seedar/ From pje at telecommunity.com Fri Jun 3 05:58:36 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 02 Jun 2005 23:58:36 -0400 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <8816fcf805060219431de2740@mail.gmail.com> References: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> Message-ID: <5.1.1.6.0.20050602235645.02201980@mail.telecommunity.com> At 08:13 AM 6/3/2005 +0530, Sridhar Ratna wrote: >On 6/3/05, mike bayer wrote: > > > > referring to this one: > > > > http://freespace.virgin.net/hamish.sanderson/htmltemplate.html > > > > it looks very cool and elegant, and clearly produces templates that are > > super-clean. however, I would wonder how convenient it really is to > >I see that as reduced and simplified version of Nevow templates - >http://www.nevow.com I'd say more a crippled relative than a simplified version. From aurora00 at gmail.com Fri Jun 3 06:51:48 2005 From: aurora00 at gmail.com (aurora) Date: Thu, 2 Jun 2005 21:51:48 -0700 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <429F8FD3.3040706@colorstudy.com> References: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> <1117753003.17379.235532492@webmail.messagingengine.com> <429F8FD3.3040706@colorstudy.com> Message-ID: On 6/2/05, Ian Bicking wrote: > If there's no code then it's not a presentation layer. It's an HTML > layer, nothing more, just dumb data. Presentation requires logic. > > From that perspective, I think doing proper model/presentation > separation using something like HTMLTemplate requires discipline and the > addition of another Python presentation layer. Precisely. I find this one of the biggest advantage of HTMLTemplate. You don't have to learn yet another half baked scripting language to build the presentation logic. You just use Python. So how to structure the model/presentation in HTMLTemplate's case? Here is what I have done: foo.cgi - application logic fooView.py - presentation logic fooView.html - template fooView would have the presentation logic. It is mainly methods like renderXXX(). Trust me it is really straightforward. In reality I find most of my model and view are one-one corresponding. So I even cut some corner and merged foo.cgi and fooView.py into one file ;-) Some of you find the callback API a little convoluted or clumsy. After using it for a while I think I have found the way to work efficiently. I'm going to get down on some detail here. Hopefully even people never seen HTMLTemplate can have a glimpse of how it works. First you parsed the template. The template object have a wonderful (diagnosis) method called structure(), which dump the view of the tree. It helps a lot in coding the presentation logic. For example, my template has: root con:title con:query con:error_msg con:num_match con:item_from con:item_to con:item_total rep:match_item con:address con:description con:archive_link con:host con:page_nav con:goto_prev rep:goto_page con:goto_next You'll need 3 render methods, one for the root and 2 for each rep node. render() render_match_item() render_goto_page() In each method, you are going to fill out the child items. For example, in the main render() method, you should probably fill title query error_msg num_match num_match.item_from num_match.item_to num_match.item_total page_nav? page_nav.goto_prev page_nav.goto_next Once I discovered the structure() method writing the presentation logic becomes natural. ---------------------------------------------------------------------- Let me cast another perspective to show why I like HTMLTemplate so much. Cheetah's user guide have about 100 pages. Enhancement and new features are probably on the way. HTMLTemplate's documentation consist of one short file & a few examples. That's probably all you'll ever need. It is finished. No significant enhancement is expected. I think that says a lot about its conceptual complexity and completeness. Wai Yip From dp at ulaluma.com Fri Jun 3 06:53:38 2005 From: dp at ulaluma.com (Donovan Preston) Date: Thu, 2 Jun 2005 21:53:38 -0700 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <5.1.1.6.0.20050602235645.02201980@mail.telecommunity.com> References: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> <5.1.1.6.0.20050602235645.02201980@mail.telecommunity.com> Message-ID: <7DF0F8F5-8C81-4159-8F88-7B80F909E575@ulaluma.com> On Jun 2, 2005, at 8:58 PM, Phillip J. Eby wrote: > At 08:13 AM 6/3/2005 +0530, Sridhar Ratna wrote: > >> On 6/3/05, mike bayer wrote: >> >>> >>> referring to this one: >>> >>> http://freespace.virgin.net/hamish.sanderson/htmltemplate.html >>> >>> it looks very cool and elegant, and clearly produces templates >>> that are >>> super-clean. however, I would wonder how convenient it really is to >>> >> >> I see that as reduced and simplified version of Nevow templates - >> http://www.nevow.com >> > > I'd say more a crippled relative than a simplified version. From looking through it briefly this was my impression. I think I asked HAS when he created it why he didn't use Nevow, and I think the answer was mostly because he just wanted to create static HTML files (as opposed to web applications). Being the author of Nevow I didn't want to dis HTMLTemplate, but Nevow really can do much, much more. Nevow's templates are, in my opinion, even cleaner. The original goal of both Woven and Nevow templates was to remove all view logic from the template and put logic where logic belongs -- as view logic in Python. This doesn't mean that your "controller" is rendering the views, but it does mean that rather than inventing a new mini- language, Python is used to manipulate an intermediate template format (a DOM). Woven used the W3C DOM, which turned out to be cumbersome and heavyweight, so Nevow uses a custom document object model called "stan". Stan is made up of Python lists, strings, and instances of nevow.stan.Tag (Actually, the stan dom can contain any Python object as long as a flattener function is registered for the type). The view logic consists of manipulating this DOM in pure python code. You do things that you would expect -- perform tests against certain conditions and return one or another node depending on the result, loop over things copying a node and applying one piece of data to that node (that node then gets rendered by trampolining it back up to the main render loop). In practice it works out very well, with ultra-clean templates marked up in strategic places and pure Python code which is very short and easy to read. To address concerns about how convenient it is in the long run, many of the common things one might do, like looping over a list of data and applying a node to each in turn or filling template slots with the values provided in a Python dictionary, are provided as default renderers out of the box. You don't have to write any code to get them. But they are provided as normal renderers, just like you would write. There is no template language which is fixed in stone and you're screwed if it doesn't do what you need it to do. If you find that things don't work quite the way you want them to, write your own renderer. It's easily extensible by design. One non-obvious thing about this approach is that it becomes falling- off-a-log easy to render recursive structures. The frustration that lead to the approach was the experience of trying to render a recursive tree in ZPT -- what was I going to do, repeat the skin for each level of possible depth? What about infinitely deep trees? I ended up rendering that portion of the template using string concatenation in Python, and at that point all the advantages of ZPT had been lost. Nevow avoids this problem by giving the Python code the ability to manipulate the template abstractly, allowing natural expression of recursion (it's expressed as tail-recursion -- a function returns an object which will invoke the same function, with a different node). So. That's what Nevow templates are about. Donovan From aurora00 at gmail.com Fri Jun 3 07:26:41 2005 From: aurora00 at gmail.com (aurora) Date: Thu, 2 Jun 2005 22:26:41 -0700 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <489AB64A-B328-4698-AF5D-B9864C633456@ksenia.nl> References: <489AB64A-B328-4698-AF5D-B9864C633456@ksenia.nl> Message-ID: On 6/2/05, Ksenia Marasanova wrote: > I used it quite a lot, and while it is fast, bugs free and simple to > use, total separation of markup and code is IMHO contraproductive - > every simple change mostly requires changing two files. > > --Ksenia Is changing two files too much trouble? Anytime we add an abstraction layer its a trade off. On one hand things are more structured and presumably more adaptable to change. On the other hand it adds complexity. Here on one end of the sprectrum we have highly structured framework like Java struts. Adding one field may need a change to a bean class, a form class, an action class, the template and perhaps a few XML files. On the other end of the spectrum we have PHP where you do everything in place. You just need to find you equilibrium on how much is too much. (by the way didn't I just heard somebody scream 'they put the whole $%^* application code in PHP templates!!!' ;-) Actually rather than editing two files, the biggest issue I encounter is the presentation code is supposed to match the DOM tree. So anytime you change the DOM you would have to change the structure of your code as well. Again you have to make the call what works best. For me the advantage of having a clean template is great while having to make coordinated code changes is just a matter of life. If it is not HTMLTemplate you'd probably have to do it for something else. Wai Yip From aurora00 at gmail.com Fri Jun 3 07:33:52 2005 From: aurora00 at gmail.com (aurora) Date: Thu, 2 Jun 2005 22:33:52 -0700 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <7DF0F8F5-8C81-4159-8F88-7B80F909E575@ulaluma.com> References: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> <5.1.1.6.0.20050602235645.02201980@mail.telecommunity.com> <7DF0F8F5-8C81-4159-8F88-7B80F909E575@ulaluma.com> Message-ID: On 6/2/05, Donovan Preston wrote: > So. That's what Nevow templates are about. > > Donovan I hope a few of us would at least agree on one thing: DOM based template rocks! Wai Yip From ianb at colorstudy.com Fri Jun 3 08:18:24 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 03 Jun 2005 01:18:24 -0500 Subject: [Web-SIG] DOM-based templating Message-ID: <429FF630.3020609@colorstudy.com> While we're on the topic of DOM-based templating... FormEncode has a module htmlfill (http://formencode.org/docs/htmlfill.html), which is basically like DOM-based templating that just knows about HTML forms. But currently it doesn't use a DOM, it uses an HTMLParser subclass. This makes it much more complex than it would otherwise be, and misses out on some potential performance gains -- many times the input to htmlfill will be output from a template or HTML generator, and so often the DOM from the template is serialized to text, then parsed again. I had thought about moving this to a DOM or DOM-ish thing of some sort, but I don't know which one. Unfortunately many of the options are not very humane -- that is, they are "correct", but not user-friendly. Here's what I'd like, and maybe someone can suggest something (I won't claim HTMLParser is that humane either; but I'm looking to improve this). Here's what I'd like: * Can parse HTML, not just XHTML. Not the crazy HTML browsers parse, but unambiguous well-formed HTML. I don't like the idea of putting the HTML through tidy; that's fine for a screen-scraper, but is way too defensive for this kind of thing. * Can generate HTML. This is probably easy to tack onto most systems, even if it isn't present now -- it's just a couple rules about how to serialize tags. * Doesn't modify the output at all for areas where no transformations occurred. It doesn't wipe out whitespace. It *definitely* doesn't lose comments. It keeps attribute order. When nodes are modified it's sometimes ambiguous how that effects the output, so if attribute order is lost there it's not that big a deal. * Can output nicely-formatted code. Probably easy to add, but nice if it's already there. This is, of course, entirely contrary to the previous item ;) When generating nodes *purely* from Python, systems tend to produce HTML/XML with no extra whitespace at all, and completely unreadable. * Keeps around enough information to produce good error messages. It needs to be possible to figure out the line and maybe column where a node was originally defined. If we're supporting multiple transformations by multiple systems, then this information needs to persist through the transformations. I think this is a really important and undervalued feature; anyone can write a templating system with crappy error messages (and lots and lots of people do). Good error messages set a templating system apart. * Reasonably fast. I've played around just a bit with ElementTree, but I only felt so-so about it. I felt like it was pretty correct, but not very humane -- maybe that'd be good enough if I was processing big XML documents, but it doesn't work for HTML templating. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From sridharinfinity at gmail.com Fri Jun 3 09:39:51 2005 From: sridharinfinity at gmail.com (Sridhar Ratna) Date: Fri, 3 Jun 2005 13:09:51 +0530 Subject: [Web-SIG] HTMLTemplate In-Reply-To: <7DF0F8F5-8C81-4159-8F88-7B80F909E575@ulaluma.com> References: <62739.66.192.34.8.1117752167.squirrel@66.192.34.8> <5.1.1.6.0.20050602235645.02201980@mail.telecommunity.com> <7DF0F8F5-8C81-4159-8F88-7B80F909E575@ulaluma.com> Message-ID: <8816fcf8050603003931c30b22@mail.gmail.com> > > So. That's what Nevow templates are about. > At this point, I would like point to the online nevow docs (which is not yet listed in nevow.com, for some unknown reason) Tutorial: http://divmod.org/users/mg/nevow-doc/ [available in rst format in the tarball] API doc: http://divmod.org/users/mg/nevow-api/ [generated from source using epydoc] -- Sridhar Ratna - http://www.livejournal.com/users/seedar/ From dp at ulaluma.com Fri Jun 3 09:45:13 2005 From: dp at ulaluma.com (Donovan Preston) Date: Fri, 3 Jun 2005 00:45:13 -0700 Subject: [Web-SIG] DOM-based templating In-Reply-To: <429FF630.3020609@colorstudy.com> References: <429FF630.3020609@colorstudy.com> Message-ID: <9A9B3BFA-E179-444A-A9AB-ABD25C000CD8@ulaluma.com> On Jun 2, 2005, at 11:18 PM, Ian Bicking wrote: > While we're on the topic of DOM-based templating... > > FormEncode has a module htmlfill > (http://formencode.org/docs/htmlfill.html), which is basically like > DOM-based templating that just knows about HTML forms. But > currently it > doesn't use a DOM, it uses an HTMLParser subclass. This makes it much > more complex than it would otherwise be, and misses out on some > potential performance gains -- many times the input to htmlfill > will be > output from a template or HTML generator, and so often the DOM from > the > template is serialized to text, then parsed again. > > I had thought about moving this to a DOM or DOM-ish thing of some > sort, > but I don't know which one. Unfortunately many of the options are not > very humane -- that is, they are "correct", but not user-friendly. > Here's what I'd like, and maybe someone can suggest something (I won't > claim HTMLParser is that humane either; but I'm looking to improve > this). Here's what I'd like: We've talked about this slightly before, but I think now more than ever stan can be that DOM. I don't think it would be too much work; it would mostly require removing assumptions that other nevow modules are available. I think stan could be broken out of nevow and into a standalone thing by pulling these modules: nevow.stan nevow.tags nevow.loaders nevow.context And the package: nevow.flat I'm willing to do the work, and I'm willing to remove assumptions it makes and refactor things until they are clean. The module which would require the most work is nevow.context -- an internal rendering implementation detail that Nevow makes explicit but I would want to hide from non-Nevow users of stan. nevow.context was the first module of nevow to be written, and has a bunch of crufty bad decisions that haven't yet been refactored out of existence. But I'd like to do it, and this would give me an excuse to. > * Can parse HTML, not just XHTML. Not the crazy HTML browsers parse, > but unambiguous well-formed HTML. I don't like the idea of putting > the > HTML through tidy; that's fine for a screen-scraper, but is way too > defensive for this kind of thing. nevow.loaders.htmlfile does a good job of parsing normal html. nevow.loaders.xmlfile parses strict XHTML and allows more tag tricks, but I think casual users won't notice the difference, especially for the purpose you desire. > * Can generate HTML. This is probably easy to tack onto most systems, > even if it isn't present now -- it's just a couple rules about how to > serialize tags. HTML rather than XHTML? I'm curious what the motivation for this is, and if you know what the couple of rules would be. I think it wouldn't be too hard. Hmm, I guess the motivation for the previous point is the next point? > * Doesn't modify the output at all for areas where no transformations > occurred. It doesn't wipe out whitespace. It *definitely* doesn't > lose > comments. It keeps attribute order. When nodes are modified it's > sometimes ambiguous how that effects the output, so if attribute order > is lost there it's not that big a deal. stan is whitespace in, whitespace out. It keeps comments. It uses a dict for attributes, but this could be changed easily. nevow.url uses a list of tuples, because order is actually important there. This means it needs to have a different API; it has .add() as well as .replace(). Add adds a new key value pair, even if the key is already present; replace finds any existing keys and puts a new value in it's place, preserving the original order. > * Can output nicely-formatted code. Probably easy to add, but nice if > it's already there. This is, of course, entirely contrary to the > previous item ;) When generating nodes *purely* from Python, systems > tend to produce HTML/XML with no extra whitespace at all, and > completely > unreadable. This is really, really, really a bad idea. While browsers claim to be whitespace agnostic, they make a huge rendering distinction between "no whitespace present" and "any whitespace present". Nevow preserves any whitespace that was originally in your template, but when generating tags from Python it can't, so it doesn't. That said, it is something I have considered writing before. Woven had it. I found it to be more trouble than it is worth. I think it should be added, but you should have to go out of your way to turn it on, and it should be off by default. > * Keeps around enough information to produce good error messages. It > needs to be possible to figure out the line and maybe column where a > node was originally defined. If we're supporting multiple > transformations by multiple systems, then this information needs to > persist through the transformations. I think this is a really > important > and undervalued feature; anyone can write a templating system with > crappy error messages (and lots and lots of people do). Good error > messages set a templating system apart. A great idea. It would be trivial to add file/line/column information and populate it differently in each of the loaders. I love it, I'm going to go do it right now. > * Reasonably fast. Nevow was designed as an optimization of woven, and as a result is pretty fast. It has a two-pass system where one pass is taken when the template is initially loaded (once per template per process, assuming the template doesn't change on disk) and non-important nodes are optimized out of what actually happens at render time. There's also a bunch of low hanging optimization work in nevow.context. When I originally wrote it, I was worried about people mutating things so I made lots of copies. In the meantime, it turns out that the "correct" style of using it is to not mutate things but be somewhat functional and side effect free. Since mutating is still nice for some things, the objects which get mutated get copied before you get called to mutate them. But, a lot, lot more copying currently happens than is necessary. Yet another thing I have been meaning to do but haven't gotten around to, that this might encourage me to do. > I've played around just a bit with ElementTree, but I only felt so-so > about it. I felt like it was pretty correct, but not very humane -- > maybe that'd be good enough if I was processing big XML documents, but > it doesn't work for HTML templating. Agreed. dp -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/web-sig/attachments/20050603/60bf4100/attachment.htm From ksenia at ksenia.nl Fri Jun 3 10:20:18 2005 From: ksenia at ksenia.nl (Ksenia Marasanova) Date: Fri, 3 Jun 2005 10:20:18 +0200 Subject: [Web-SIG] HTMLTemplate In-Reply-To: References: <489AB64A-B328-4698-AF5D-B9864C633456@ksenia.nl> Message-ID: > On 6/2/05, Ksenia Marasanova wrote: > > >> I used it quite a lot, and while it is fast, bugs free and simple to >> use, total separation of markup and code is IMHO contraproductive - >> every simple change mostly requires changing two files. >> >> --Ksenia >> > > Is changing two files too much trouble? Anytime we add an abstraction > layer its a trade off. On one hand things are more structured and > presumably more adaptable to change. On the other hand it adds > complexity. Here on one end of the sprectrum we have highly structured > framework like Java struts. Adding one field may need a change to a > bean class, a form class, an action class, the template and perhaps a > few XML files. On the other end of the spectrum we have PHP where you > do everything in place. You just need to find you equilibrium on how > much is too much. Yes, exactly. I liked the idea of DOM temlating a lot, but after using HTMLTemplate for a while felt the need for (IMHO) more freedom and moved toward Preppy/Cheetah approach. Other people move the opposite direction. > (by the way didn't I just heard somebody scream > 'they put the whole $%^* application code in PHP templates!!!' ;-) :)) That hurts... > > Actually rather than editing two files, the biggest issue I encounter > is the presentation code is supposed to match the DOM tree. So anytime > you change the DOM you would have to change the structure of your code > as well. Yes, me too, that was one of the reasons for editing multiple files. > Again you have to make the call what works best. For me the > advantage of having a clean template is great while having to make > coordinated code changes is just a matter of life. If it is not > HTMLTemplate you'd probably have to do it for something else. > > Wai Yip > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/ksenia% > 40ksenia.nl > > --Ksenia From ianb at colorstudy.com Fri Jun 3 17:33:39 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 03 Jun 2005 10:33:39 -0500 Subject: [Web-SIG] DOM-based templating In-Reply-To: <9A9B3BFA-E179-444A-A9AB-ABD25C000CD8@ulaluma.com> References: <429FF630.3020609@colorstudy.com> <9A9B3BFA-E179-444A-A9AB-ABD25C000CD8@ulaluma.com> Message-ID: <42A07853.8050209@colorstudy.com> Donovan Preston wrote: > We've talked about this slightly before, but I think now more than ever > stan can be that DOM. I don't think it would be too much work; it would > mostly require removing assumptions that other nevow modules are > available. I think stan could be broken out of nevow and into a > standalone thing by pulling these modules: > > nevow.stan > nevow.tags > nevow.loaders > nevow.context > > And the package: > > nevow.flat > > I'm willing to do the work, and I'm willing to remove assumptions it > makes and refactor things until they are clean. The module which would > require the most work is nevow.context -- an internal rendering > implementation detail that Nevow makes explicit but I would want to hide > from non-Nevow users of stan. nevow.context was the first module of > nevow to be written, and has a bunch of crufty bad decisions that > haven't yet been refactored out of existence. But I'd like to do it, and > this would give me an excuse to. I'd be very interested in that. The DOM is just annoying and HTMLParser and other SAX-ish parsers are just way too tedious and difficult. >> * Can parse HTML, not just XHTML. Not the crazy HTML browsers parse, >> but unambiguous well-formed HTML. I don't like the idea of putting the >> HTML through tidy; that's fine for a screen-scraper, but is way too >> defensive for this kind of thing. > > > nevow.loaders.htmlfile does a good job of parsing normal html. > nevow.loaders.xmlfile parses strict XHTML and allows more tag tricks, > but I think casual users won't notice the difference, especially for the > purpose you desire. > >> * Can generate HTML. This is probably easy to tack onto most systems, >> even if it isn't present now -- it's just a couple rules about how to >> serialize tags. > > > HTML rather than XHTML? I'm curious what the motivation for this is, and > if you know what the couple of rules would be. I think it wouldn't be > too hard. Well, there's a couple rules I guess. First, you don't use the XML-style empty tags, which I suspect will confuse browsers in some cases. Then there's a list of contentless tags in the HTML spec, and you just have rules about them specifically. I know Ryan Tamayko added this to Kid, which I believe uses ElementTree as its backend. He might be interested in Stan as well, since all these features would be useful to him as well. > Hmm, I guess the motivation for the previous point is the next point? > >> * Doesn't modify the output at all for areas where no transformations >> occurred. It doesn't wipe out whitespace. It *definitely* doesn't lose >> comments. It keeps attribute order. When nodes are modified it's >> sometimes ambiguous how that effects the output, so if attribute order >> is lost there it's not that big a deal. > > > stan is whitespace in, whitespace out. It keeps comments. It uses a dict > for attributes, but this could be changed easily. nevow.url uses a list > of tuples, because order is actually important there. This means it > needs to have a different API; it has .add() as well as .replace(). Add > adds a new key value pair, even if the key is already present; replace > finds any existing keys and puts a new value in it's place, preserving > the original order. Maybe just an ordered dictionary would work -- avoid any backward compatibility problems as well. For instance, here's an implementation: http://www.voidspace.org.uk/python/recipebook.shtml#odict and http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/107747 One thing this would allow -- while I understand why ZPT has a fixed order of attribute evaluation that ignores the textual order, I must admit I wish it was not so. Even now you probably can't add an option to warn if they show up out of order if it loses the order of the tags (though I have no idea what ZPT's internal representation is -- some ad hoc DOM I'd guess). I'm sure other systems have similar issues. >> * Can output nicely-formatted code. Probably easy to add, but nice if >> it's already there. This is, of course, entirely contrary to the >> previous item ;) When generating nodes *purely* from Python, systems >> tend to produce HTML/XML with no extra whitespace at all, and completely >> unreadable. > > > This is really, really, really a bad idea. While browsers claim to be > whitespace agnostic, they make a huge rendering distinction between "no > whitespace present" and "any whitespace present". Nevow preserves any > whitespace that was originally in your template, but when generating > tags from Python it can't, so it doesn't. > > That said, it is something I have considered writing before. Woven had > it. I found it to be more trouble than it is worth. I think it should be > added, but you should have to go out of your way to turn it on, and it > should be off by default. I think you can generally safely add whitespace to block-level tags. So if the indenter is HTML-aware it should be okay. I seem to remember problems with whitespace in elements (which I think count as block level, but maybe don't), but I don't think I've noticed that since the NS4 days. Another option is for some flavor of markup generation, where newlines would get appended automatically. Just the occassional newline would be helpful. OTOH, this might all be better resolved with a Firefox extension or bookmarklet or somesuch, that may or may not already exist. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From foom at fuhm.net Fri Jun 3 18:20:07 2005 From: foom at fuhm.net (James Y Knight) Date: Fri, 3 Jun 2005 12:20:07 -0400 Subject: [Web-SIG] DOM-based templating In-Reply-To: <429FF630.3020609@colorstudy.com> References: <429FF630.3020609@colorstudy.com> Message-ID: <9716AB4E-FE5A-4AB0-A21B-205F56A8964F@fuhm.net> On Jun 3, 2005, at 2:18 AM, Ian Bicking wrote: > * Can parse HTML, not just XHTML. Not the crazy HTML browsers parse, > but unambiguous well-formed HTML. I don't like the idea of putting > the > HTML through tidy; that's fine for a screen-scraper, but is way too > defensive for this kind of thing. Ha ha. As far as I've seen, there is no python module that can do this. And yes, by "this" I do mean actual correct HTML, not random crud real browsers have to put up with. Even unambiguous well-formed HTML is difficult to parse into a useful DOM. I'd suggest perl -- the perl HTML::TreeBuilder and HTML::Tagset modules seem to get it right, and can also parse lots of random crud. I've been trying for a while to convince someone to copy the algorithms from those perl modules into twisted.web.microdom and twisted.web.sux so that they would actually work right. It purports to parse HTML into a DOM, and currently does get a lot of stuff right, but also gets a bunch wrong. It doesn't depend on twisted very much, so it could make sense for someone to adopt as a separate package if you're into that. Getting whitespace rules right, in particular, is quite difficult. While the actual rules from the actual HTML spec are easy enough, they are not what you think they are, and not what anybody has implemented. The actual rules are much more complex, and depend on the element the whitespace is near and also seem to allow whitespace to float into and out of elements with relative abandon. Quick, tell me when the space in here is significant: " foo"? Just inserting all the whitespace from the original document into the DOM is a pretty safe thing to do, but it'd be nice to not have to do that, as you end up with excessive numbers of text nodes that have no meaning. Dealing with the optional closing of tags in HTML is somewhat irritating as well. Here's an example of a correct HTML document: "Hello

    Foo

    Bar
    ". Microdom gets the table there wrong -- it has a list of which opening tags can close which others, but only looks at the current level, not up the tree, to find the tag to close. So it creates a structure like: table[tr[td[p["Foo", tr[td["Bar"]]]]]]. Besides the obvious issue of the tr being inside the p, the DOM should probably include inferred elements as well, such as html, head, body, and tbody. On Jun 3, 2005, at 11:33 AM, Ian Bicking wrote: > OTOH, this might all be better resolved with a Firefox extension or > bookmarklet or somesuch, that may or may not already exist. http://users.skynet.be/mgueury/mozilla/ James From pje at telecommunity.com Fri Jun 3 18:40:23 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 03 Jun 2005 12:40:23 -0400 Subject: [Web-SIG] DOM-based templating In-Reply-To: <9716AB4E-FE5A-4AB0-A21B-205F56A8964F@fuhm.net> References: <429FF630.3020609@colorstudy.com> <429FF630.3020609@colorstudy.com> Message-ID: <5.1.1.6.0.20050603123414.022043c0@mail.telecommunity.com> At 12:20 PM 6/3/2005 -0400, James Y Knight wrote: >Just inserting all the whitespace from the original document into the >DOM is a pretty safe thing to do, but it'd be nice to not have to do >that, as you end up with excessive numbers of text nodes that have no >meaning. PWT (peak.web.templates) deals with this by concatenating all non-dynamic nodes into a single "literal" (i.e. unescaped) text node. PWT also isn't anything like a traditional DOM, in that it's a hierarchical structure, but that structure need not resemble the document structure in any way. So, if you guys manage to come up with a decent HTML parser, I'll be interested in gluing it to PWT, as long as it can deal with XML namespaces. PWT trees are currently constructed with expat, but the actual template components don't know anything about parsing and don't care. (Another difference between PWT and other DOM-based templating systems is that in PWT, a component is more like a function than a data structure. Instead of code receiving DOM data structures, they receive the moral equivalent of function parameters; so they can invoke them and pass parameters in, but they can't directly manipulate the DOM structure, because it's not a DOM structure. So, the Python code parts are largely immune to presentational concerns; they can't even *see* the content of the HTML they're manipulating.) From ianb at colorstudy.com Fri Jun 3 18:41:38 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 03 Jun 2005 11:41:38 -0500 Subject: [Web-SIG] DOM-based templating In-Reply-To: <9716AB4E-FE5A-4AB0-A21B-205F56A8964F@fuhm.net> References: <429FF630.3020609@colorstudy.com> <9716AB4E-FE5A-4AB0-A21B-205F56A8964F@fuhm.net> Message-ID: <42A08842.1000504@colorstudy.com> James Y Knight wrote: > On Jun 3, 2005, at 2:18 AM, Ian Bicking wrote: > >> * Can parse HTML, not just XHTML. Not the crazy HTML browsers parse, >> but unambiguous well-formed HTML. I don't like the idea of putting the >> HTML through tidy; that's fine for a screen-scraper, but is way too >> defensive for this kind of thing. > > > Ha ha. As far as I've seen, there is no python module that can do this. > And yes, by "this" I do mean actual correct HTML, not random crud real > browsers have to put up with. Even unambiguous well-formed HTML is > difficult to parse into a useful DOM. ZPT seems to do okay. It's picky about a few things, but when it's picky it also gives errors, so you can get by. For instance, I think this example you give: > Dealing with the optional closing of tags in HTML is somewhat > irritating as well. Here's an example of a correct HTML document: > "Hello

    Foo

    Bar
    ". > Microdom gets the table there wrong -- it has a list of which opening > tags can close which others, but only looks at the current level, not > up the tree, to find the tag to close. So it creates a structure like: > table[tr[td[p["Foo", tr[td["Bar"]]]]]]. Besides the obvious issue of > the tr being inside the p, the DOM should probably include inferred > elements as well, such as html, head, body, and tbody. That would cause an error in ZPT. Always a good principle -- when in doubt, don't guess; it's better to complain than to guess incorrectly. Well, that can be hard in some cases, like when you are parsing someone else's markup. But the idea here is that you are parsing markup that you also control. > Getting whitespace rules right, in particular, is quite difficult. > While the actual rules from the actual HTML spec are easy enough, they > are not what you think they are, and not what anybody has implemented. > The actual rules are much more complex, and depend on the element the > whitespace is near and also seem to allow whitespace to float into and > out of elements with relative abandon. Quick, tell me when the space in > here is significant: " foo"? Um lemme thing... x foo! I think I'll just retract the whitespace issue, though; it can be solved better at other levels. > Just inserting all the whitespace from the original document into the > DOM is a pretty safe thing to do, but it'd be nice to not have to do > that, as you end up with excessive numbers of text nodes that have no > meaning. Well, they have some meaning irregardless, because someone thought they made the document prettier, and it makes the resulting document look more like the original. That's important, even if the browser doesn't care. One issue might be that the DOM has a fairly heavy representation of text. Maybe this isn't an issue when text is just a Python string. ElementTree has a weird but seemingly light way of representing text. OTOH, I don't see any problem with just putting strings in the list of children. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From jjinux at gmail.com Fri Jun 3 19:31:25 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Fri, 3 Jun 2005 10:31:25 -0700 Subject: [Web-SIG] DOM-based templating In-Reply-To: <9A9B3BFA-E179-444A-A9AB-ABD25C000CD8@ulaluma.com> References: <429FF630.3020609@colorstudy.com> <9A9B3BFA-E179-444A-A9AB-ABD25C000CD8@ulaluma.com> Message-ID: If you made Stan separate, I'd be very inclined to add any explicity support for it necessary to Aquarium. I think templatings engines are a matter of opinion, and it'd be nice to share more code here. Thanks to Ksenia's hard work, Aquarium now supports WSGI. I see a brighter future ahead of us all :) Best Regards, -jj On 6/3/05, Donovan Preston wrote: > > > On Jun 2, 2005, at 11:18 PM, Ian Bicking wrote: > > While we're on the topic of DOM-based templating... > > FormEncode has a module htmlfill > (http://formencode.org/docs/htmlfill.html), which is > basically like > DOM-based templating that just knows about HTML forms. But currently it > doesn't use a DOM, it uses an HTMLParser subclass. This makes it much > more complex than it would otherwise be, and misses out on some > potential performance gains -- many times the input to htmlfill will be > output from a template or HTML generator, and so often the DOM from the > template is serialized to text, then parsed again. > > I had thought about moving this to a DOM or DOM-ish thing of some sort, > but I don't know which one. Unfortunately many of the options are not > very humane -- that is, they are "correct", but not user-friendly. > Here's what I'd like, and maybe someone can suggest something (I won't > claim HTMLParser is that humane either; but I'm looking to improve > this). Here's what I'd like: > > We've talked about this slightly before, but I think now more than ever stan > can be that DOM. I don't think it would be too much work; it would mostly > require removing assumptions that other nevow modules are available. I think > stan could be broken out of nevow and into a standalone thing by pulling > these modules: > > nevow.stan > nevow.tags > nevow.loaders > nevow.context > > And the package: > > nevow.flat > > I'm willing to do the work, and I'm willing to remove assumptions it makes > and refactor things until they are clean. The module which would require the > most work is nevow.context -- an internal rendering implementation detail > that Nevow makes explicit but I would want to hide from non-Nevow users of > stan. nevow.context was the first module of nevow to be written, and has a > bunch of crufty bad decisions that haven't yet been refactored out of > existence. But I'd like to do it, and this would give me an excuse to. > > * Can parse HTML, not just XHTML. Not the crazy HTML browsers parse, > but unambiguous well-formed HTML. I don't like the idea of putting the > HTML through tidy; that's fine for a screen-scraper, but is way too > defensive for this kind of thing. > > nevow.loaders.htmlfile does a good job of parsing normal html. > nevow.loaders.xmlfile parses strict XHTML and allows more tag tricks, but I > think casual users won't notice the difference, especially for the purpose > you desire. > > * Can generate HTML. This is probably easy to tack onto most systems, > even if it isn't present now -- it's just a couple rules about how to > serialize tags. > > HTML rather than XHTML? I'm curious what the motivation for this is, and if > you know what the couple of rules would be. I think it wouldn't be too hard. > > Hmm, I guess the motivation for the previous point is the next point? > > * Doesn't modify the output at all for areas where no transformations > occurred. It doesn't wipe out whitespace. It *definitely* doesn't lose > comments. It keeps attribute order. When nodes are modified it's > sometimes ambiguous how that effects the output, so if attribute order > is lost there it's not that big a deal. > > stan is whitespace in, whitespace out. It keeps comments. It uses a dict for > attributes, but this could be changed easily. nevow.url uses a list of > tuples, because order is actually important there. This means it needs to > have a different API; it has .add() as well as .replace(). Add adds a new > key value pair, even if the key is already present; replace finds any > existing keys and puts a new value in it's place, preserving the original > order. > > * Can output nicely-formatted code. Probably easy to add, but nice if > it's already there. This is, of course, entirely contrary to the > previous item ;) When generating nodes *purely* from Python, systems > tend to produce HTML/XML with no extra whitespace at all, and completely > unreadable. > > This is really, really, really a bad idea. While browsers claim to be > whitespace agnostic, they make a huge rendering distinction between "no > whitespace present" and "any whitespace present". Nevow preserves any > whitespace that was originally in your template, but when generating tags > from Python it can't, so it doesn't. > > That said, it is something I have considered writing before. Woven had it. I > found it to be more trouble than it is worth. I think it should be added, > but you should have to go out of your way to turn it on, and it should be > off by default. > > * Keeps around enough information to produce good error messages. It > needs to be possible to figure out the line and maybe column where a > node was originally defined. If we're supporting multiple > transformations by multiple systems, then this information needs to > persist through the transformations. I think this is a really important > and undervalued feature; anyone can write a templating system with > crappy error messages (and lots and lots of people do). Good error > messages set a templating system apart. > > A great idea. It would be trivial to add file/line/column information and > populate it differently in each of the loaders. I love it, I'm going to go > do it right now. > > * Reasonably fast. > > Nevow was designed as an optimization of woven, and as a result is pretty > fast. It has a two-pass system where one pass is taken when the template is > initially loaded (once per template per process, assuming the template > doesn't change on disk) and non-important nodes are optimized out of what > actually happens at render time. > > There's also a bunch of low hanging optimization work in nevow.context. When > I originally wrote it, I was worried about people mutating things so I made > lots of copies. In the meantime, it turns out that the "correct" style of > using it is to not mutate things but be somewhat functional and side effect > free. Since mutating is still nice for some things, the objects which get > mutated get copied before you get called to mutate them. But, a lot, lot > more copying currently happens than is necessary. > > Yet another thing I have been meaning to do but haven't gotten around to, > that this might encourage me to do. > > I've played around just a bit with ElementTree, but I only felt so-so > about it. I felt like it was pretty correct, but not very humane -- > maybe that'd be good enough if I was processing big XML documents, but > it doesn't work for HTML templating. > Agreed. > > dp > > > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: > http://mail.python.org/mailman/options/web-sig/jjinux%40gmail.com > > > -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through. From tsoehnli at gmu.edu Fri Jun 3 23:59:19 2005 From: tsoehnli at gmu.edu (Timothy Soehnlin) Date: Fri, 03 Jun 2005 16:59:19 -0500 Subject: [Web-SIG] DOM Implementation In-Reply-To: References: Message-ID: <200506031659.19441.tsoehnli@gmu.edu> What if instead of intersplicing node commands, such as iterating through lists, into the actual document, you were to operate on a different idea. What if you were to parse an xml file, and then manipulate the entire page as through a system of blocks. i.e. ----------------------
    %(heading)s
    %(day)s

    Updated, click here to return

    ------------------------ via the id tags, you could modify the output without ever knowing what it is. Assuming there is a wrapper around the xml file, to be indexed as a dictionary given 'id' tags, if you successfully deleted something, you would return as output, xmlObject['deleted'], and with string splicing, you could merge data, as with xmlObject['blogEntry'] % blogData, where blogData is a dictionary of data that could be generated by another xml file, or a database query. This would then would return a fully constructed html block, without the script ever knowing or touching any html. This orientation has some drawbacks, but allows for complete separation of code from content. This may happen to be too verbose for certain things, as it seems to add a lot of extra code for doing simple things, but with some simple coding, you could easily build a module that works around that. I do know that this works quite well for a system I have built, and keeps people who are editing the xml files from seeing any control structure, or any indication of python. Which means you could have people who know only html build the xml files, and then coders manipulate it, neither of the two would need to interact minus the definition of the variables for interpolation. Also, to hide python even more, the %( )s is normally replaced with [[ ]], as to make it seem more variable like, but that is superficial. -- I would rather be known as a Christian and despised, than to be overlooked, and thought of as one of the world. From jjinux at gmail.com Fri Jun 3 22:37:05 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Fri, 3 Jun 2005 13:37:05 -0700 Subject: [Web-SIG] DOM Implementation In-Reply-To: <200506031659.19441.tsoehnli@gmu.edu> References: <200506031659.19441.tsoehnli@gmu.edu> Message-ID: Yeah, this is more along the idea that Paul Abrams had, as I mentioned in my blog, leading me to bring up this subject. In his idea, he had a piece of Python splicing together pieces of XML, and treating the DOM somewhat like a dictionary where the ids where the keys. Best Regards, -jj On 6/3/05, Timothy Soehnlin wrote: > What if instead of intersplicing node commands, such as iterating through lists, into > the actual document, you were to operate on a different idea. > > What if you were to parse an xml file, and then manipulate the entire > page as through a system of blocks. i.e. > > ---------------------- > > >
    >
    %(heading)s
    >
    %(day)s
    >
    >
    > > >

    Updated, click here to return

    >
    >
    > ------------------------ > via the id tags, you could modify the output without ever knowing what it is. > > Assuming there is a wrapper around the xml file, to be indexed as a dictionary given 'id' tags, > if you successfully deleted something, you would return as output, xmlObject['deleted'], and > with string splicing, you could merge data, as with xmlObject['blogEntry'] % blogData, where > blogData is a dictionary of data that could be generated by another xml file, or a database query. > This would then would return a fully constructed html block, without the script ever knowing or > touching any html. > > This orientation has some drawbacks, but allows for complete separation of code from > content. > > This may happen to be too verbose for certain things, as it seems to add a lot of extra code > for doing simple things, but with some simple coding, you could easily build a module that works > around that. > > I do know that this works quite well for a system I have built, and keeps people who are editing > the xml files from seeing any control structure, or any indication of python. Which means you > could have people who know only html build the xml files, and then coders manipulate it, neither > of the two would need to interact minus the definition of the variables for interpolation. Also, to > hide python even more, the %( )s is normally replaced with [[ ]], as to make it seem more > variable like, but that is superficial. > -- > I would rather be known as a Christian > and despised, than to be overlooked, > and thought of as one of the world. > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/jjinux%40gmail.com > -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through. From pje at telecommunity.com Fri Jun 3 23:22:31 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 03 Jun 2005 17:22:31 -0400 Subject: [Web-SIG] DOM Implementation In-Reply-To: <200506031659.19441.tsoehnli@gmu.edu> References: Message-ID: <5.1.1.6.0.20050603171532.02202260@mail.telecommunity.com> At 04:59 PM 6/3/2005 -0500, Timothy Soehnlin wrote: >What if instead of intersplicing node commands, such as iterating through >lists, into > the actual document, you were to operate on a different idea. > >What if you were to parse an xml file, and then manipulate the entire > page as through a system of blocks. i.e. > >---------------------- > > >
    >
    %(heading)s
    >
    %(day)s
    >
    >
    > > >

    Updated, click here to return

    >
    >
    >------------------------ >via the id tags, you could modify the output without ever knowing what it is. PyMeld does something more or less like that. However, it still produces an excess of coupling between code and presentation at a global scale in the document - while keeping all the drawbacks of a static document model. Conceptually, this approach is like calling a function with a large number of arguments, and the number of arguments just gets bigger and bigger, and the function becomes more complex. Approaches in the woven/PWT family let a document make several function calls, each with its own set of unique local parameters, so that you don't end up with one huge super-coupled function. (PWT further refines this approach by making the sub-templates be functions themselves, as well as arguments, so that functions can be passed to functions, making the system extremely flexible while still barring any actual computation from occurring in the XML, or any XML -- including DOM manipulation -- from being needed in the Python parts.) From iv at lantic.net Wed Jun 8 14:59:17 2005 From: iv at lantic.net (Iwan Vosloo) Date: Wed, 08 Jun 2005 14:59:17 +0200 Subject: [Web-SIG] PEP222 and python on the server? Message-ID: <87psuxngwq.fsf@lantic.net> Hello, I see that PEP222 has status of deferred. Is anyone still interested in getting this done, or working on it? The naive summary of PEP222 is that it adds standardised Request and Response objects to the standard Python library. I think this is an extremely dire need since every web framework in Python is currently rolling their own (I'm doing a study of 40 such beasts in Python and another 40 in Java). The really bad news IMO regarding this lack is not the fact that each Python framework rolls their own, but that, if they want to integrate with one another, they have to provide kludging mappings between their respective project-specific Request and Response types. -i From ianb at colorstudy.com Wed Jun 8 17:26:47 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 08 Jun 2005 10:26:47 -0500 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <87psuxngwq.fsf@lantic.net> References: <87psuxngwq.fsf@lantic.net> Message-ID: <42A70E37.5000900@colorstudy.com> Iwan Vosloo wrote: > I see that PEP222 has status of deferred. Is anyone still interested > in getting this done, or working on it? > > The naive summary of PEP222 is that it adds standardised Request and > Response objects to the standard Python library. > > I think this is an extremely dire need since every web framework in > Python is currently rolling their own (I'm doing a study of 40 such > beasts in Python and another 40 in Java). The really bad news IMO > regarding this lack is not the fact that each Python framework rolls > their own, but that, if they want to integrate with one another, they > have to provide kludging mappings between their respective > project-specific Request and Response types. I think this is made obsolete by WSGI -- even though it doesn't call them request and response, WSGI does create those objects. The request is a dictionary, well specified in the spec, and the response is a protocol (start_response(status, headers) -> return body). I believe as long as the WSGI dictionary can be extracted from framework request objects, and that creation of framework request objects is deferred as long as possible during the processing of a request, that the WSGI dictionary is a reasonable standard object. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Jun 9 20:26:17 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 09 Jun 2005 13:26:17 -0500 Subject: [Web-SIG] Daemon server management Message-ID: <42A889C9.7080900@colorstudy.com> I asked this on the Paste list, but no opinions there... maybe someone here has a thought on this... Does anyone have opinions on how to start and stop daemon servers? I've added a --daemon option to paster serve, but I'd like to implement stop, restart, and reload as well. Whenever I encounter servers that clobber pid files, or where the only way you can tell you've started a server twice is that you get an error message about not being able to bind to the port, it annoys me. But I'm not sure how to best implement a better system. Especially cross-platform -- though an entirely separate process for Windows might make sense (as a windows service or something). Opinions? Or examples of other servers (preferably Python-based) that do this well? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From smulloni at smullyan.org Thu Jun 9 20:41:21 2005 From: smulloni at smullyan.org (Jacob Smullyan) Date: Thu, 9 Jun 2005 14:41:21 -0400 Subject: [Web-SIG] Daemon server management In-Reply-To: <42A889C9.7080900@colorstudy.com> References: <42A889C9.7080900@colorstudy.com> Message-ID: <20050609184121.GA13880@smullyan.org> On Thu, Jun 09, 2005 at 01:26:17PM -0500, Ian Bicking wrote: > Does anyone have opinions on how to start and stop daemon servers? I've > added a --daemon option to paster serve, but I'd like to implement stop, > restart, and reload as well. Whenever I encounter servers that clobber > pid files, or where the only way you can tell you've started a server > twice is that you get an error message about not being able to bind to > the port, it annoys me. But I'm not sure how to best implement a better > system. Especially cross-platform -- though an entirely separate > process for Windows might make sense (as a windows service or something). > > Opinions? Or examples of other servers (preferably Python-based) that > do this well? Clobbering pid files is a no-no; but getting an error about a port being already in use doesn't seem terrible to me. What would you rather it say when two different server installations on the same system try to open the same port? Should the server configuration really need to know that it is or is not the "same instance" of the server as the one owning the port? A program being stupid in the right way is a feature. In fact, a user getting annoyed by an error message in the right way is a feature, too, especially when the user has made a mistake! I'd advocate the standard UNIX behavior for UNIX machines; pid file, conventional signal handling (in particular, HUP reloads). For Windows, the standard Windows behavior, whatever that might be; a cross-platform solution would be neither fish nor fowl. This is not just a matter of taste; conforming to the platform's expectations in this area is the gracious thing to do, since packagers and system administrators do not relish constantly having to write special wrappers for non-standard daemons. My two cents, anyway. Cheers, j -- Jacob Smullyan -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/web-sig/attachments/20050609/a9012cbc/attachment.pgp From ianb at colorstudy.com Thu Jun 9 20:41:52 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 09 Jun 2005 13:41:52 -0500 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <87hdg7ockz.fsf@lantic.net> References: <87psuxngwq.fsf@lantic.net> <42A70E37.5000900@colorstudy.com> <87hdg7ockz.fsf@lantic.net> Message-ID: <42A88D70.5020708@colorstudy.com> Hope you don't mind, I copied this bad to web-sig... Iwan Vosloo wrote: > I suppose you may be right. Somehow we have gotten used to having "a > request object" and "a response object". And perhaps, in an > absent-minded way we expect this sort of thing in another form than > provided by the WSGI? For example, I would prefer to query an object > for something like req.headers['Content-Type'] (an expression close > to the HTTP spec), and not work with a dictionary that uses a naming > convention to do it (which feels one level removed from HTTP). > Another: I'd prefer to have a response object I can pass around and > add headers and whatever else to it - assuming that something will, > when its time to send the thing, send everything in the right order, > etc... > > Force of habit? Well, no one is going to argue that WSGI give pretty objects to work with. WSGI is very thorough and non-limiting. It also avoids any discussion of mixed case vs. underscores, and other details which are hard to decide on, while also being quite unproductive. I think of it as being aesthetically neutral. There's some other useful aspects of it as well. *But*, you shouldn't program directly to it, unless you are trying to be framework neutral (or maybe really low level), and even then it isn't always necessary. So it's expected there should be wrappers; especially request wrappers, but response wrappers are also fine (but they do interupt the call sequence a little in comparison to request wrappers). > Regarding the WSGI - are there plans to have an implementation of it > in the standard library? No, there aren't any plans. I wasn't really sure what would go in the standard library, though now I'm thinking it would be really good to get some stuff in there. In particular, I'd like to see SimpleHTTPServer extended to do WSGI, and a WSGI server added to cgi. I don't know how practical it is to add new servers to the standard library, but the ones that are already there could definitely support it. On the application side the cgi library mostly is sufficient, though it could probably use some relevant examples and documentation -- right now you have to read the source of that module to get the full FieldStorage signature. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Jun 9 20:52:52 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 09 Jun 2005 13:52:52 -0500 Subject: [Web-SIG] Daemon server management In-Reply-To: <20050609184121.GA13880@smullyan.org> References: <42A889C9.7080900@colorstudy.com> <20050609184121.GA13880@smullyan.org> Message-ID: <42A89004.4070409@colorstudy.com> Jacob Smullyan wrote: > On Thu, Jun 09, 2005 at 01:26:17PM -0500, Ian Bicking wrote: > >>Does anyone have opinions on how to start and stop daemon servers? I've >>added a --daemon option to paster serve, but I'd like to implement stop, >>restart, and reload as well. Whenever I encounter servers that clobber >>pid files, or where the only way you can tell you've started a server >>twice is that you get an error message about not being able to bind to >>the port, it annoys me. But I'm not sure how to best implement a better >>system. Especially cross-platform -- though an entirely separate >>process for Windows might make sense (as a windows service or something). >> >>Opinions? Or examples of other servers (preferably Python-based) that >>do this well? > > > Clobbering pid files is a no-no; but getting an error about a port > being already in use doesn't seem terrible to me. Yes, but how to avoid clobbering pid files? It's probably a beginner question, and I've found workable things in the os module, but I don't actually know the right way to do this. > What would you > rather it say when two different server installations on the same > system try to open the same port? Should the server configuration > really need to know that it is or is not the "same instance" of the > server as the one owning the port? A program being stupid in the > right way is a feature. In fact, a user getting annoyed by an error > message in the right way is a feature, too, especially when the user > has made a mistake! I'd agree it's wrong to be clever and notice that the process is already running, then exiting without error. But it's right to notice the other process is running, and exit with a helpful error; helpful errors are always right. Should I even try to connect to a port if the process in the pid file is still alive, or should I bail immediately? > I'd advocate the standard UNIX behavior for UNIX machines; pid file, > conventional signal handling (in particular, HUP reloads). For > Windows, the standard Windows behavior, whatever that might be; a > cross-platform solution would be neither fish nor fowl. This is not > just a matter of taste; conforming to the platform's expectations in > this area is the gracious thing to do, since packagers and system > administrators do not relish constantly having to write special > wrappers for non-standard daemons. I'm happy to copy conventions. Does anyone recommend a particular document on those conventions? For things like, do I open log files before or after I change user id (assuming the server is started as root)? And I'm a complete blank slate when it comes to the Windows side. Or even Macs, though I'm okay treating them like Unix to start. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From bwinton at latte.ca Thu Jun 9 21:03:15 2005 From: bwinton at latte.ca (Blake Winton) Date: Thu, 9 Jun 2005 15:03:15 -0400 Subject: [Web-SIG] Daemon server management In-Reply-To: <42A89004.4070409@colorstudy.com> Message-ID: <20050609185018.DD9964A@short.latte.ca> > And I'm a complete blank slate when it comes to the Windows side. On the Windows side, I think the relevant analog would be a service. (Okay, on the NT side, anyways. I'm not sure how many people would want to try to run a daemon on Windows 95/98... ;) Theres a short bit on how to check for various service-related info-bits here: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/59872 And to create a windows service, check out the sample at: http://cvs.sourceforge.net/viewcvs.py/pywin32/pywin32/win32/Demos/service/ Later, Blake. From gtalvola at nameconnector.com Thu Jun 9 21:15:22 2005 From: gtalvola at nameconnector.com (Geoffrey Talvola) Date: Thu, 9 Jun 2005 15:15:22 -0400 Subject: [Web-SIG] Daemon server management Message-ID: <61957B071FF421419E567A28A45C7FE5029D27A5@mailbox.nameconnector.com> Blake Winton wrote: >> And I'm a complete blank slate when it comes to the Windows side. > > On the Windows side, I think the relevant analog would be a service. > (Okay, on the NT side, anyways. I'm not sure how many people would > want to try to run a daemon on Windows 95/98... ;) Windows Services are definitely the way to go. One gotcha with Windows Services is that you MUST not write to stdout/stderr or your service will crash, because they don't exist. The crash won't happen right away, but as soon as you've written enough to fill up the buffer and cause it to attempt to flush. In practice this just means replacing sys.stdout and sys.stderr right at the start. Another gotcha is that I always seem to have to fiddle around with either the current directory and/or sys.path because they seem to get set up incorrectly when run as a service. You can look at the service-related code in Webware for more example code. - Geoff From jjinux at gmail.com Thu Jun 9 22:34:53 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Thu, 9 Jun 2005 13:34:53 -0700 Subject: [Web-SIG] Daemon server management In-Reply-To: <42A889C9.7080900@colorstudy.com> References: <42A889C9.7080900@colorstudy.com> Message-ID: I usually just write a wrapper that matches whatever OS I'm on. I.e. I sacrifice the cross-platform requirement. Creating a FreeBSD rc.sh-type script is quite simple. Perhaps I'm being naive. As for the pid file, if the user tries to start the server and I see that it exists, I exit with an error. If the user tries to stop a server, and there is no pid file, I exit with an error. I do it in shell. Best Regards, -jj On 6/9/05, Ian Bicking wrote: > I asked this on the Paste list, but no opinions there... maybe someone > here has a thought on this... > > Does anyone have opinions on how to start and stop daemon servers? I've > added a --daemon option to paster serve, but I'd like to implement stop, > restart, and reload as well. Whenever I encounter servers that clobber > pid files, or where the only way you can tell you've started a server > twice is that you get an error message about not being able to bind to > the port, it annoys me. But I'm not sure how to best implement a better > system. Especially cross-platform -- though an entirely separate > process for Windows might make sense (as a windows service or something). > > Opinions? Or examples of other servers (preferably Python-based) that > do this well? > > -- > Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/jjinux%40gmail.com > -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through. From mso at oz.net Thu Jun 9 23:12:25 2005 From: mso at oz.net (mso@oz.net) Date: Thu, 9 Jun 2005 14:12:25 -0700 (PDT) Subject: [Web-SIG] Daemon server management In-Reply-To: References: <42A889C9.7080900@colorstudy.com> Message-ID: <32982.161.55.66.227.1118351545.squirrel@www.oz.net> Shannon -jj Behrens said: > I usually just write a wrapper that matches whatever OS I'm on. I.e. > I sacrifice the cross-platform requirement. Creating a FreeBSD > rc.sh-type script is quite simple. Perhaps I'm being naive. I've got a similar situation now. I've been using start-stop-daemon on Linux to daemonize a naive Quixote application, but there's no equivalent wrapper for Mac OSX 10.3. We're looking at teaching the Python program itself to daemonize, although I think that's less ideal than using a platform-standard wrapper. There's a couple examples in the Python Cookbook (search for "daemon"); this is the one we're going to try: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278731 Essentially, your application calls a function and then continues, blissfully unaware it's been forked. However, it doesn't make a PID file or change uid. These are essential for production services. We're looking at extending the function to do these. The top-level executable needs --pidfile, --user and --group flags or the equivalent. (start-stop-daemon uses --chuid USER:GROUP). > As for the pid file, if the user tries to start the server and I see > that it exists, I exit with an error. If the user tries to stop a > server, and there is no pid file, I exit with an error. Yes, that's what most applications do. If the process doesn't exist it's OK to overwrite the file, but there's also an argument for forcing the user to do it manually. If the process does exist, we can't tell whether it's a conflicting Paste or an unrelated process, so we have to abort. -- -- Mike Orr From jjinux at gmail.com Thu Jun 9 23:24:35 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Thu, 9 Jun 2005 14:24:35 -0700 Subject: [Web-SIG] Daemon server management In-Reply-To: <32982.161.55.66.227.1118351545.squirrel@www.oz.net> References: <42A889C9.7080900@colorstudy.com> <32982.161.55.66.227.1118351545.squirrel@www.oz.net> Message-ID: On 6/9/05, mso at oz.net wrote: > Shannon -jj Behrens said: > > I usually just write a wrapper that matches whatever OS I'm on. I.e. > > I sacrifice the cross-platform requirement. Creating a FreeBSD > > rc.sh-type script is quite simple. Perhaps I'm being naive. > > I've got a similar situation now. I've been using start-stop-daemon on > Linux to daemonize a naive Quixote application, but there's no equivalent > wrapper for Mac OSX 10.3. We're looking at teaching the Python program > itself to daemonize, although I think that's less ideal than using a > platform-standard wrapper. There's a couple examples in the Python > Cookbook (search for "daemon"); this is the one we're going to try: > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278731 > > Essentially, your application calls a function and then continues, > blissfully unaware it's been forked. However, it doesn't make a PID file > or change uid. These are essential for production services. We're > looking at extending the function to do these. > > The top-level executable needs --pidfile, --user and --group flags or the > equivalent. (start-stop-daemon uses --chuid USER:GROUP). I would give up root manually. Afterall, no library code knows when the right time is, eh? > > As for the pid file, if the user tries to start the server and I see > > that it exists, I exit with an error. If the user tries to stop a > > server, and there is no pid file, I exit with an error. > > Yes, that's what most applications do. If the process doesn't exist it's > OK to overwrite the file, but there's also an argument for forcing the > user to do it manually. If the process does exist, we can't tell whether > it's a conflicting Paste or an unrelated process, so we have to abort. In the hope that it's useful, here's code to change your eid: ========================================================================== """This module contains the ``seteids`` function.""" __docformat__ = "restructuredtext" import os def seteids(user, group): """Change the ``euid`` and ``egid`` if run as root. user, group The user and group to change to. This function is self contained so that subclasses can easily drop this behavior. I won't, however, bother to catch exceptions because this is something you need to think about. """ import pwd import grp UID = GID = 2 if not os.geteuid(): os.setegid(grp.getgrnam(group)[GID]) # This must come first. os.seteuid(pwd.getpwnam(user)[UID]) ========================================================================== From paul at boddie.org.uk Thu Jun 9 23:36:59 2005 From: paul at boddie.org.uk (Paul Boddie) Date: Thu, 9 Jun 2005 23:36:59 +0200 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <87psuxngwq.fsf@lantic.net> References: <87psuxngwq.fsf@lantic.net> Message-ID: <200506092337.00038.paul@boddie.org.uk> On Wednesday 08 June 2005 14:59, Iwan Vosloo wrote: > Hello, > > I see that PEP222 has status of deferred. Is anyone still interested in > getting this done, or working on it? I doubt it (unfortunately). > The naive summary of PEP222 is that it adds standardised Request and > Response objects to the standard Python library. As has already been said, WSGI has been the only PEP-sanctioned standardisation effort to produce code so far - arguably at a different conceptual level - and I don't know where it stands with respect to the standard library. > I think this is an extremely dire need since every web framework in Python > is currently rolling their own (I'm doing a study of 40 such beasts in > Python and another 40 in Java). The really bad news IMO regarding this > lack is not the fact that each Python framework rolls their own, but that, > if they want to integrate with one another, they have to provide kludging > mappings between their respective project-specific Request and Response > types. If you take a look at WebStack (http://www.python.org/pypi/WebStack), you'll find a framework which provides its own "transaction" API and which works on top of other frameworks, precisely to avoid single project mappings such as those found in MoinMoin, for example. Given that you're studying so many "beasts", I'd appreciate anything you might have to say on the matter. Another framework which follows the standardisation path, albeit with a Java Servlet emphasis, is Snakelets (http://www.python.org/pypi/Snakelets). WebStack and Snakelets are similar in certain respects, making me tempted to consider a bit of integration of the two. Paul From mso at oz.net Fri Jun 10 00:00:36 2005 From: mso at oz.net (mso@oz.net) Date: Thu, 9 Jun 2005 15:00:36 -0700 (PDT) Subject: [Web-SIG] Daemon server management In-Reply-To: References: <42A889C9.7080900@colorstudy.com> <32982.161.55.66.227.1118351545.squirrel@www.oz.net> Message-ID: <32990.161.55.66.227.1118354436.squirrel@www.oz.net> Shannon -jj Behrens said: > On 6/9/05, mso at oz.net wrote: >> Shannon -jj Behrens said: >> > I usually just write a wrapper that matches whatever OS I'm on. I.e. >> > I sacrifice the cross-platform requirement. Creating a FreeBSD >> > rc.sh-type script is quite simple. Perhaps I'm being naive. >> >> I've got a similar situation now. I've been using start-stop-daemon on >> Linux to daemonize a naive Quixote application, but there's no >> equivalent >> wrapper for Mac OSX 10.3. We're looking at teaching the Python program >> itself to daemonize, although I think that's less ideal than using a >> platform-standard wrapper. There's a couple examples in the Python >> Cookbook (search for "daemon"); this is the one we're going to try: >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278731 >> >> Essentially, your application calls a function and then continues, >> blissfully unaware it's been forked. However, it doesn't make a PID >> file >> or change uid. These are essential for production services. We're >> looking at extending the function to do these. >> >> The top-level executable needs --pidfile, --user and --group flags or >> the >> equivalent. (start-stop-daemon uses --chuid USER:GROUP). > > I would give up root manually. Afterall, no library code knows when > the right time is, eh? Perhaps it should be a separate function then. I'm thinking mostly of code reuse. It would be nice to have a common module Paste and Quixote and others can use to handle daemonizing, one that works with Unix and Mac and Windows. daemon.daemonize(pidfile=None, overwrite=False) daemon.chuid(user=None, group=None) But who does the stop operation? I guess the executable needs a --stop flag, which would imply: # === daemon.py === daemon.start(pidfile=None, overwrite=False) @param pidfile filename or None. If none, no PID file. @param overwrite boolean. If false and PID file exists, abort. If true and PID file exists but process doesn't, overwrite. PURPOSE: a reference implementation for daemozing and PID creation. daemon.chuid(user=None, group=None) @param user name or uid. None means don't change. @param group name or gid. None means don't change. PURPOSE: do the name-to-id lookup and change in one step. daemon.stop(pidfile) @param pidfile filename. As for the RC script itself, those are very OS dependent. For instance, Gentoo, Red Hat, and Debian are all different. The best you can do is a directory of sample RC scripts. But even the "right" sample can be wrong if the target OS has recently changed a feature. The best advice is to copy a currently-installed script on the target platform. -- -- Mike Orr From mso at oz.net Fri Jun 10 00:34:56 2005 From: mso at oz.net (mso@oz.net) Date: Thu, 9 Jun 2005 15:34:56 -0700 (PDT) Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <200506092337.00038.paul@boddie.org.uk> References: <87psuxngwq.fsf@lantic.net> <200506092337.00038.paul@boddie.org.uk> Message-ID: <32995.161.55.66.227.1118356496.squirrel@www.oz.net> WSGI is a protocol, not a specific function or class, so you can't put it "in" the standard library. But you can recommend it in the documentation. And a wsgitools module a la itertools would certainly be useful. Having used WSGI a bit with Paste and Quxiote, I'd say it's a better common denominator than Request/Response objects. We'll never agree on the best Request/Response format, so people will be packing/unpacking it to their preferred structure anyway -- exactly as they do with WSGI. Classes imply overhead, and you can't assume they'll be installed on Python < 2.5. WSGI uses only Python primitives so it can be used anywhere. Some of the more coloful ideological battles regarding Request/Response: - Should the Response be inside the Request or separate? - Is .field('foo') an abomination or is .fields['foo'] more satanic? - Return the body or call a .write() method or...? If certain actions are inconvenient in WSGI -- like modifying a response header -- we can provide wrapper functions for these. The biggest part in this is identifying the actions. The only limitation in WSGI I've found is you can't serialize it for IPC since it contains an open file descriptor. But others have argued SCGI is more appropriate for this anyway. -- -- Mike Orr From pje at telecommunity.com Fri Jun 10 01:44:53 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Jun 2005 19:44:53 -0400 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <32995.161.55.66.227.1118356496.squirrel@www.oz.net> References: <200506092337.00038.paul@boddie.org.uk> <87psuxngwq.fsf@lantic.net> <200506092337.00038.paul@boddie.org.uk> Message-ID: <5.1.1.6.0.20050609194115.01fb5a90@mail.telecommunity.com> At 03:34 PM 6/9/2005 -0700, mso at oz.net wrote: >The only limitation in WSGI I've found is you can't serialize it for IPC >since it contains an open file descriptor. That's quite a generalization there. First, many WSGI implementations don't have a file descriptor involved. Second, there are lots of ways to serialize things, including e.g. transmitting file descriptors via Unix-domain sockets. Third, nothing stops you from reading the data and sending *that* via your IPC mechanism. Fourth, you can always do IPC via HTTP. :) But if what you meant was that pickle, marshal, and XML-RPC won't handle some kinds of file-like objects very well, then, yes, and disregard the previous paragraph. :) From renesd at gmail.com Fri Jun 10 03:01:46 2005 From: renesd at gmail.com (Rene Dudfield) Date: Fri, 10 Jun 2005 11:01:46 +1000 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <5.1.1.6.0.20050609194115.01fb5a90@mail.telecommunity.com> References: <87psuxngwq.fsf@lantic.net> <200506092337.00038.paul@boddie.org.uk> <32995.161.55.66.227.1118356496.squirrel@www.oz.net> <5.1.1.6.0.20050609194115.01fb5a90@mail.telecommunity.com> Message-ID: <64ddb72c05060918015697ffab@mail.gmail.com> Is not possible to use something like sendfile(2) with wsgi? Where you need a socket file descriptor to use it. If not, then you can't send files very efficiently. On 6/10/05, Phillip J. Eby wrote: > At 03:34 PM 6/9/2005 -0700, mso at oz.net wrote: > >The only limitation in WSGI I've found is you can't serialize it for IPC > >since it contains an open file descriptor. > > That's quite a generalization there. First, many WSGI implementations > don't have a file descriptor involved. Second, there are lots of ways to > serialize things, including e.g. transmitting file descriptors via > Unix-domain sockets. Third, nothing stops you from reading the data and > sending *that* via your IPC mechanism. Fourth, you can always do IPC via > HTTP. :) > > But if what you meant was that pickle, marshal, and XML-RPC won't handle > some kinds of file-like objects very well, then, yes, and disregard the > previous paragraph. :) > From jjinux at gmail.com Fri Jun 10 04:05:24 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Thu, 9 Jun 2005 19:05:24 -0700 Subject: [Web-SIG] (sendfile was: PEP222 and python on the server?) In-Reply-To: <64ddb72c05060918015697ffab@mail.gmail.com> References: <87psuxngwq.fsf@lantic.net> <200506092337.00038.paul@boddie.org.uk> <32995.161.55.66.227.1118356496.squirrel@www.oz.net> <5.1.1.6.0.20050609194115.01fb5a90@mail.telecommunity.com> <64ddb72c05060918015697ffab@mail.gmail.com> Message-ID: (Off topic:) It's too bad sendfile isn't very consistent among the platforms, and it's also too bad Python doesn't come with a sendfile module builtin :-/ -jj On 6/9/05, Rene Dudfield wrote: > Is not possible to use something like sendfile(2) with wsgi? > > Where you need a socket file descriptor to use it. > > If not, then you can't send files very efficiently. > > > On 6/10/05, Phillip J. Eby wrote: > > At 03:34 PM 6/9/2005 -0700, mso at oz.net wrote: > > >The only limitation in WSGI I've found is you can't serialize it for IPC > > >since it contains an open file descriptor. > > > > That's quite a generalization there. First, many WSGI implementations > > don't have a file descriptor involved. Second, there are lots of ways to > > serialize things, including e.g. transmitting file descriptors via > > Unix-domain sockets. Third, nothing stops you from reading the data and > > sending *that* via your IPC mechanism. Fourth, you can always do IPC via > > HTTP. :) > > > > But if what you meant was that pickle, marshal, and XML-RPC won't handle > > some kinds of file-like objects very well, then, yes, and disregard the > > previous paragraph. :) > > > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/jjinux%40gmail.com > -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through. From pje at telecommunity.com Fri Jun 10 05:54:47 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Jun 2005 23:54:47 -0400 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <64ddb72c05060918015697ffab@mail.gmail.com> References: <5.1.1.6.0.20050609194115.01fb5a90@mail.telecommunity.com> <87psuxngwq.fsf@lantic.net> <200506092337.00038.paul@boddie.org.uk> <32995.161.55.66.227.1118356496.squirrel@www.oz.net> <5.1.1.6.0.20050609194115.01fb5a90@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050609235313.01fa48b8@mail.telecommunity.com> At 11:01 AM 6/10/2005 +1000, Rene Dudfield wrote: >Is not possible to use something like sendfile(2) with wsgi? See the specification for details of how this is accomplished, and how an application can take advantage of it, if the server provides the feature: http://www.python.org/peps/pep-0333.html#optional-platform-specific-file-handling From mso at oz.net Fri Jun 10 06:46:10 2005 From: mso at oz.net (Mike Orr) Date: Thu, 09 Jun 2005 21:46:10 -0700 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <64ddb72c05060918015697ffab@mail.gmail.com> References: <87psuxngwq.fsf@lantic.net> <200506092337.00038.paul@boddie.org.uk> <32995.161.55.66.227.1118356496.squirrel@www.oz.net> <5.1.1.6.0.20050609194115.01fb5a90@mail.telecommunity.com> <64ddb72c05060918015697ffab@mail.gmail.com> Message-ID: <42A91B12.2010303@oz.net> I may have read the PEP wrong. I thought it said 'wsgi.input' had to be a file object. But it says it can be a file-like object. I guess a StringIO would suffice, and that would be serializable. Rene Dudfield wrote: >Is not possible to use something like sendfile(2) with wsgi? > >Where you need a socket file descriptor to use it. > >If not, then you can't send files very efficiently. > > >On 6/10/05, Phillip J. Eby wrote: > > >>At 03:34 PM 6/9/2005 -0700, mso at oz.net wrote: >> >> >>>The only limitation in WSGI I've found is you can't serialize it for IPC >>>since it contains an open file descriptor. >>> >>> >>That's quite a generalization there. First, many WSGI implementations >>don't have a file descriptor involved. Second, there are lots of ways to >>serialize things, including e.g. transmitting file descriptors via >>Unix-domain sockets. Third, nothing stops you from reading the data and >>sending *that* via your IPC mechanism. Fourth, you can always do IPC via >>HTTP. :) >> >>But if what you meant was that pickle, marshal, and XML-RPC won't handle >>some kinds of file-like objects very well, then, yes, and disregard the >>previous paragraph. :) >> >> >> >_______________________________________________ >Web-SIG mailing list >Web-SIG at python.org >Web SIG: http://www.python.org/sigs/web-sig >Unsubscribe: http://mail.python.org/mailman/options/web-sig/mso%40oz.net > > From davidf at sjsoft.com Fri Jun 10 07:20:33 2005 From: davidf at sjsoft.com (David Fraser) Date: Fri, 10 Jun 2005 07:20:33 +0200 Subject: [Web-SIG] Daemon server management In-Reply-To: <42A889C9.7080900@colorstudy.com> References: <42A889C9.7080900@colorstudy.com> Message-ID: <42A92321.2080600@sjsoft.com> Ian Bicking wrote: >I asked this on the Paste list, but no opinions there... maybe someone >here has a thought on this... > >Does anyone have opinions on how to start and stop daemon servers? I've >added a --daemon option to paster serve, but I'd like to implement stop, >restart, and reload as well. Whenever I encounter servers that clobber >pid files, or where the only way you can tell you've started a server >twice is that you get an error message about not being able to bind to >the port, it annoys me. But I'm not sure how to best implement a better >system. Especially cross-platform -- though an entirely separate >process for Windows might make sense (as a windows service or something). > >Opinions? Or examples of other servers (preferably Python-based) that >do this well? > > I think other people have expressed better ideas, but here are some of mine I recently implemented... This is from jToolkit (jtoolkit.sf.net but the site is horribly out of date) which I use for commercial work and for Pootle (pootle.wordforge.net) I wanted to make the server reload as quickly as possible with minimal handover time. So what I did is make a parent process which simply runs a child process that does all the actual work. When a reload is signalled, a new child is run. When it is ready to bind to the socket, a signal is sent to the first child, which finishes handling its current requests and then closes its listening socket. As soon as it closes the socket, it sends a signal back to the new child which binds to the socket. I think this could be improved by using the same bound socket and forking, but the idea of signalling and handover is quite nice. Not sure how to do it on Windows though... I also had a look into trying to pass sockets between processes which you're meant to be able to do with the win32 api but didn't quite get it working. As an aside, I have found problems with Python not releasing allocated memory which is why I need to restart every so often in the above manner. David From renesd at gmail.com Fri Jun 10 09:13:06 2005 From: renesd at gmail.com (Rene Dudfield) Date: Fri, 10 Jun 2005 17:13:06 +1000 Subject: [Web-SIG] Daemon server management In-Reply-To: <42A92321.2080600@sjsoft.com> References: <42A889C9.7080900@colorstudy.com> <42A92321.2080600@sjsoft.com> Message-ID: <64ddb72c0506100013c61f6fe@mail.gmail.com> This is a great idea! It gets around the problem of broken modules too. If a module is updated on the live site, and it doesn't work for whatever reason, then the old server is still around to serve requests. Keeping the old server around until it has finished its requests would be good. It would need to be told to stop handling new requests... Like stop binding. But then you still have a tiny window when they switch over that requests may get a connection refused message? Still 100% better than the kill/start method. I'm interested in handing sockets over to child proccesses(without a fork). Not quite sure what the best way is for that though. If you find out please let me know :) The similar idea that needs handing over sockets, is for sending files... you make a really tiny process that all it does is send a file to a socket. I've done something like this based on the httpd in asmutils, that only takes up 12KB of memory per process. It uses linux clone(2) and sendfile(2) instead of fork, and read/write. So it is very platform specific(x86/linux). It still has some bugs with really high loads 1000+ concurrent connections. I'm going to fix them before adapting it for my python server use. However it can do 1500+ req/s of 32KB files on linux/amd duron 850. It can also serve 10000+ connections at once without the machine dying. Here's the work in progress of the httpd part, if anyone is interested: http://www.madecollective.com/~rene/httpd_sendfile.asm On 6/10/05, David Fraser wrote: > Ian Bicking wrote: > > >I asked this on the Paste list, but no opinions there... maybe someone > >here has a thought on this... > > > >Does anyone have opinions on how to start and stop daemon servers? I've > >added a --daemon option to paster serve, but I'd like to implement stop, > >restart, and reload as well. Whenever I encounter servers that clobber > >pid files, or where the only way you can tell you've started a server > >twice is that you get an error message about not being able to bind to > >the port, it annoys me. But I'm not sure how to best implement a better > >system. Especially cross-platform -- though an entirely separate > >process for Windows might make sense (as a windows service or something). > > > >Opinions? Or examples of other servers (preferably Python-based) that > >do this well? > > > > > I think other people have expressed better ideas, but here are some of > mine I recently implemented... > > This is from jToolkit (jtoolkit.sf.net but the site is horribly out of > date) which I use for commercial work and for Pootle (pootle.wordforge.net) > I wanted to make the server reload as quickly as possible with minimal > handover time. So what I did is make a parent process which simply runs > a child process that does all the actual work. When a reload is > signalled, a new child is run. When it is ready to bind to the socket, a > signal is sent to the first child, which finishes handling its current > requests and then closes its listening socket. As soon as it closes the > socket, it sends a signal back to the new child which binds to the socket. > I think this could be improved by using the same bound socket and > forking, but the idea of signalling and handover is quite nice. Not sure > how to do it on Windows though... > I also had a look into trying to pass sockets between processes which > you're meant to be able to do with the win32 api but didn't quite get it > working. > > As an aside, I have found problems with Python not releasing allocated > memory which is why I need to restart every so often in the above manner. > > David > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/renesd%40gmail.com > From iv at lantic.net Fri Jun 10 10:13:22 2005 From: iv at lantic.net (Iwan Vosloo) Date: Fri, 10 Jun 2005 10:13:22 +0200 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <200506092337.00038.paul@boddie.org.uk> (Paul Boddie's message of "Thu, 9 Jun 2005 23:36:59 +0200") References: <87psuxngwq.fsf@lantic.net> <200506092337.00038.paul@boddie.org.uk> Message-ID: <87r7fa8w9p.fsf@lantic.net> Hi, Paul Boddie writes: > > If you take a look at WebStack (http://www.python.org/pypi/WebStack), you'll > find a framework which provides its own "transaction" API and which works on > top of other frameworks, precisely to avoid single project mappings such as > those found in MoinMoin, for example. Given that you're studying so many > "beasts", I'd appreciate anything you might have to say on the matter. I don't feel I quite understand these things well enough yet to make to many statements... But I'm planning to float my impressions here when I get to that stage. But, WRT WebStack - I had it loosely categorised as "server infrastructure" with things like twisted, Webware, or PEAK. But I see your point that it is trying to create a single interface to all of those. And that would indeed be useful. My focus is more on the different models used by higher level frameworks that sit on top of, for example, WebStack (think CherryPy, Spyce, Zope, JSF, Struts, Tapestry and so on). The common point of interest here though has to do with the fact that I feel it is better to put something out there that tries to solve a small piece of the problem only - and that provides easy ways to use a collection of other projects together to accomplish your total solution. You have to be able to mix and match. Webstack plays in this field. And if there is support in terms of common libraries that you can count on being part of the standard Python distribution so much the better. I am concerned with the quantity of concepts that each framework (especially large ones & especially the Java ones) invents. This makes it difficult to understand "what they really do" (if that phrase makes at all sense). Also, I find that the concepts they invent are rarely well researched - they are mostly ad-hoc inventions of the authors and never relate to the known body of computer science literature. I'm also struggling to find much in the academic world regarding these animals. Nice big gap, thus. > Another framework which follows the standardisation path, albeit with a Java > Servlet emphasis, is Snakelets (http://www.python.org/pypi/Snakelets). > WebStack and Snakelets are similar in certain respects, making me tempted to > consider a bit of integration of the two. I don't want to make provocative statements without being able to back them up, but I have a _feeling_ (which I am investigating more closely at the moment) that the whole Java servlet model is a bit passe. Contrast it with something like CherryPy, for example, and just see what your gut feels... Its like the servlet model was born back when people thought of web sites as "having dynamic content". Now they see them as "web applications" and have taken that idea to a whole new level (in the Java world, anyway, albeit in its cumbersome ways). -i From iv at lantic.net Fri Jun 10 10:32:24 2005 From: iv at lantic.net (Iwan Vosloo) Date: Fri, 10 Jun 2005 10:32:24 +0200 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <42A88D70.5020708@colorstudy.com> (Ian Bicking's message of "Thu, 09 Jun 2005 13:41:52 -0500") References: <87psuxngwq.fsf@lantic.net> <42A70E37.5000900@colorstudy.com> <87hdg7ockz.fsf@lantic.net> <42A88D70.5020708@colorstudy.com> Message-ID: <87mzpy8vdz.fsf@lantic.net> Ian Bicking writes: > Hope you don't mind, I copied this bad to web-sig... Yea, sorry, that was the intention... > Well, no one is going to argue that WSGI give pretty objects to work > with. WSGI is very thorough and non-limiting. It also avoids any > discussion of mixed case vs. underscores, and other details which are > hard to decide on, while also being quite unproductive. I think of it > as being aesthetically neutral. There's some other useful aspects of > it as well. True.. > *But*, you shouldn't program directly to it, unless you are trying to > be framework neutral (or maybe really low level), and even then it > isn't always necessary. So it's expected there should be wrappers; > especially request wrappers, but response wrappers are also fine (but > they do interupt the call sequence a little in comparison to request > wrappers). Mmm, this is a bit outside of my main focus, so I'd have to have a better look at it - I understand, and those wrappers can be very very thin also. The only point I would like to make is that it is very useful to glue several frameworks together to accomplish a task. And they often communicate via request and response instances. I'll have to do more homework and try to make a better case for it. >> Regarding the WSGI - are there plans to have an implementation of it >> in the standard library? > > No, there aren't any plans. I wasn't really sure what would go in the > standard library, though now I'm thinking it would be really good to > get some stuff in there. In particular, I'd like to see > SimpleHTTPServer extended to do WSGI, and a WSGI server added to cgi. > I don't know how practical it is to add new servers to the standard > library, but the ones that are already there could definitely support > it. On the application side the cgi library mostly is sufficient, > though it could probably use some relevant examples and documentation > -- right now you have to read the source of that module to get the > full FieldStorage signature. Definately a good idea to get some stuff in there, I think. Some people are publishing implementations of WSGI compliant code in single files on their websites, others grab these, modify them - and you end up getting several mutations of the same thing installed by different packaged products... And FieldStorage is also not the prettiest thing in the world :-)! (But I'm happy its there..) -i From speno at isc.upenn.edu Fri Jun 10 13:52:10 2005 From: speno at isc.upenn.edu (John Speno) Date: Fri, 10 Jun 2005 07:52:10 -0400 Subject: [Web-SIG] Daemon server management In-Reply-To: <64ddb72c0506100013c61f6fe@mail.gmail.com> References: <42A889C9.7080900@colorstudy.com> <42A92321.2080600@sjsoft.com> <64ddb72c0506100013c61f6fe@mail.gmail.com> Message-ID: <67F89ED4-D1FD-4EEB-9AE1-44A3B3FAC0BF@isc.upenn.edu> On Jun 10, 2005, at 3:13 AM, Rene Dudfield wrote: > I'm interested in handing sockets over to child proccesses(without a > fork). Not quite sure what the best way is for that though. If you > find out please let me know :) The python scgi module has this feature in its passfd module. It can be found at http://www.mems-exchange.org/software/scgi/ And it's easy to hook up scgi to wsgi these days. Here's a CherryPy example that uses the scgiserver module from Python Paste: http://www.cherrypy.org/wiki/ScgiWsgi Take care. From davidf at sjsoft.com Fri Jun 10 15:19:23 2005 From: davidf at sjsoft.com (David Fraser) Date: Fri, 10 Jun 2005 15:19:23 +0200 Subject: [Web-SIG] Daemon server management In-Reply-To: <67F89ED4-D1FD-4EEB-9AE1-44A3B3FAC0BF@isc.upenn.edu> References: <42A889C9.7080900@colorstudy.com> <42A92321.2080600@sjsoft.com> <64ddb72c0506100013c61f6fe@mail.gmail.com> <67F89ED4-D1FD-4EEB-9AE1-44A3B3FAC0BF@isc.upenn.edu> Message-ID: <42A9935B.1050208@sjsoft.com> John Speno wrote: >On Jun 10, 2005, at 3:13 AM, Rene Dudfield wrote: > > > >>I'm interested in handing sockets over to child proccesses(without a >>fork). Not quite sure what the best way is for that though. If you >>find out please let me know :) >> >> > >The python scgi module has this feature in its passfd module. > >It can be found at http://www.mems-exchange.org/software/scgi/ > > Right, although this is only useful inside Apache right? Also it uses C to do it I was wondering if it was feasible to do it directly in Python... but maybe not. Cheers David From speno at isc.upenn.edu Fri Jun 10 15:37:12 2005 From: speno at isc.upenn.edu (John Speno) Date: Fri, 10 Jun 2005 09:37:12 -0400 Subject: [Web-SIG] Daemon server management In-Reply-To: <42A9935B.1050208@sjsoft.com> References: <42A889C9.7080900@colorstudy.com> <42A92321.2080600@sjsoft.com> <64ddb72c0506100013c61f6fe@mail.gmail.com> <67F89ED4-D1FD-4EEB-9AE1-44A3B3FAC0BF@isc.upenn.edu> <42A9935B.1050208@sjsoft.com> Message-ID: On Jun 10, 2005, at 9:19 AM, David Fraser wrote: >> The python scgi module has this feature in its passfd module. >> >> It can be found at http://www.mems-exchange.org/software/scgi/ >> >> > Right, although this is only useful inside Apache right? Yes, the scgi package only has scgi modules for apache1 and apache2. It's a very simple protocol, so adding support for other servers isn't too hard. > Also it uses C to do it I was wondering if it was feasible to do it > directly in Python... but maybe not. Yes, it's a python module written in C, but I don't see why that would matter. I think you're probably already using a few other modules in the same category. :-) From ianb at colorstudy.com Fri Jun 10 17:26:26 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 10 Jun 2005 10:26:26 -0500 Subject: [Web-SIG] Daemon server management In-Reply-To: References: <42A889C9.7080900@colorstudy.com> <42A92321.2080600@sjsoft.com> <64ddb72c0506100013c61f6fe@mail.gmail.com> <67F89ED4-D1FD-4EEB-9AE1-44A3B3FAC0BF@isc.upenn.edu> <42A9935B.1050208@sjsoft.com> Message-ID: <42A9B122.3000508@colorstudy.com> John Speno wrote: > Yes, the scgi package only has scgi modules for apache1 and apache2. > It's a very simple protocol, so adding support for other servers > isn't too hard. There's also cgi2scgi, which is a very small CGI program that any web server could run, which connects to an SCGI server. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Fri Jun 10 17:36:02 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 10 Jun 2005 10:36:02 -0500 Subject: [Web-SIG] Daemon server management In-Reply-To: <64ddb72c0506100013c61f6fe@mail.gmail.com> References: <42A889C9.7080900@colorstudy.com> <42A92321.2080600@sjsoft.com> <64ddb72c0506100013c61f6fe@mail.gmail.com> Message-ID: <42A9B362.4010401@colorstudy.com> Rene Dudfield wrote: > This is a great idea! It gets around the problem of broken modules > too. If a module is updated on the live site, and it doesn't work for > whatever reason, then the old server is still around to serve > requests. I hadn't thought about this aspect. One problem with the reloader in Paste is that it can restart the server, and when there is a syntax error or something buggy in the startup code the whole server will die there. Of course, I guess I could keep trying to restart the server regardless of its exit code, and just restart and restart until the bug gets fixed... but it would be better to leave the old one around until the new one shows at least some indication that it's ready to serve requests. > Keeping the old server around until it has finished its requests would > be good. It would need to be told to stop handling new requests... > Like stop binding. But then you still have a tiny window when they > switch over that requests may get a connection refused message? Still > 100% better than the kill/start method. One way that mod_webkit handles this (and PCGI and others -- but NOT SCGI, sadly) is it will keep retrying the connection for about 10 seconds. So connections during that downtime get caught in the gateway between Apache and the Python server, and eventually get through (as long as the server really does come back up). It works quite well, and it's simple. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Fri Jun 10 17:55:47 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 10 Jun 2005 10:55:47 -0500 Subject: [Web-SIG] Daemon server management In-Reply-To: <20050609215800.GA14572@smullyan.org> References: <42A889C9.7080900@colorstudy.com> <20050609184121.GA13880@smullyan.org> <42A89004.4070409@colorstudy.com> <20050609215800.GA14572@smullyan.org> Message-ID: <42A9B803.5040308@colorstudy.com> I'm guessing you also meant to copy web-sig... Jacob Smullyan wrote: > On Thu, Jun 09, 2005 at 01:52:52PM -0500, Ian Bicking wrote: > >>Jacob Smullyan wrote: >> >>>On Thu, Jun 09, 2005 at 01:26:17PM -0500, Ian Bicking wrote: >>> >>> >>>>Does anyone have opinions on how to start and stop daemon servers? I've >>>>added a --daemon option to paster serve, but I'd like to implement stop, >>>>restart, and reload as well. Whenever I encounter servers that clobber >>>>pid files, or where the only way you can tell you've started a server >>>>twice is that you get an error message about not being able to bind to >>>>the port, it annoys me. But I'm not sure how to best implement a better >>>>system. Especially cross-platform -- though an entirely separate >>>>process for Windows might make sense (as a windows service or something). >>>> >>>>Opinions? Or examples of other servers (preferably Python-based) that >>>>do this well? >>> >>> >>>Clobbering pid files is a no-no; but getting an error about a port >>>being already in use doesn't seem terrible to me. >> >>Yes, but how to avoid clobbering pid files? It's probably a beginner >>question, and I've found workable things in the os module, but I don't >>actually know the right way to do this. > > > The os module has the best way to open a file, making sure that it > doesn't exist: > > try: > fd=os.open(fname, os.O_CREATE | os.O_EXCL) > except OSError, e: > if e.errno == errno.EEXIST: > logger.exception("File exists: %s", fname) > # actually, you should bomb out here > else: > logger.exception("IO error opening pid file!") > # same > else: > fp=os.fdopen(fd, 'w') > fp.write(str(os.getpid())) > fp.flush() > fp.close() > > but it isn't foolproof -- there can be a race condition on NFS, as > documented in "man open" (on Linux, at least). But anyone who stores > a pid file on an NFS filesystem is probably asking for it.... > > I recently drafted a new version of the skunkweb daemon, which tries > to be pretty traditional: > > http://svn.berlios.de/viewcvs/skunkweb/sandbox/smulloni/skunk4/src/skunk/net/server/ > > In light of your musing, I see several flaws. There are some things > it doesn't do that most textbook examples do -- it doesn't dup stdout > and stderr, for instance -- that I was aware of. I notice that I > didn't open the pid file so carefully -- I guess I'll change that. In > practice, starting a daemon twice would probably cause a port conflict > before the pid file is written, since two instances sharing the same > pid file are likely to have the same configuration, too. In Paste I can't really do that, since the pid file gets written before the server starts up, because it's server-agnostic, and none of the servers currently supported have any of this infrastructure themselves. >>I'd agree it's wrong to be clever and notice that the process is already >>running, then exiting without error. But it's right to notice the other >>process is running, and exit with a helpful error; helpful errors are >>always right. Should I even try to connect to a port if the process in >>the pid file is still alive, or should I bail immediately? > > > I think that if the pid file exists in any form, you are right to > refuse to start, with an error message about the pid file already > existing. But if this is a separate test, you could still clobber one > a moment later when you write one yourself; so a careful open is > probably the most important thing. I don't like this way of working -- a stale pid file should be overwritten automatically. Otherwise the admin has to figure out whether the pid file wasn't cleaned up properly, or the server really is alive. The server can figure that out just as well as the admin can manually (probably better). Though some cases are ambiguous, e.g., you can't be sure the live process is the same process that created the pid file. >>>I'd advocate the standard UNIX behavior for UNIX machines; pid file, >>>conventional signal handling (in particular, HUP reloads). For >>>Windows, the standard Windows behavior, whatever that might be; a >>>cross-platform solution would be neither fish nor fowl. This is not >>>just a matter of taste; conforming to the platform's expectations in >>>this area is the gracious thing to do, since packagers and system >>>administrators do not relish constantly having to write special >>>wrappers for non-standard daemons. >> >>I'm happy to copy conventions. Does anyone recommend a particular >>document on those conventions? For things like, do I open log files >>before or after I change user id (assuming the server is started as >>root)? And I'm a complete blank slate when it comes to the Windows >>side. Or even Macs, though I'm okay treating them like Unix to start. > > > Well, the books I like are the usual suspects: Stevens' UNIX Network > Programming, Vol. I, Johnson & Troan's Linux Application Development, > and I also rather like Lincoln Stein's treatment of the same territory > for Perl -- Network Pr. in Perl. Copying a good model, like Apache, > isn't a bad thing either. > > As for log files, I *think* that they end up belonging to root even if > the child processes setuid to a nobody-style person. That is what > I've done. That seems to be what apache does. Yes, I think that is the case. But I think the group ownership might change? I generally like how Apache works now, since they've combined httpd and apachectl, but I'm not sure how easy it would be for me to discover the particulars. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Jun 10 18:24:52 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 10 Jun 2005 12:24:52 -0400 Subject: [Web-SIG] Daemon server management In-Reply-To: <42A9B803.5040308@colorstudy.com> References: <20050609215800.GA14572@smullyan.org> <42A889C9.7080900@colorstudy.com> <20050609184121.GA13880@smullyan.org> <42A89004.4070409@colorstudy.com> <20050609215800.GA14572@smullyan.org> Message-ID: <5.1.1.6.0.20050610121435.0213beb0@mail.telecommunity.com> I don't know if it's directly useful to what you guys are talking about, but a lot of this sounds similar to PEAK's "supervisor" tool (peak.tools.supervisor), which is a framework for singleton daemons with multiple forked children. It manages a pid file, lock file, and startup lock file that ensure that only one copy of the thing can be started at once. If you start up a second copy while another copy is *running*, the second one gets its children ready to run, and then signals the first copy to shut down gracefully. The main use of this in PEAK is to run FastCGI servers listening on the same socket, but the base classes are designed to support any sort of system like this; all the FastCGI-specific stuff is in subclasses or plugins. Anyway, if you're interested, have a look at peak.tools.supervisor, especially the 'process' module which contains all the child monitoring code, startup lock management, pidfile handling, etc. It can use an Apache configuration-style config file (implemented w/ZConfig) to tell it how many child processes to run (minimum/maximum), how long to wait between starting new children, etc. At 10:55 AM 6/10/2005 -0500, Ian Bicking wrote: >I'm guessing you also meant to copy web-sig... > >Jacob Smullyan wrote: > > On Thu, Jun 09, 2005 at 01:52:52PM -0500, Ian Bicking wrote: > > > >>Jacob Smullyan wrote: > >> > >>>On Thu, Jun 09, 2005 at 01:26:17PM -0500, Ian Bicking wrote: > >>> > >>> > >>>>Does anyone have opinions on how to start and stop daemon servers? I've > >>>>added a --daemon option to paster serve, but I'd like to implement stop, > >>>>restart, and reload as well. Whenever I encounter servers that clobber > >>>>pid files, or where the only way you can tell you've started a server > >>>>twice is that you get an error message about not being able to bind to > >>>>the port, it annoys me. But I'm not sure how to best implement a better > >>>>system. Especially cross-platform -- though an entirely separate > >>>>process for Windows might make sense (as a windows service or something). > >>>> > >>>>Opinions? Or examples of other servers (preferably Python-based) that > >>>>do this well? > >>> > >>> > >>>Clobbering pid files is a no-no; but getting an error about a port > >>>being already in use doesn't seem terrible to me. > >> > >>Yes, but how to avoid clobbering pid files? It's probably a beginner > >>question, and I've found workable things in the os module, but I don't > >>actually know the right way to do this. > > > > > > The os module has the best way to open a file, making sure that it > > doesn't exist: > > > > try: > > fd=os.open(fname, os.O_CREATE | os.O_EXCL) > > except OSError, e: > > if e.errno == errno.EEXIST: > > logger.exception("File exists: %s", fname) > > # actually, you should bomb out here > > else: > > logger.exception("IO error opening pid file!") > > # same > > else: > > fp=os.fdopen(fd, 'w') > > fp.write(str(os.getpid())) > > fp.flush() > > fp.close() > > > > but it isn't foolproof -- there can be a race condition on NFS, as > > documented in "man open" (on Linux, at least). But anyone who stores > > a pid file on an NFS filesystem is probably asking for it.... > > > > I recently drafted a new version of the skunkweb daemon, which tries > > to be pretty traditional: > > > > > http://svn.berlios.de/viewcvs/skunkweb/sandbox/smulloni/skunk4/src/skunk/net/server/ > > > > In light of your musing, I see several flaws. There are some things > > it doesn't do that most textbook examples do -- it doesn't dup stdout > > and stderr, for instance -- that I was aware of. I notice that I > > didn't open the pid file so carefully -- I guess I'll change that. In > > practice, starting a daemon twice would probably cause a port conflict > > before the pid file is written, since two instances sharing the same > > pid file are likely to have the same configuration, too. > >In Paste I can't really do that, since the pid file gets written before >the server starts up, because it's server-agnostic, and none of the >servers currently supported have any of this infrastructure themselves. > > >>I'd agree it's wrong to be clever and notice that the process is already > >>running, then exiting without error. But it's right to notice the other > >>process is running, and exit with a helpful error; helpful errors are > >>always right. Should I even try to connect to a port if the process in > >>the pid file is still alive, or should I bail immediately? > > > > > > I think that if the pid file exists in any form, you are right to > > refuse to start, with an error message about the pid file already > > existing. But if this is a separate test, you could still clobber one > > a moment later when you write one yourself; so a careful open is > > probably the most important thing. > >I don't like this way of working -- a stale pid file should be >overwritten automatically. Otherwise the admin has to figure out >whether the pid file wasn't cleaned up properly, or the server really is >alive. The server can figure that out just as well as the admin can >manually (probably better). Though some cases are ambiguous, e.g., you >can't be sure the live process is the same process that created the pid >file. > > >>>I'd advocate the standard UNIX behavior for UNIX machines; pid file, > >>>conventional signal handling (in particular, HUP reloads). For > >>>Windows, the standard Windows behavior, whatever that might be; a > >>>cross-platform solution would be neither fish nor fowl. This is not > >>>just a matter of taste; conforming to the platform's expectations in > >>>this area is the gracious thing to do, since packagers and system > >>>administrators do not relish constantly having to write special > >>>wrappers for non-standard daemons. > >> > >>I'm happy to copy conventions. Does anyone recommend a particular > >>document on those conventions? For things like, do I open log files > >>before or after I change user id (assuming the server is started as > >>root)? And I'm a complete blank slate when it comes to the Windows > >>side. Or even Macs, though I'm okay treating them like Unix to start. > > > > > > Well, the books I like are the usual suspects: Stevens' UNIX Network > > Programming, Vol. I, Johnson & Troan's Linux Application Development, > > and I also rather like Lincoln Stein's treatment of the same territory > > for Perl -- Network Pr. in Perl. Copying a good model, like Apache, > > isn't a bad thing either. > > > > As for log files, I *think* that they end up belonging to root even if > > the child processes setuid to a nobody-style person. That is what > > I've done. That seems to be what apache does. > >Yes, I think that is the case. But I think the group ownership might >change? > >I generally like how Apache works now, since they've combined httpd and >apachectl, but I'm not sure how easy it would be for me to discover the >particulars. > >-- >Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org >_______________________________________________ >Web-SIG mailing list >Web-SIG at python.org >Web SIG: http://www.python.org/sigs/web-sig >Unsubscribe: >http://mail.python.org/mailman/options/web-sig/pje%40telecommunity.com From smulloni at smullyan.org Fri Jun 10 18:39:21 2005 From: smulloni at smullyan.org (Jacob Smullyan) Date: Fri, 10 Jun 2005 12:39:21 -0400 Subject: [Web-SIG] Daemon server management In-Reply-To: <42A9B803.5040308@colorstudy.com> References: <42A889C9.7080900@colorstudy.com> <20050609184121.GA13880@smullyan.org> <42A89004.4070409@colorstudy.com> <20050609215800.GA14572@smullyan.org> <42A9B803.5040308@colorstudy.com> Message-ID: <20050610163921.GA1858@smullyan.org> On Fri, Jun 10, 2005 at 10:55:47AM -0500, Ian Bicking wrote: > >I think that if the pid file exists in any form, you are right to > >refuse to start, with an error message about the pid file already > >existing. But if this is a separate test, you could still clobber one > >a moment later when you write one yourself; so a careful open is > >probably the most important thing. > > I don't like this way of working -- a stale pid file should be > overwritten automatically. Otherwise the admin has to figure out > whether the pid file wasn't cleaned up properly, or the server really is > alive. The server can figure that out just as well as the admin can > manually (probably better). Though some cases are ambiguous, e.g., you > can't be sure the live process is the same process that created the pid > file. Although in that case, you'd refuse to start anyway and the admin would have to figure out the situation; not an incorrect behavior. I agree, nothing wrong with checking whether the pid in an existing file corresponds to a process and overwriting it if it doesn't -- at least log that you are doing so, though. In the case of a forking server, this really doesn't matter much, because the papa proc of a forking server crashes so rarely. (I have never ever seen skunkweb crash, for instance, even when I was using pygresql and its children segfaulted every few minutes.) > >As for log files, I *think* that they end up belonging to root even if > >the child processes setuid to a nobody-style person. That is what > >I've done. That seems to be what apache does. > > Yes, I think that is the case. But I think the group ownership might > change? I see my apache logs owned by root:root. And this makes sense, actually; if the group changed, someone else might be able to write to the file and that could be bad. From the apache security tips page: If you allow non-root users to modify any files that root either executes or writes on then you open your system to root compromises. For example, someone could replace the httpd binary so that the next time you start it, it will execute some arbitrary code. If the logs directory is writeable (by a non-root user), someone could replace a log file with a symlink to some other system file, and then root might overwrite that file with arbitrary data. If the log files themselves are writeable (by a non-root user), then someone may be able to overwrite the log itself with bogus data. [http://httpd.apache.org/docs-2.0/misc/security_tips.html] > I generally like how Apache works now, since they've combined httpd and > apachectl, but I'm not sure how easy it would be for me to discover the > particulars. The 2.0 source intimidates me a bit, because of the growth of the codebase, apr, and the various worker modules. The 1.3 source is easier to navigate, although probably not to maintain, because of the maze of #ifdefs. j -- Jacob Smullyan -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/web-sig/attachments/20050610/ea91c425/attachment.pgp From jjinux at gmail.com Fri Jun 10 20:35:41 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Fri, 10 Jun 2005 11:35:41 -0700 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: <87r7fa8w9p.fsf@lantic.net> References: <87psuxngwq.fsf@lantic.net> <200506092337.00038.paul@boddie.org.uk> <87r7fa8w9p.fsf@lantic.net> Message-ID: Just for the record, Aquarium is both a proper framework (MVC, and all that) as well as server glue (like WSGI--it existed long before WSGI did), but it can also run on top of WSGI. http://aquarium.sf.net Best Regards, -jj On 6/10/05, Iwan Vosloo wrote: > Hi, > > Paul Boddie writes: > > > > If you take a look at WebStack (http://www.python.org/pypi/WebStack), you'll > > find a framework which provides its own "transaction" API and which works on > > top of other frameworks, precisely to avoid single project mappings such as > > those found in MoinMoin, for example. Given that you're studying so many > > "beasts", I'd appreciate anything you might have to say on the matter. > > I don't feel I quite understand these things well enough yet to make to many statements... But I'm planning to float my impressions here when I get to that stage. > > But, WRT WebStack - I had it loosely categorised as "server infrastructure" with things like twisted, Webware, or PEAK. But I see your point that it is trying to create a single interface to all of those. And that would indeed be useful. > My focus is more on the different models used by higher level frameworks that sit on top of, for example, WebStack (think CherryPy, Spyce, Zope, JSF, Struts, Tapestry and so on). The common point of interest here though has to do with the fact that I feel it is better to put something out there that tries to solve a small piece of the problem only - and that provides easy ways to use a collection of other projects together to accomplish your total solution. You have to be able to mix and match. Webstack plays in this field. And if there is support in terms of common libraries that you can count on being part of the standard Python distribution so much the better. > > I am concerned with the quantity of concepts that each framework (especially large ones & especially the Java ones) invents. This makes it difficult to understand "what they really do" (if that phrase makes at all sense). Also, I find that the concepts they invent are rarely well researched - they are mostly ad-hoc inventions of the authors and never relate to the known body of computer science literature. I'm also struggling to find much in the academic world regarding these animals. Nice big gap, thus. > > > > Another framework which follows the standardisation path, albeit with a Java > > Servlet emphasis, is Snakelets (http://www.python.org/pypi/Snakelets). > > WebStack and Snakelets are similar in certain respects, making me tempted to > > consider a bit of integration of the two. > > I don't want to make provocative statements without being able to back them up, but I have a _feeling_ (which I am investigating more closely at the moment) that the whole Java servlet model is a bit passe. Contrast it with something like CherryPy, for example, and just see what your gut feels... Its like the servlet model was born back when people thought of web sites as "having dynamic content". Now they see them as "web applications" and have taken that idea to a whole new level (in the Java world, anyway, albeit in its cumbersome ways). > > -i > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/jjinux%40gmail.com > -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through. From ianb at colorstudy.com Sat Jun 11 00:28:49 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 10 Jun 2005 17:28:49 -0500 Subject: [Web-SIG] Daemon server management In-Reply-To: <20050610163921.GA1858@smullyan.org> References: <42A889C9.7080900@colorstudy.com> <20050609184121.GA13880@smullyan.org> <42A89004.4070409@colorstudy.com> <20050609215800.GA14572@smullyan.org> <42A9B803.5040308@colorstudy.com> <20050610163921.GA1858@smullyan.org> Message-ID: <42AA1421.2030207@colorstudy.com> As a followup on this, this is what I have so far: When starting: if os.path.exists(pidfile): f = open(pidfile) pid = int(f.read().strip()) f.close() try: os.kill(pid, signal.SIG_DFL) # SIG 0, should do nothing except OSError: # Means no such process exists log('PID file %s exists, but process %i no longer running' % (pidfile, pid)) os.unlink(pidfile) else: print "Can't start, another server running (%i)" % pid sys.exit(1) I guess there's some possible race conditions at this point. But I'm not sure how to resolve those. I guess I could do Jacob's fd=os.open(fname, os.O_CREATE | os.O_EXCL) later. And match instead of os.path.exists() and open() I should open the pid file similarly in that code block above. Later, when killing the process, I'm doing: os.kill(pid, signal.SIGTERM) But I'm not sure if that's right either. The process seems to die, instead of properly terminating. Should I send another signal, and set up a signal handler in the server? Then perhaps I would send that signal, wait a bit, and send SIGTERM if it didn't stop on its own? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From iv at lantic.net Sat Jun 11 09:33:55 2005 From: iv at lantic.net (Iwan Vosloo) Date: Sat, 11 Jun 2005 09:33:55 +0200 Subject: [Web-SIG] PEP222 and python on the server? In-Reply-To: (Shannon's message of "Fri, 10 Jun 2005 11:35:41 -0700") References: <87psuxngwq.fsf@lantic.net> <200506092337.00038.paul@boddie.org.uk> <87r7fa8w9p.fsf@lantic.net> Message-ID: <87wtp14aak.fsf@lantic.net> Shannon -jj Behrens writes: > Just for the record, Aquarium is both a proper framework (MVC, and all > that) as well as server glue (like WSGI--it existed long before WSGI > did), but it can also run on top of WSGI. > > http://aquarium.sf.net Aquarium is on my list already, but thanks for the info! -i From davidf at sjsoft.com Mon Jun 13 12:47:16 2005 From: davidf at sjsoft.com (David Fraser) Date: Mon, 13 Jun 2005 12:47:16 +0200 Subject: [Web-SIG] Daemon server management In-Reply-To: <42AA1421.2030207@colorstudy.com> References: <42A889C9.7080900@colorstudy.com> <20050609184121.GA13880@smullyan.org> <42A89004.4070409@colorstudy.com> <20050609215800.GA14572@smullyan.org> <42A9B803.5040308@colorstudy.com> <20050610163921.GA1858@smullyan.org> <42AA1421.2030207@colorstudy.com> Message-ID: <42AD6434.9090602@sjsoft.com> Ian Bicking wrote: >As a followup on this, this is what I have so far: > >When starting: > >if os.path.exists(pidfile): > f = open(pidfile) > pid = int(f.read().strip()) > f.close() > try: > os.kill(pid, signal.SIG_DFL) # SIG 0, should do nothing > except OSError: > # Means no such process exists > log('PID file %s exists, but process %i no longer running' > % (pidfile, pid)) > os.unlink(pidfile) > else: > print "Can't start, another server running (%i)" % pid > sys.exit(1) > >I guess there's some possible race conditions at this point. But I'm >not sure how to resolve those. I guess I could do Jacob's >fd=os.open(fname, os.O_CREATE | os.O_EXCL) later. And match instead of >os.path.exists() and open() I should open the pid file similarly in that >code block above. > > >Later, when killing the process, I'm doing: > >os.kill(pid, signal.SIGTERM) > >But I'm not sure if that's right either. The process seems to die, >instead of properly terminating. Should I send another signal, and set >up a signal handler in the server? Then perhaps I would send that >signal, wait a bit, and send SIGTERM if it didn't stop on its own? > > It would be really nice if there was a builtin Python module that handled interprocess mutexes, locks etc on a high level, in the same way that threading does (but for processes, not threads). I've found most of my time dealing with this issue taken up with dealing with the interprocess locking on a low level. Anyone know of any such modules (not neccessarily builtin, of course)? David From davidf at sjsoft.com Tue Jun 14 13:46:20 2005 From: davidf at sjsoft.com (David Fraser) Date: Tue, 14 Jun 2005 13:46:20 +0200 Subject: [Web-SIG] Daemon server management In-Reply-To: <42AD6434.9090602@sjsoft.com> References: <42A889C9.7080900@colorstudy.com> <20050609184121.GA13880@smullyan.org> <42A89004.4070409@colorstudy.com> <20050609215800.GA14572@smullyan.org> <42A9B803.5040308@colorstudy.com> <20050610163921.GA1858@smullyan.org> <42AA1421.2030207@colorstudy.com> <42AD6434.9090602@sjsoft.com> Message-ID: <42AEC38C.2020903@sjsoft.com> David Fraser wrote: >Ian Bicking wrote: > > > >>As a followup on this, this is what I have so far: >> >>When starting: >> >>if os.path.exists(pidfile): >> f = open(pidfile) >> pid = int(f.read().strip()) >> f.close() >> try: >> os.kill(pid, signal.SIG_DFL) # SIG 0, should do nothing >> except OSError: >> # Means no such process exists >> log('PID file %s exists, but process %i no longer running' >> % (pidfile, pid)) >> os.unlink(pidfile) >> else: >> print "Can't start, another server running (%i)" % pid >> sys.exit(1) >> >>I guess there's some possible race conditions at this point. But I'm >>not sure how to resolve those. I guess I could do Jacob's >>fd=os.open(fname, os.O_CREATE | os.O_EXCL) later. And match instead of >>os.path.exists() and open() I should open the pid file similarly in that >>code block above. >> >> >>Later, when killing the process, I'm doing: >> >>os.kill(pid, signal.SIGTERM) >> >>But I'm not sure if that's right either. The process seems to die, >>instead of properly terminating. Should I send another signal, and set >>up a signal handler in the server? Then perhaps I would send that >>signal, wait a bit, and send SIGTERM if it didn't stop on its own? >> >> >> >> >It would be really nice if there was a builtin Python module that >handled interprocess mutexes, locks etc on a high level, in the same way >that threading does (but for processes, not threads). I've found most of >my time dealing with this issue taken up with dealing with the >interprocess locking on a low level. > >Anyone know of any such modules (not neccessarily builtin, of course)? > > I found glock from http://rgruet.free.fr/ It's quite nice except that it had a few problems when I tried it, so I fixed them... Basically you can create a GlobalLock that uses flocks on Unix and named mutexes on Win32, and acquire / release them. Tested and have a nice handover system working in jToolkit which works by having a socket lock that controls access to the socket, and a series of running locks which control the handover process Basic logic as follows: Parent: 1) Starts up, creates socket lock called "socket-$pid" and running lock called "running-$pid-$n" (starts at n=1) 2) Starts child process, passing socket lock name and running lock name on command line 3) Waits for child process to acquire running lock 4) Waits for child process to release running lock 5) n += 1, creates new running lock "running-$pid-$n" 5) go to 2 Child: 1) Acquires Running lock 2) Acquires Socket lock 3) Binds socket, starts listening for requests 4) When listened to maxrequests, releases running lock 5) waits for some process to acquire "running-$pid-n+1" (handling requests in the mean time) 6) closes the socket, and releases the socket lock My altered glock.py is at http://davidf.sjsoft.com/files/glock.py David From renesd at gmail.com Thu Jun 16 12:41:02 2005 From: renesd at gmail.com (Rene Dudfield) Date: Thu, 16 Jun 2005 20:41:02 +1000 Subject: [Web-SIG] lighttpd supports Message-ID: <64ddb72c05061603412fe2eed9@mail.gmail.com> Hey, the prerelease of lighttpd 1.3.14 supports SCGI. Which means better support for webware, wsgi, and zope :) Yah! Details here: http://blog.lighttpd.net/ http://www.lighttpd.net/ lighttpd is a really sweet featureful, high performance, and low resource httpd server that lots of our rails friends use. Cheers, Rene Dudfield. www.madecollective.com www.pretendpaper.com From chad at zetaweb.com Thu Jun 16 13:21:51 2005 From: chad at zetaweb.com (Chad Whitacre) Date: Thu, 16 Jun 2005 07:21:51 -0400 Subject: [Web-SIG] lighttpd supports In-Reply-To: <64ddb72c05061603412fe2eed9@mail.gmail.com> References: <64ddb72c05061603412fe2eed9@mail.gmail.com> Message-ID: <42B160CF.2010306@zetaweb.com> > lighttpd is a really sweet featureful, high performance, and low > resource httpd server that lots of our rails friends use. Too bad you have to use PHP to get WebDAV.[1] Any viable Python DAV implementations out there? You would also need an svn solution to really ditch apache. chad ----- [1] http://trac.lighttpd.net/trac/ticket/22 From ianb at colorstudy.com Thu Jun 16 17:41:04 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 16 Jun 2005 10:41:04 -0500 Subject: [Web-SIG] lighttpd supports In-Reply-To: <42B160CF.2010306@zetaweb.com> References: <64ddb72c05061603412fe2eed9@mail.gmail.com> <42B160CF.2010306@zetaweb.com> Message-ID: <42B19D90.5060902@colorstudy.com> Chad Whitacre wrote: >>lighttpd is a really sweet featureful, high performance, and low >>resource httpd server that lots of our rails friends use. > > > Too bad you have to use PHP to get WebDAV.[1] Any viable Python DAV > implementations out there? Well, there are a couple: * http://sandbox.xerox.com/webdav/ * http://webdav.de/ * Zope * http://akadav.sourceforge.net/ I can't vouch for them, but I'd be very interested in other people's experiences. I haven't seen people using any of them as very generic way, except for Zope, but you get the entirety of Zope in the process. Outside of Zope, I don't know of any, though I'd be very interested in such a thing. Incidentally making such a thing was one of the ideas I put up for Google's Summer of Code. But I don't know if anyone took that one up. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From chad at zetaweb.com Thu Jun 16 17:43:58 2005 From: chad at zetaweb.com (Chad Whitacre) Date: Thu, 16 Jun 2005 11:43:58 -0400 Subject: [Web-SIG] lighttpd supports In-Reply-To: <42B19D90.5060902@colorstudy.com> References: <64ddb72c05061603412fe2eed9@mail.gmail.com> <42B160CF.2010306@zetaweb.com> <42B19D90.5060902@colorstudy.com> Message-ID: <42B19E3E.90705@zetaweb.com> Ian, Thanks for the reply. > * http://sandbox.xerox.com/webdav/ > * http://webdav.de/ > * Zope > * http://akadav.sourceforge.net/ I found the first two on google, but it looks like they haven't been touched in 5 years. Zope is, well, Zope. akadav is news to me, thanks. chad From jjinux at gmail.com Thu Jun 16 20:06:22 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Thu, 16 Jun 2005 11:06:22 -0700 Subject: [Web-SIG] Laszlo Message-ID: Please forgive me if this is a bit off topic, but many of you may be interested in . Best Regards, -jj -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through. From todd at slack.net Fri Jun 17 06:11:30 2005 From: todd at slack.net (Todd Grimason) Date: Fri, 17 Jun 2005 00:11:30 -0400 Subject: [Web-SIG] Rails pushing FastCGI into megahost Message-ID: <20050617041130.GA6230@detroit.slack.net> I think this might be of interest to some here - apparently Dreamhost has just rolled out support for Rails (apparently all ruby installs are up to date at 1.8.2 - python's up to 2.2.x) via FastCGI. [0] I don't have an account with them these days, but it was mentioned on the Rails list - someone over there (and I'd guess others) had been asking them for it apparently. No word yet on performance or reliability. I suppose it could be in response to something besides Rails, but based on the publicity and explicit listing on their pages I'd say Rails was the impetus (well, the promise of $ to be made with it). Anyway - point is - good news for increasing the "deployability" for python, ruby, perl, etc. and allowing apps to be written in langs besides PHP for clients who can only swing commodity shared hosts. Maybe someone here should petition them re: WSGI? Or maybe that's pushing one's luck... [0] http://dreamhost.com/panel/features.html#ruby -- ________________________________ toddgrimason*todd[ at ]slack.net From pje at telecommunity.com Fri Jun 17 06:30:23 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 17 Jun 2005 00:30:23 -0400 Subject: [Web-SIG] Rails pushing FastCGI into megahost In-Reply-To: <20050617041130.GA6230@detroit.slack.net> Message-ID: <5.1.1.6.0.20050617002802.033d1970@mail.telecommunity.com> At 12:11 AM 6/17/2005 -0400, Todd Grimason wrote: >I think this might be of interest to some here - apparently Dreamhost >has just rolled out support for Rails (apparently all ruby installs are >up to date at 1.8.2 - python's up to 2.2.x) via FastCGI. [0] > >I don't have an account with them these days, but it was mentioned on >the Rails list - someone over there (and I'd guess others) had been >asking them for it apparently. No word yet on performance or >reliability. I suppose it could be in response to something besides >Rails, but based on the publicity and explicit listing on their pages >I'd say Rails was the impetus (well, the promise of $ to be made with >it). > >Anyway - point is - good news for increasing the "deployability" for >python, ruby, perl, etc. and allowing apps to be written in langs >besides PHP for clients who can only swing commodity shared hosts. Maybe >someone here should petition them re: WSGI? Or maybe that's pushing >one's luck... WSGI does FastCGI; PEAK in particular has WSGI gateways for both single-process and forking multi-process FastCGI, that can run as either "external" (you start the process and it listens on a connection) or "internal" (the web server starts the process with a socket on stdin). I believe others have implemented FastCGI gateways for WSGI as well. It's not rocket science, unless you need an extremely reliable setup. :) From renesd at gmail.com Fri Jun 17 07:01:24 2005 From: renesd at gmail.com (Rene Dudfield) Date: Fri, 17 Jun 2005 15:01:24 +1000 Subject: [Web-SIG] Rails pushing FastCGI into megahost In-Reply-To: <20050617041130.GA6230@detroit.slack.net> References: <20050617041130.GA6230@detroit.slack.net> Message-ID: <64ddb72c050616220141c9a0c2@mail.gmail.com> Nice one :) Hopefully this will help things... I think the key is python keeping a low memory foot print for it to be available on lots of shared hosts. Where it uses up 0 memory if there are no requests for the site, and maybe 5-10MB per site when there are requests. With 50-200 sites per machine, that is 250MB- 2gigs of memory. FCGI and python can be used with 0 memory for inactive sites... so hopefully it catches on :) Having support in cpanel, and other popular control panels would be key I think. Also having the apache module in core apache would be good. Here's something about FCGI + ruby on rails with cpanel. Luckily fcgi seems quite easy for hosts to set up with cpanel. http://wiki.rubyonrails.com/rails/show/HowtoInstallAndRunRubyOnRailsOnCpanel However until the time comes when it's on 90% of hosts, I'll have to keep doing 99% of client sites with php, and only get to use python on my own sites. Clients with hosts allready do not want to change. Even if the alternatives are cheaper, and faster. It's just extra work for them. On 6/17/05, Todd Grimason wrote: > I think this might be of interest to some here - apparently Dreamhost > has just rolled out support for Rails (apparently all ruby installs are > up to date at 1.8.2 - python's up to 2.2.x) via FastCGI. [0] > > I don't have an account with them these days, but it was mentioned on > the Rails list - someone over there (and I'd guess others) had been > asking them for it apparently. No word yet on performance or > reliability. I suppose it could be in response to something besides > Rails, but based on the publicity and explicit listing on their pages > I'd say Rails was the impetus (well, the promise of $ to be made with > it). > > Anyway - point is - good news for increasing the "deployability" for > python, ruby, perl, etc. and allowing apps to be written in langs > besides PHP for clients who can only swing commodity shared hosts. Maybe > someone here should petition them re: WSGI? Or maybe that's pushing > one's luck... > > [0] http://dreamhost.com/panel/features.html#ruby > > -- From foom at fuhm.net Sat Jun 18 03:40:18 2005 From: foom at fuhm.net (James Y Knight) Date: Fri, 17 Jun 2005 21:40:18 -0400 Subject: [Web-SIG] twisted.web2 0.1.0 released Message-ID: I'm happy to announce the first release of twisted.web2 today. This is a rewrite/refactor of the twisted.web HTTP server and framework, providing a nicer API (some parts inspired by the resource API that Nevow introduced on top of twisted.web) and many internal improvements, especially in the area of HTTP conformance. Some highlights: - Fully asynchronous twisted server. - Can run as a HTTP/HTTPS server, or behind an apache mod_proxy, or serve requests via SCGI, or even just plain CGI. - Supports SCGI clients, CGI scripts, WSGI applications. No new HTTP client/proxy yet, it's still on the TODO list. - Also supports most twisted.web resources through a compatibility layer. - Implements HTTP 1.1, including keep-alive, chunking, and pipelining. - Has premade, correct, header parsers and generators for most of the headers. - Supports streaming data input and output, has a nonblocking multipart/form-data parser. - Automatic support for If-[Un]Modified-Since, If-[None-]Match and Range. You can find downloads and documentation here: http://twistedmatrix.com/projects/web2/ You'll need at least Twisted 2.0.1 as well: http://twistedmatrix.com/projects/core/ This is still a preliminary release -- the API is not guaranteed to be stable to the next release. James From jjinux at gmail.com Thu Jun 30 19:12:39 2005 From: jjinux at gmail.com (Shannon -jj Behrens) Date: Thu, 30 Jun 2005 10:12:39 -0700 Subject: [Web-SIG] need a Python Web guy Message-ID: Hi, IronPort is looking for a Python Web application developer. We are located in San Bruno, California. Unfortunately, we don't allow employees to work remotely, but I can attest to the fact that IronPort is a great place to work. Please send your resume to , , and . Best Regards, -jj -- I have decided to switch to Gmail, but messages to my Yahoo account will still get through.