Mailman 3 December 2005 - Twisted-web

Using Twisted Web for browser-based control
by Drake Smith Dec. 18, 2005

Dec. 18, 2005

(non-HTML version.....sorry about that) I use Twisted Web on an embedded Linux device to provide browser-based configuration & control. The browser displays a bar graph of a parameter that is quickly changing on the embedded device (peak audio level). In order to provide a reasonable feedback to the user, the browser connects to the Twisted Web CGI server once a second via XMLHTTPRequest to obtain an updated bar graph level, then Twisted Web dutifully closes the connection. Is there a way to achieve the same thing without opening & closing a connection each time? My code: # ----- Set up the web/CGI server to handle browser-based control ----- from twisted.web import server, static, twcgi root = static.File("/myResourceDir") root.putChild("cgi-bin", twcgi.CGIDirectory("/myResourceDir/cgi-bin")) reactor.listenTCP(8000, server.Site(root)) Thank you.

3 3

Standard sequence renderer "loosing" tags?
by A. Goryunov Dec. 17, 2005

Dec. 17, 2005

I've noticed that Page.renderString() "swallows" some of the stan nodes. I've also tried the example below in an actual Twisted service and got the same result. Doing it with an XML template also yields erroneous output. Namely, it looses "table", "tr", a couple of "td" tags were MIA too I believe, as well as two "input"s. The code I was using for testing is attached. It's supposed to be a library of widgets that are constructed using Fragments and render_sequence. from nevow import tags as T from nevow import flat, rend, loaders class TestPage(rend.Page): docFactory = loaders.stan( T.html[ T.body[ T.invisible(render=T.directive("widget"))[ T.invisible(pattern="widget_name")['PortalContent'], T.invisible(pattern="widget_slot")['main_content'], T.invisible(pattern="widget_slot")[ T.invisible(render=T.directive("widget"))[ T.invisible(pattern="widget_name")['PortalBarVert'], T.invisible(pattern="widget_slot")[ T.invisible(render=T.directive("widget"))[ T.invisible(pattern="widget_name")['LoginForm'], T.invisible(pattern="widget_slot")['Sign in'], T.invisible(pattern="widget_slot")['User: '], T.invisible(pattern="widget_slot")['Password: '], T.invisible(pattern="widget_slot")[T.input(type="submit", name="null", value="Submit")] ] ] ] ] ] ]]) def render_widget(self,ctx,data): wName = ctx.tag.onePattern("widget_name").children[0].strip('\n\t ') attrs = ctx.tag.allPatterns("widget_slot") args = [] for attr in attrs: args.append(T.xml(attr.children)) try: wdgt = globals()[wName](args) return ctx.tag[wdgt] except IndexError: return ctx.tag[T.p(style="font-color : red")["Unknown widget name: %s" % wName]] class LoginForm(rend.Fragment): docFactory = loaders.stan( T.div(class_="login-form", render=T.directive("sequence"))[ T.p(class_="small-form-title", pattern="item", render=T.directive('arg')), T.form(action="_submit", method="post")[ T.table(id="login-form-table")[ T.tr[ T.td(align="right")[ T.label(for_="user", pattern="item", render=T.directive('arg')) ], T.td(align="left")[ T.input(type="text", name="user", class_="login-input") ] ], T.tr[ T.td(align="right")[ T.label(for_="pass", pattern="item", render=T.directive('arg')) ], T.td(align="left")[ T.input(type="password", name="pass", class_="login-input") ] ], T.tr[T.td(colspan="2")], T.tr[ T.td(colspan="2", align="center", pattern="item", render=T.directive('arg')) ] ] ] ]) def render_arg(self,ctx,data): return ctx.tag[data] class PortalContent(rend.Fragment): docFactory = loaders.stan( T.table(id="portal-content", render=T.directive("sequence"))[ T.tr[ T.td(class_="portal-content-main", pattern='item', render=T.directive('arg')), T.td(class_="portal-content-sidebar", id="portal-sidebar-right", pattern="item", render=T.directive("arg")) ] ]) def render_arg(self,ctx,data): return ctx.tag[data] class PortalBarVert(rend.Fragment): docFactory = loaders.stan( T.table(class_="portal-bar-vertical", render=T.directive('sequence'))[ T.tr(pattern='item', render=T.directive('arg')) ]) def render_arg(self,ctx,data): return ctx.tag[T.td(class_='vert-sidebar-slot')[data]] page = TestPage() df = page.renderString() def printPage(result): print result def printErr(err): print err df.addCallback(printPage) df.addErrback(printErr)

1 1

interface.queryDescriptionFor instead of getattr
by Andrea Arcangeli Dec. 17, 2005

Dec. 17, 2005

I noticed a deprecation warning, and I tried to fix it but it breaks stuff so I backed it out, I post it just in case somebody can make a right fix for the deprecation warning. This intermediate step between compy and zope interfaces is very confusing. Also note I totally dislike interfaces, all of them, no matter if they're form zope, twisted, nevow.copy or whatever else interface. They're an huge waste of cpu too, I wish all code was written without them. Unless I'm absolutely forced by things like nevow, I never use the interfaces insanity in my code. Perhaps I will change my mind after I stop seeing all my cpu sucked by them, but at the moment I would never dream to touch them in my code. I strongly reccomend people to never use interfaces, they're a bad idea, language lawyers may appreciate them so they can put to work all features of python, but if you want your code simple, fast and readable you should always avoid them. 2005/12/17 15:22 CET [HTTPChannel,0,127.0.0.1] /home/andrea/bin/x86_64/python/lib/python2.4/site-packages/formless/webform.py:460: twisted.python.components.ComponentsDeprecationWarning: Don't get attributes (in this case, '__spec__') off Interface, use .queryDescriptionFor() etc. instead Index: configurable.py =================================================================== --- configurable.py (revision 3683) +++ configurable.py (working copy) @@ -30,7 +30,7 @@ for interface in ifs: ## TypedInterfaces have a __spec__ attribute which is a list of all Typed properties and ## autocallable methods - for binding in getattr(interface, '__spec__', []): + for binding in interface.queryDescriptionFor('__spec__', []): bindingDict[binding.name] = binding if binding.name not in bindingNames: bindingNames.append(binding.name) @@ -162,7 +162,7 @@ bindingNames = [] self.bindingDict = bindingDict = {} interface = self.groupInterface - for binding in getattr(interface, '__spec__', []): + for binding in interface.queryDescriptionFor('__spec__', []): bindingDict[binding.name] = binding if binding.name not in bindingNames: bindingNames.append(binding.name)

1 0

Re: [Twisted-Python] WSGI Thread-management strategy
by Jim Fulton Dec. 16, 2005

Dec. 16, 2005

James Y Knight wrote: > (BTW, the correct mailing list for twisted webbish stuff is twisted- > web(a)twistedmatrix.com) > On Dec 15, 2005, at 4:22 PM, Jim Fulton wrote: > >> The strategy used by twisted WSGI, as I understand it, doesn't meet >> our needs. Currently, a thread is created for each request. The total >> number of threads is throttled, I gather using a general Twisted >> thread limit. WSGI applications are called as soon as input headers >> have been received completely. An application may be called before all >> body input is received. We need application calls to be delayed until >> all request input has been received, >> >> [...] >> >> I propose that the default thread-management strategy should be to delay >> calling an application until all request input has been received. If >> this isn't the default, then there should at least be an option to get >> this behavior. (Of course, the buffering strategy needs to be clever >> enough to switch to a file when the input gets over some size.) > > > Sounds sensible, Does this mean you will make this the default in the future? > and is doable external to the WSGI wrapper. Here's a > little bit I whipped up. (works on the 2.1.x branch and head). Could be > smarter, by starting out the buffer in memory and switching to a file > if necessary. Also shows off a couple of minor bugs I need to fix. :) > > def simple_wsgi_app(environ, start_response): > print "Starting wsgi app" > start_response("200 OK", [('Content-type','text/html; > charset=ISO-8859-1')]) > data = environ['wsgi.input'].read() > return ['<pre>', data, '</pre>'] > > > class Prebuffer(resource.WrapperResource): > def hook(self, ctx): > req = iweb.IRequest(ctx) > temp = tempfile.TemporaryFile() > def done(_): > temp.seek(0) > # Replace the request's stream object with the tempfile > req.stream = stream.FileStream(temp) > # Hm, this shouldn't be required: > req.stream.doStartReading = None > return stream.readStream(req.stream, temp.write).addCallback (done) > > # Oops, fix missing () in lambda in WrapperResource > def locateChild(self, ctx, segments): > x = self.hook(ctx) > if x is not None: > return x.addCallback(lambda data: (self.res, segments)) > return self.res, segments > > if __name__ == '__builtin__': > from twisted.application import service, strports > from twisted.web2 import server, channel > > res = Prebuffer(wsgi.WSGIResource(simple_wsgi_app)) > > site = server.Site(res) > application = service.Application("demo") > > s = strports.service('tcp:8080', channel.HTTPFactory(site)) > s.setServiceParent(application) This is greek to me. I'd forward this to the people who did the Twisted Zope integration. Thanks! Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

1 0

pgasync INTERVAL
by Andrea Arcangeli Dec. 16, 2005

Dec. 16, 2005

I added the interval type conversion to allow KLive to work on top of pgasync without requiring changes to the KLive code. However in trying to run cpushare on top of pgasync, I'm getting this error that I didn't finish debugging yet. 2005/12/16 14:18 CET [PgProtocol,client] [Failure instance: Traceback (failure with no frames): pgasync.errors.DatabaseError: 34000: cursor "none" does not exist 2005/12/16 14:18 CET [PgProtocol,client] ] I didn't extract any meaningful stack trace yet. Index: pgasync/pgasync/fe.py =================================================================== --- pgasync/pgasync/fe.py (revision 86) +++ pgasync/pgasync/fe.py (working copy) @@ -349,7 +349,7 @@ DEFERRED - Execute a query with he given parameters properly formatted. + Execute a query with the given parameters properly formatted. The parameter style is 'pyparam,' which uses a syntax like the following: Index: pgasync/pgasync/pgtypes.py =================================================================== --- pgasync/pgasync/pgtypes.py (revision 86) +++ pgasync/pgasync/pgtypes.py (working copy) @@ -120,7 +120,7 @@ @staticmethod def fromDatabase(ins): if ins.count("-"): - ins = ins.split("-")[0] #discarding timestamp for right tnow + ins = ins.split("-")[0] #discarding timestamp for right now try: h,m,sparts = ins.split(":") @@ -157,6 +157,44 @@ return datetime.datetime(dt.year,dt.month,dt.day, tm.hour,tm.minute,tm.second,tm.microsecond) +class INTERVAL(datetime.timedelta): + def __init__(self, *args): + datetime.timedelta.__init__(self, *args) + + @staticmethod + def toDatabase(s): + return "'%d days %d seconds %d microseconds'" % (s.days, s.seconds, s.microsecond) + + def __str__(self): + return INTERVAL.toDatabase(self) + + @staticmethod + def fromDatabase(ins): + try: + ins = re.split(' days? ', ins) + if len(ins) == 1: + ins = ins[0] + d = 0 + elif len(ins) == 2: + d = int(ins[0]) + ins = ins[1] + + h,m,sparts = ins.split(":") + h,m = map(int,(h,m)) + if sparts.count("."): + ssegs = sparts.split(".") + s,ms = map(int,ssegs) + if ms: + l = len(ssegs[1]) + ms *= 10 ** (6 - l) + else: + s = int(sparts) + ms = 0 + except: + raise + raise DataError, ("Cannot convert string '%s' to datetime.timedelta" % ins) + return datetime.timedelta(days=d,hours=h,minutes=m,seconds=s,microseconds=ms) + class BOOL: def __init__(self, b): self.__b = b Index: pgasync/pgasync/__init__.py =================================================================== --- pgasync/pgasync/__init__.py (revision 86) +++ pgasync/pgasync/__init__.py (working copy) @@ -47,6 +47,7 @@ registerAdapter(DATE, [datetime.date], [PgOids.DATE]) registerAdapter(TIME, [datetime.time], [PgOids.TIME,PgOids.ABSTIME,PgOids.RELTIME]) registerAdapter(DATETIME, [datetime.datetime], [PgOids.TIMESTAMP]) +registerAdapter(INTERVAL, [datetime.timedelta], [PgOids.INTERVAL,PgOids.TINTERVAL]) registerAdapter(BOOL, [bool], [PgOids.BOOL])

1 2

remove ugliness when creating Nevow sessions
by Andrea Arcangeli Dec. 15, 2005

Dec. 15, 2005

This isn't only ugly, it also screwup wget. I left it commented just in case. Index: Nevow/nevow/guard.py =================================================================== --- Nevow/nevow/guard.py (revision 3434) +++ Nevow/nevow/guard.py (working copy) @@ -299,7 +299,8 @@ if path.startswith(SESSION_KEY): key = path[len(SESSION_KEY):] if key not in self.sessions: - return urlToChild(ctx, *segments[1:], **{'__start_session__':1}), () + #return urlToChild(ctx, *segments[1:], **{'__start_session__':1}), () + return urlToChild(ctx, *segments[1:]), () self.sessions[key].setLifetime(self.sessionLifetime) if cookie == key: # /sessionized-url/${SESSION_KEY}aef9c34aecc3d9148/foo @@ -307,6 +308,7 @@ # we are this getChild # with a matching cookie self.sessions[key].sessionJustStarted = True + #return urlToChild(ctx, *segments[1:], **{'__session_just_started__':1}), () return urlToChild(ctx, *segments[1:]), () else: # We attempted to negotiate the session but failed (the user

2 6

socket.error: (24, 'Too many open files')
by 이영남 Dec. 14, 2005

Dec. 14, 2005

Hi. I'm twisted beginner. and My english is not good. ^^;; I have some troubles using Twisted. I make a web server with Twisted. I'd like to test my webserver performance. So Using "httperf", I test my webserver. But *result in*, File "/usr/local/lib/python2.3/site-packages/twisted/python/log.py", line 52, in callWithContext File "/usr/local/lib/python2.3/site-packages/twisted/python/context.py", line 64, in callWithContext File "/usr/local/lib/python2.3/site-packages/twisted/python/context.py", line 43, in callWithContext File "/usr/local/lib/python2.3/site-packages/twisted/internet/default.py", line 535, in _doReadOrWrite --- <exception caught here> --- File "/usr/local/lib/python2.3/site-packages/twisted/internet/tcp.py", line 625, in doRead File "/usr/local/lib/python2.3/socket.py", line 167, in accept socket.error: (24, 'Too many open files') I think the reason is *file open limit per one process(maybe 1024).* how can I solve this problem? I'd like to If Process *can not open file*, that request make wait-status*(not allow socket accept).* and when Process can open file, handle the request.

3 2

twisted-web file upload processing on the server
by Matthew Thorley Dec. 14, 2005

Dec. 14, 2005

Please forgive me if this is obvious, but I couldn't find the answer any where! I searched google, the new twisted book, the twisted-web docs, and the mailing list archive. I saw other posts with similar questions but not the answer, so here goes. I'm using twisted version 2.1.0 and I did the sumo download to make it easy on myself I've got a setup like this on the server: class UploadPage(resource.Resource): def render_POST(self, request): filename = request.args['cml'][0] # filecontent = ??? return "You uploaded file: %s" %filename And I'm using a form like this to send the file: <form enctype="mulitpart/form-data" action="/upload" method="post"> <p>CML: <input name="cml" type="file"/></p> <p><input type="submit" name="submit" value="Upload"/></p> </form> When I upload the file everything works and the correct file name shows in the output. But what I can't do is read/get the contents of the file. Can some one please explain how to do server side file upoad processing??? I'm totaly lost here. I'm sure that it must be something painfully obvious, since uploads are such a common thing. But I can't figure it out. Thanks much Matthew Thorley

3 2

forwarding client information from twisted proxy to nevow monster
by Andrea Arcangeli Dec. 13, 2005

Dec. 13, 2005

Without this patch, a nevow server behind twisted proxy can't receive the correct client information. It's backwards compatible, passClient = True must be explicitly passed when creating both the proxy and monster classes for this to work (and as usual the monster resource must be under the firewall, only the "trusted" proxy must be allowed to connect to it, for the client info to be trustable). Here the Nevow part. Index: Nevow/nevow/vhost.py =================================================================== --- Nevow/nevow/vhost.py (revision 3434) +++ Nevow/nevow/vhost.py (working copy) @@ -134,13 +134,15 @@ *after* it returns the (resource,segments) tuple. """ implements(inevow.IResource) + + def __init__(self, vhost_segments): + self.prepath_segments = vhost_segments+1 # +1 is for /vhost + def locateChild(self, ctx, segments): request = inevow.IRequest(ctx) - request.prepath = request.prepath[3:] + request.prepath = request.prepath[self.prepath_segments:] return request.site.resource, segments -_prepathCleaner = _VHostMonsterResourcePrepathCleaner() - class VHostMonsterResource: """VHostMonster resource that helps to deploy a Nevow site behind a proxy. @@ -174,17 +176,30 @@ This also means your private server should serve the real content at /foo/bar, and not at the root of the tree. + If passClient is set to True it expects the client to be passed by the + proxy (see ReverseProxyResource passClient parameter for details). When + passClient is True this page should not be reacheable directly from the + internet if logging the IP address and port securely is needed. + Warning: anyone who can access a VHostMonsterResource can fake the host name they are contacting. This can lead to cookie stealing or cross site scripting attacks. Never expose /vhost to the Internet. """ implements(inevow.IResource) + vhost_segments = 2 + + def __init__(self, passClient = False): + self.passClient = passClient + if passClient: + self.vhost_segments = 3 + self._prepathCleaner = _VHostMonsterResourcePrepathCleaner(self.vhost_segments) + def locateChild(self, ctx, segments): request = inevow.IRequest(ctx) - if len(segments) < 2: + if len(segments) < self.vhost_segments: return rend.NotFound else: if segments[0] == 'http': @@ -197,13 +212,18 @@ port = int(port) else: host, port = segments[1], 80 - request.setHost(host, port) - prefixLen = len('/'+'/'.join(request.prepath)+'/'+'/'.join(segments[:2])) - request.path = '/'+'/'.join(segments[2:]) + if self.passClient: + if ':' not in segments[2]: + return rend.NotFound + host, port = segments[2].split(':') + request.setClient(host, port) + + prefixLen = len('/'+'/'.join(request.prepath)+'/'+'/'.join(segments[:self.vhost_segments])) + request.path = '/'+'/'.join(segments[self.vhost_segments:]) request.uri = request.uri[prefixLen:] - return _prepathCleaner, segments[2:] + return self._prepathCleaner, segments[self.vhost_segments:] compy.backwardsCompatImplements(VHostMonsterResource) Here the twisted part: Index: Twisted/twisted/web/http.py =================================================================== --- Twisted/twisted/web/http.py (revision 15293) +++ Twisted/twisted/web/http.py (working copy) @@ -876,6 +876,10 @@ self.received_headers["host"] = host self.host = address.IPv4Address("TCP", host, port) + def setClient(self, host, port): + """Same as setHost but for the client address""" + self.client = address.IPv4Address("TCP", host, port) + def getClientIP(self): if isinstance(self.client, address.IPv4Address): return self.client.host Index: Twisted/twisted/web/proxy.py =================================================================== --- Twisted/twisted/web/proxy.py (revision 15293) +++ Twisted/twisted/web/proxy.py (working copy) @@ -162,23 +162,33 @@ to a different server. """ - def __init__(self, host, port, path): + def __init__(self, host, port, path, passClient = False): resource.Resource.__init__(self) self.host = host self.port = port self.path = path + self.passClient = passClient + def getPath(self, request): + path = self.path + if self.passClient: + path += '/%s:%d' % (request.client.host, request.client.port) + return path + def getChild(self, path, request): - return ReverseProxyResource(self.host, self.port, self.path+'/'+path) + return ReverseProxyResource(self.host, self.port, self.getPath(request)+'/'+path, False) def render(self, request): request.received_headers['host'] = self.host request.content.seek(0, 0) + + path = self.getPath(request) + qs = urlparse.urlparse(request.uri)[4] if qs: - rest = self.path + '?' + qs + rest = path + '?' + qs else: - rest = self.path + rest = path clientFactory = ProxyClientFactory(request.method, rest, request.clientproto, request.getAllHeaders(),

1 0

twisted-web file upload processing on the server
by Matthew Thorley Dec. 13, 2005

Dec. 13, 2005

> IIRC, request.content should contain what you are looking for. Note that > twisted.web doesn't handle file uploads optimally; twisted.web2 does a > better job in that particular area. I gave that a shot earlier and didn't get much. print request.content <cStringIO.StringO object at 0xb7742ac0> print request.content.readline() [] print request.content.readlines() [] I have tried this with different files, and also double checked that the file I was uploading wasn't empty. -- Matthew Thorley

1 0