Mailman 3 October 2005 - Twisted

[Twisted-Python] Re: [Twisted-commits] r14614 - Describe log.debug, and 'printed' and 'debug' log dict items.
by Itamar Shtull-Trauring Oct. 5, 2005

Oct. 5, 2005

On Thu, 2005-09-29 at 23:17 -0600, Andrew Bennetts wrote: > Author: spiv > Date: Thu Sep 29 23:17:15 2005 > New Revision: 14614 > > Modified: > trunk/doc/core/howto/logging.xhtml > Log: > Describe log.debug, and 'printed' and 'debug' log dict items. log.debug should die, I think. So it probably shouldn't be documented :)

7 10

[Twisted-Python] Weekly Bug Summary
by James Y Knight Oct. 5, 2005

Oct. 5, 2005

Bug summary ______________________ Since 2005-09-28 06:00:02.620000: Bugs opened: 26 Bugs closed: 12 Total open bugs: 471 (+14) Urgent: 8 (+3) Bug: 224 (+12) Feature: 161 (-2) Wish: 78 (+1) New / Reopened Bugs ______________________ ==== Urgent ==== [CORE] whenThreaded no longer exists in SVN (yesterday) http://twistedmatrix.com/bugs/issue1242 opened by spiv Change cfreactor to use threadedselectreactor (yesterday) CLOSED http://twistedmatrix.com/bugs/… [View More]

1 0

[Twisted-Python] flow example doesn't appear to be 'flowing'
by Todd Greenwood Oct. 5, 2005

Oct. 5, 2005

First, thanks for creating this cool tool! I tried the following example (http://twistedmatrix.com/projects/flow/documentation/howto/flow.html ) ...and what I saw doesn't match the documentation. The docs state that if you launch two browsers pointed at this url, that both pages will be built simultaneously, due to the flow/generator constructs. However, when I tried this, I saw the web pages being generated sequentially. The second page blocked until the first page finished. Is this simply … [View More]my misunderstanding of the docs, or is this page outdated? REPRO: $ twistd --version twistd (the Twisted daemon) 2.0.1 Copyright (c) 2001-2004 Twisted Matrix Laboratories. $ python flow-demo.py visit http://localhost:8081/ to view the example #mozilla browser tab #1 => http://localhost:8081 30 Random numbers: 1. 46905 2. 17955 3. 62281 4. 99248 5. 73635 #mozilla browser tab #2 => http://localhost:8081 blank until tab #1 has finished (has 30 random numbers) DETAILS: http://twistedmatrix.com/projects/flow/documentation/howto/flow.html Flow Resources By using flow.Deferred it is easy to make up a web resource which is both long running, but also can serve more than one customer at a time. Run the example below, and with two browsers, view the webpage. Notice that both web pages are being created at the same time. from __future__ import generators from twisted.internet import reactor from twisted.web import server, resource from twisted.flow import flow def cooperative(count): """ simulate a cooperative resource, that not block """ from random import random idx = 0 while idx < count: val = random() yield flow.Cooperate(val) yield str(val)[-5:] idx += 1 def flowRender(req): count = int(req.args.get("count",["30"])[0]) req.write("<html><body>") req.write(" %s Random numbers: <ol>\n" % count) source = flow.wrap(cooperative(count)) yield source for itm in source: req.write("<li>%s</li>\n" % itm) yield source req.write("</ol></body></html>\n") class FlowResource(resource.Resource): def __init__(self, gen): resource.Resource.__init__(self) self.gen = gen def isLeaf(self): return true def render(self, req): self.d = flow.Deferred(self.gen(req)) self.d.addCallback(lambda _: req.finish()) return server.NOT_DONE_YET print "visit http://localhost:8081/ to view the example" root = FlowResource(flowRender) site = server.Site(root) reactor.listenTCP(8081,site) reactor.run() -- Todd Greenwood-Geer tgreenwoodgeer(a)yahoo.com ________________________________________________________________________________ BEAWorld 2005: coming to a city near you. Everything you need for SOA and enterprise infrastructure success. Register now at http://www.bea.com/4beaworld London 11-12 Oct| Paris13-14 Oct| Prague18-19 Oct |Tokyo 25-26 Oct| Beijing 7-8 Dec [View Less]

2 1

[Twisted-Python] Unicode
by Ken Kinder Oct. 4, 2005

Oct. 4, 2005

Perhaps like many developers, I came across this surprising bit of code inside a couple of Twisted's methods: if isinstance(data, unicode): # no, really, I mean it raise TypeError("Data must be not be unicode") And of course, I simply removed those lines. But I'm sure if I submit that patch, a discussion similar to this one would develop, because it's unlikely that such code would have been accidentally included: http://twistedmatrix.com/pipermail/twisted-python/2005-… [View More]

9 13

[Twisted-Python] conch.forwarding tutorial
by Yves Glodt Oct. 4, 2005

Oct. 4, 2005

Hello, can somebody point me at a conch.forwarding tutorial or code example? I need to create an ssh tunnel on the fly from my daemon and pass a database connection through that tunnel, is that possible at all? best regards, Yves

1 0

[Twisted-Python] [announce] tada-1.3.1 released
by clive＠darkarts.co.za Oct. 4, 2005

Oct. 4, 2005

Description: This python module is a wrapper module for any DBAPI 2 compatible database module. It wraps all functions and methods of the origional module and forces the return of deferreds instead. tada can therefore be considered to be 'almost' DBAPI compatible as it inherits the compatiblity level of the wrapped module, but returns twisted deferreds. tadah - is an additional class: "Tada with Helpers". This class adds some helper functionality to tada, but breaks strict DBAPI 2 spec. tao - … [View More]

2 2

[Twisted-Python] Sprinty Releases
by Christopher Armstrong Oct. 4, 2005

Oct. 4, 2005

On 10/3/05, Moof <moof(a)metamoof.net> wrote: > Right now I see five bugs tagged with "release management". I'll have a > proper look at them later and see what I can do. I suspect, though, that > much of it will require me to know the build system inside out. Is it > documented anywhre? Or who can I talk to about working out how it works and > what sort of times does he or she tend to be online? Hello, I am he-or-she. This weekend is the Virtual Twisted Sprint. To All: … [View More]

1 0

Re: [Twisted-Python] TCP KeepAlive
by Brian Warner Oct. 3, 2005

Oct. 3, 2005

> 1. When the TCP connection times-out, is pb.Copyable.stoppedObserving() > called? I think you mean pb.Cacheable.stoppedObserving, right? pb.Copyable is fire-and-forget. I see a call to stoppedObserving() inside Broker.connectionLost, so I'd hazard a guess and say yes, when the TCP connection is lost (which could be due to timeout, or the other end closing the connection, or the near end closing the connection), all the current Cacheables will have their stoppedObserving methods … [View More]invoked. Remember, however, that TCP timeouts are somewhat tricky, and rather "forgiving": they are intended to ignore "transient" network failures that only last a few minutes (once upon a time, when the internet was a slower place than it is now, and the phrase "five nines" referred to a dubious but still lucky poker hand, connection losses of several minutes at a time were no cause for alarm, and certainly not a reason to abandon your hard-won RFC88 Remote Job Entry Protocol session). The primary timer is a "short" exponential-backoff retransmission timer for data that has been sent but not yet acknowledged. It depends upon the kernel, but I tend to see this one give up on the connection after 5 to 15 minutes of non-connectivity. If you set the SO_KEEPALIVE option, a second timer is activated which basically pings the remote host every once in a while (although it does this within the context of the TCP session, so it will also detect a remote host that has shut down your connection but for which the FIN somehow went missing). This helps detect remote hosts that have powered off abruptly (so they weren't able to kill off processes and thus send FIN packets to terminate the TCP connections), and failures in intermediate routers (or, more commonly, a NAT box which has forgotten about the connection because it hasn't sent any traffic for 10 or 20 minutes). If the connection is busy, that is if each end sends some data to the other every couple of minutes, then the normal retransmission timer will catch connectivity losses after 5 to 15 minutes. The default interval for the keepalive timer tends to be one the order of 2 hours, plus it tries several times if it can't get through, which adds another 10 or 15 minutes. So even with SO_KEEPALIVE turned on, it may be hours before an otherwise-idle connection is detected as being broken. This drove me crazy on the buildbot, because many of the buildslaves are behind NAT boxes with short (20 minute) connection timeouts, and because the buildslaves can be idle for days at a time (waiting for someone to make an SVN checkin). So I added some application-level keepalives. The actual scheme I used is kind of weird, I'm not sure I would recommend it for new code, but the basic idea was to simply add a 'def remote_doNothing(self): pass' method to a pb.Referenceable at the far end, set up a timer to invoke target.callRemote("doNothing") once every 10 minutes, and then let the TCP retransmission timer take care of the rest. On top of that, you could reduce the extra traffic by keeping track of your normal callRemote invocations and setting an .activity flag, and then only send the remote_doNothing if there hadn't been any normal activity in the last 10 minutes. You could also detect connectivity losses faster by starting a shorter (perhaps 3 minute) timer when you send the doNothing, and if it doesn't complete (i.e. its Deferred doesn't fire) before that timer expires, abandon the connection with target.broker.transport.loseConnection() . If you choose this short timeout, remember that it could be triggered accidentally if you have a method which pushes a large amount of data over a slow (dialup) connection: this bit us several times when the buildslave was dumping _trial_temp/test.log up to the master, because it filled the Broker's transmit buffer with data that took 5 to 10 minutes to send, and the app-level keepalive message got queued after that, so the response couldn't possibly get back in time. Anyways, probably a larger answer than you really wanted :). Hope you (or someone else, some day) finds it useful. cheers, -Brian [View Less]

3 2

[Twisted-Python] threadedselctreactor and releases
by Moof Oct. 3, 2005

Oct. 3, 2005

Warning: mild rant venting frustration from a user, here. I'm not trying to offend, but I may have failed in this respect. Forgive me if I've failed in this aim. I've seen it mentioned quite often on the IRC channel recently, and it's starting to get a bit silly now, I'm talking, in this case, about people saying "you want to use wx? use threadedselectreactor". I'd like to remind people that threadedselectreactor isn't officially released, and people who want to release software that depends *… [View More]on* twisted prefer to work with full twisted releases, things that non-expert users can *download* without having to go through the rigmarole of installing subversion, and, in the case of windows users, learn to use the command line or some obscure right-click menu. Telling people "use svn HEAD" is, quite often, an answer that is useless for them. So, I have three suggestions to offer. I'm writing from the point of view of a user who doesn't know twisted inside out but does depend on it, and has never used wxpython: 1) Support the released versions. That's sort of the point of making a release, no? Rather than tell people "use threadedselectreactor and break any hope of a release policy you might have" try and work out how, in this case, people can get wxreactor working properly. This is, I gather, a herculean effort in most cases, so I understand people's reticence towards this course of action. 2) Make a new release of twisted.core. I gather this is unlikely to happen in the near future, especially given the amoutn of work happening in trial at the moment, so probably not the world's best idea. Plus, I'll get radix moaning at me for even suggesting it ;) 3) Given the amount of people who still seem to wish to use wx with twisted, and the frequency this thing comes up on dicussions on IRC, it's probably worth going for this one: Backport threadedselectreactor to twisted.core 2.0 (probably not too onerous) and either make a maintenance release of twisted.core with it in, or offer it for download as a separate package on the main twisted page. 4) Do nothing, and leave people like me quite frustrated. In the middle two cases, threadedselectreactor will need documenting on the twisted site. There's plenty of incidental documentation out there, it just needs someone to gather it up and put it together in the howto directly. This isn't just about threadedselectreactor, it just seems to be the most obvious problem. I first came across the "use svn HEAD" answer when I came across a bug that had been fixed in trial. It's great, it works lovely now, but the software environment I'm now developing in is not going to be the one my users would eventually be downloading form the twisted webpage. Thankfully, my users live in a controlled corporate environment, and, more to the point, won't be runnning the test suite, so I have the possibiliy of testing out my software with twisted release before sending it out for release, but I'd hate to have to release a wx-dependent piece of software here, something that is looking increasingly likely in the next three months or so. Yes, I am aware that most of you are volunteers. It's a volunteer-run and volunteer-supported project. You guys are doign great work developing the project, and I'm not tryign to denigrate that effort, merely make you aware of a particular frustration from the point of view of a user of your fabulous product, and one which probably contributes to people's diffidence towards taking the project up. Anyway, thanks for letting me vent on that subject for a bit. Moof -- Giles Antonio Radford, alias Moof "Too old to be a chicken and too young to be a dirty old man" Serving up my ego over at <http://metamoof.net/> [View Less]

5 8

[Twisted-Python] passClient for ReverseProxyResource/VHostMonsterResource
by Andrea Arcangeli Oct. 2, 2005

Oct. 2, 2005

Hello, I need a new (optional) feature in the ReverseProxyResource/VHostMonsterResource protocol. It's optional because I doubt the reverse proxy of other packages could support it without upgrades. I only had to add "passClient = True" as parameter to ReverseProxyResource and VHostMonsterResource to enable it (two liner patch to my code). In short this passes the client ip and port from the reverse proxy server to the vmonster resource, this way the nevow server that receives the proxy … [View More]connection knows who has connected to the proxy and it can react accordingly. This is needed to really allow putting more nevow servers transparently behind the same port, one of my nevow apps really needed to know the ip of the connecting client. Cross posting to both lists because it has two different parts, the first is the Nevow part and the second is the Twisted part. This code is online right now and no problems so far (if you see any problem let me know ;). Here the Nevow part: Index: Nevow/nevow/vhost.py =================================================================== --- Nevow/nevow/vhost.py (revision 1788) +++ Nevow/nevow/vhost.py (working copy) @@ -134,13 +134,15 @@ *after* it returns the (resource,segments) tuple. """ implements(inevow.IResource) + + def __init__(self, vhost_segments): + self.prepath_segments = vhost_segments+1 # +1 is for /vhost + def locateChild(self, ctx, segments): request = inevow.IRequest(ctx) - request.prepath = request.prepath[3:] + request.prepath = request.prepath[self.prepath_segments:] return request.site.resource, segments -_prepathCleaner = _VHostMonsterResourcePrepathCleaner() - class VHostMonsterResource: """VHostMonster resource that helps to deploy a Nevow site behind a proxy. @@ -173,14 +175,27 @@ This also means your private server should serve the real content at /foo/bar, and not at the root of the tree. + + If passClient is set to True it expects the client to be passed by the + proxy (see ReverseProxyResource passClient parameter for details). When + passClient is True this page should not be reacheable directly from the + internet if logging the IP address and port securely is needed. """ implements(inevow.IResource) + vhost_segments = 2 + + def __init__(self, passClient = False): + self.passClient = passClient + if passClient: + self.vhost_segments = 3 + self._prepathCleaner = _VHostMonsterResourcePrepathCleaner(self.vhost_segments) + def locateChild(self, ctx, segments): request = inevow.IRequest(ctx) - if len(segments) < 2: + if len(segments) < self.vhost_segments: return rend.NotFound else: if segments[0] == 'http': @@ -193,13 +208,18 @@ port = int(port) else: host, port = segments[1], 80 - request.setHost(host, port) - prefixLen = len('/'+'/'.join(request.prepath)+'/'+'/'.join(segments[:2])) - request.path = '/'+'/'.join(segments[2:]) + if self.passClient: + if ':' not in segments[2]: + return rend.NotFound + host, port = segments[2].split(':') + request.setClient(host, port) + + prefixLen = len('/'+'/'.join(request.prepath)+'/'+'/'.join(segments[:self.vhost_segments])) + request.path = '/'+'/'.join(segments[self.vhost_segments:]) request.uri = request.uri[prefixLen:] - return _prepathCleaner, segments[2:] + return self._prepathCleaner, segments[self.vhost_segments:] compy.backwardsCompatImplements(VHostMonsterResource) Here the twisted part: Index: Twisted/twisted/web/http.py =================================================================== --- Twisted/twisted/web/http.py (revision 14534) +++ Twisted/twisted/web/http.py (working copy) @@ -876,6 +876,10 @@ self.received_headers["host"] = host self.host = address.IPv4Address("TCP", host, port) + def setClient(self, host, port): + """Same as setHost but for the client address""" + self.client = address.IPv4Address("TCP", host, port) + def getClientIP(self): if isinstance(self.client, address.IPv4Address): return self.client.host Index: Twisted/twisted/web/proxy.py =================================================================== --- Twisted/twisted/web/proxy.py (revision 14534) +++ Twisted/twisted/web/proxy.py (working copy) @@ -160,23 +160,33 @@ to a different server. """ - def __init__(self, host, port, path): + def __init__(self, host, port, path, passClient = False): resource.Resource.__init__(self) self.host = host self.port = port self.path = path + self.passClient = passClient + def getPath(self, request): + path = self.path + if self.passClient: + path += '/%s:%d' % (request.client.host, request.client.port) + return path + def getChild(self, path, request): - return ReverseProxyResource(self.host, self.port, self.path+'/'+path) + return ReverseProxyResource(self.host, self.port, self.getPath(request)+'/'+path, False) def render(self, request): request.received_headers['host'] = self.host request.content.seek(0, 0) + + path = self.getPath(request) + qs = urlparse.urlparse(request.uri)[4] if qs: - rest = self.path + '?' + qs + rest = path + '?' + qs else: - rest = self.path + rest = path clientFactory = ProxyClientFactory(request.method, rest, request.clientproto, request.getAllHeaders(), [View Less]

1 0