Mailman 3 September 2006 - Twisted

[Twisted-Python] 2 questions: adbapi.ConnectionPool ; Defer.DeferredSemaphore , threads.deferToThread
by Jonathan Vanasco 15 Sep '06

15 Sep '06

a- can someone clarify adbapi.ConnectionPool for me? i'm specifically wondering if connect/disconnect returns a handle to the pool and it the pool blocks until a handle is ready. i think it probably doesn't work that way. but i feel the need to check. b- has anyone ever used Defer.DeferredSemaphore with threads.deferToThread ? if so, how? all of my attempts thus far have not given the desired results specifically, i'm trying to have a scheduled job queue seed 1-50 'tasks' at once, each one using a seperate DB connection and running in a seperate thread. once they all complete, the job reschedules itself to run in the future. i'm probably approaching this problem wrong. so feel free to tell me i'm an idiot and point in a better direction. the closest thing I could think of was: 1- imports= [ self.semaphore.run( threads.deferToThread ( self.import_request, queue_item ) ) for queue_item in queue_import ] 2- imports= [ threads.deferToThread( self.semaphore.run ( self.import_request, queue_item ) ) for queue_item in queue_import ] neither of which work as: 1- gives me imports all at once, as deferToThread immediately returns a deferred object 2- self.import_request blocks , as its not in its own thread

3 3

[Twisted-Python] design pattern advice needed
by Jonathan Vanasco 14 Sep '06

14 Sep '06

I can't wrap my head around this FactoryA needs an object from FactoryB FactoryB has a finite pool of objects, which can be checked in and checked out I need FactoryA to not-progress until FactoryB is ready to dispense an object. in all of my attempts, i end up with a deferred object in the Factory A code. can anyone point me in the right direction?

2 1

[Twisted-Python] Twisted Development Procedures
by Jean-Paul Calderone 14 Sep '06

14 Sep '06

It's been pointed out to me lately that, while the general procedures in place for Twisted development are somewhat understood, specifics may be lacking. So, specifically, the development process Twisted has adopted is this: Commits to trunk/ must be the result of the review process outlined below. The only exception to this is the Twisted.Quotes file. There are no other exceptions. Changes to Twisted must be accompanied by a ticket in the tracker. Changes must be reviewed by a developer other than the author of the changes. If changes are paired on, a third party must review them. If changes constitute the work of several people who worked independently, a non-author must review them. A reviewer can be anyone who is familiar with the Twisted codebase, coding standard, and review requirements. A reviewer need not necessarily be familiar with the specific area of Twisted being changed, but he or she should feel confident in his or her abilities to spot problems in the change. A reviewer must reject a set of changes in any of the following circumstances: The test suite fails on any of the supported* platforms as a result of the changes. The changes alter the behavior of code in a way which is not tested. The changes add new behavior without adding test coverage for that behavior. The changes modify public APIs which are undocumented (ie, a function which has no docstring is modified, or a class which has no docstring has methods added to it). The changes modify the behavior of a documented API without updating the documentation. The changes violate the coding standard. A reviewer may reject a set of changes for other reasons, most of which are hard to quantify. Basically, use your best judgement, and don't be afraid to point out problems which don't fit into the above list. After a reviewer has given a change set the okay, the author (or another developer, if the author is so inclined or lacks commit privileges) may commit the change set to trunk. The trunk commit must include the name of the author or authors, reviewer or reviews, and the tickets which are being resolved by the changes being committed. The suggested format for this is: Author: <names> Reviewer: <names> Fixes #<ticket number> Only the "Fixes" line actually *needs* to conform to this (since it is automatically processed in order to update the ticket's state in the tracker). The commit message should also describe the change being made in a modest amount of detail. This should be detailed enough for someone doing a release to formulate a changelog entry. If a change set somehow introduces a test suite regression or is otherwise found to be undesirable, it is to be reverted. Any developer may revert a commit which introduces a test suite regression on a supported platform. If a branch is used, it should be deleted a short time after after it is merged into trunk (leaving enough time for the buildslaves to run to completion is a good idea, in case a regression has been introduced and the merge must be reverted). Chris has put this onto the wiki: http://twistedmatrix.com/trac/wiki/ReviewProcess Developers, please ask questions if any part of this remains unclear. Apologies for not posting this sooner, Jean-Paul

1 0

Re: [Twisted-Python] mechanize in twisted ?
by Jonathan Vanasco 14 Sep '06

14 Sep '06

On Sep 12, 2006, at 11:06 PM, L. Daniel Burr wrote: > My experience has been the same; I've used twisted.web.client for > years > now, in production applications, with no issues. That said, the > original > poster didn't mention any special requirements that forced him to > go with > mechanize; rather, the choice appears to have been made on the > basis of > activity. > > That said, there *is* a web2-based client, available from OSAF, named > zanshin. I haven't used it, but it is most certainly being actively > developed > > Sorry, I wrote this while rather tired. Let me clarify. I know that twisted.web 1.0 is stable and production proven. And I know people are happy with it and love it. Right now I have a bunch of existing scripts that crunch analytics and are already built on mechanize and run as cronjobs. I'm transitioning some to a job queue system , trying to optimize some resources by handling it all within twisted. My winwin situation on choosing mechanize is in part , because a) it involves copy-pasting a few lines and reusing classes I already have -- hopefully wrapping it into the event loop -- vs porting a bunch of stuff to twisted.web.client1 and b) its actively maintained, which means i get some random new features on their development list that I need already ( like integrated beautful soup support ). most that stuff is trivial and easily implemented in twisted, but i don't have to do that work. its done (in mechanize ). So I'd like to stay within mechanize. and I think i've got a way to wrap it into twisted. Thanks for the suggestions and trying to put me on the right track.

2 1

[Twisted-Python] Newbie Problem
by Damon Kohler 13 Sep '06

13 Sep '06

This is probably something simple. I'm having some trouble with this little bit of code below. I start it up once as a server, then a second time as the client connecting to localhost. However, it never actually seems to connect. Both sides just wait indefinitely. I'm a bit stuck as to how to debug this, so if someone has a solution for me and could tell me how I would have figured it out, that would be great! Thanks, Damon import sys import optparse import zope.interface from twisted.spread import pb from twisted.internet import reactor from twisted.internet import interfaces from twisted.python import util class ConsoleInput(object): zope.interface.implements(interfaces.IReadDescriptor) def fileno(self): return 0 def connectionLost(self, reason): print "Lost connection because %s" % reason def doRead(self): d = self.root.callRemote("echo", sys.stdin.readline().strip()) def setRoot(self, root): print "got root" self.myEchoer = Echoer() self.root = root self.root.callRemote('GiveReference', self.myEchoer) def logPrefix(self): return 'ConsoleInput' class Echoer(pb.Root): def remote_echo(self, st): print 'echoing:', st return st def remote_GiveReference(self, ref): self.client_ref = ref def main(): parser = optparse.OptionParser() parser.add_option('--host', dest='host', help='Host to connect to.') options, args = parser.parse_args() ci = ConsoleInput() reactor.addReader(ci) if options.host: host = options.host factory = pb.PBClientFactory() print "Connecting to %s..." % host reactor.connectTCP(host, 8789, pb.PBClientFactory()) d = factory.getRootObject() d.addCallback(ci.setRoot) d.addErrback(lambda reason: "error %s" % reason.value) d.addCallback(util.println) else: print "Listening..." root = Echoer() factory = pb.PBServerFactory(root) reactor.listenTCP(8789, factory) reactor.run() if __name__ == '__main__': main() __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 3

[Twisted-Python] twisted.web2 and blank values in POST
by Sylvain Thénault 13 Sep '06

13 Sep '06

Hi there, I've found a strange behaviour in twisted web2'server : the behaviour with blank values in form parameters is not consistent between GET and POST: on GET blank values are kept while this is not the case on POST. I'ld have expected blank values to be kept on POST as well. Is this a bug or an expected behavour ? PS: if this is a bug, an easy fix is to add keep_blank_values=True to the fileupload.parse_urlencoded call in server.py line 91. -- Sylvain Thénault LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Python et calcul scientifique: http://www.logilab.fr/science

2 2

[Twisted-Python] observer patter in twisted?
by wang wei 13 Sep '06

13 Sep '06

hi, I want implement such situation: a client detected a message change, then it notify all the servers that registed this message arrived. Is observer patter already implemented in Twisted? BTW, in Twisted, Is a serverFactory instance one thread? thanks.

4 3

[Twisted-Python] Re: [Twisted-commits] r18104 - Some systems raise an OSError with EPERM rather than an IOError with EACCES for filesystem permissions errors.
by Jean-Paul Calderone 12 Sep '06

12 Sep '06

On Tue, 12 Sep 2006 16:09:33 -0600, Wilfredo Sanchez <wsanchez(a)wolfwood.twistedmatrix.com> wrote: >Author: wsanchez >Date: Tue Sep 12 16:09:31 2006 >New Revision: 18104 > >Modified: > trunk/twisted/web2/dav/http.py >Log: >Some systems raise an OSError with EPERM rather than an IOError with EACCES for filesystem permissions errors. >Submitted by rfmohr(a)osc.edu >Reviewed by wsanchez >Fixes #1807 > > >Modified: trunk/twisted/web2/dav/http.py >============================================================================== >--- trunk/twisted/web2/dav/http.py (original) >+++ trunk/twisted/web2/dav/http.py Tue Sep 12 16:09:31 2006 >@@ -262,9 +262,9 @@ > if what is not None: > log.msg("%s while %s" % (err, what)) > >- if failure.check(IOError): >+ if failure.check(IOError, OSError): > e = failure.value[0] >- if e == errno.EACCES: >+ if e == errno.EACCES or e == errno.EPERM: > msg("Permission denied") > return responsecode.FORBIDDEN > elif e == errno.ENOSPC: > Tests? Jean-Paul

1 0

[Twisted-Python] HTTP 302 instead of just delivering the content.
by Markus Schiltknecht 12 Sep '06

12 Sep '06

Hi, I'm using twisted-web on port 8083 and apache as a reverse proxy on port 80. I can connect to my domain on port 80 and the HTTP request gets forwarded to twisted correctly. Twisted also replies and I get the website. But as soon as I click a link, the 8083 port appears in the URL. And according to tcpdump, it also got transfered via 8083. I figured Twisted replied with the following (my real domain replaced by some 'x'): HTTP/1.1 302 Found Date: Mon, 11 Sep 2006 12:22:57 GMT Server: TwistedWeb/2.4.0+r17646 Content-type: text/html; charset=UTF-8 Location: http://www.xxxxxxxx.ch:8083/firma/ Transfer-Encoding: chunked Why does it do that? The request sent was: GET /firma HTTP/1.1 Host: www.xxxxxxxxx.ch User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6 (Debian-1.5.dfsg+1.5.0.6-4) .... The apache2 reverse proxy configuration part is: RewriteRule ^(.*)$ http://localhost:8083$1 [P] ProxyRequests Off ProxyPreserveHost On ProxyVia Off <Proxy *> Order deny,allow Allow from all </Proxy> How can I tell twisted to not only accept requests on port 8083, but also serve those with Host: www.xxxxxxxxx.ch? Regards Markus

4 5

[Twisted-Python] how to scp with twisted?
by f rom 11 Sep '06

11 Sep '06

hi, can you point me to a sample or whatever on how to implement "scp" secure file copy with twisted for python on windows ? thanx Joe __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

3 3