Mailman 3 March 2007 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz Aug. 5, 2021

Aug. 5, 2021

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] Get IP of connecting client in XMLRPC server
by Remy C. Cool April 28, 2013

April 28, 2013

Hello, On 18 May 2004, the following message was posted to this mailinglist: Jp Calderone exarkun at divmod.com wrote: >Daniel Newton wrote: > > I have a simple XML-PRC server similar to the example below: > > from twisted.web import xmlrpc, server > > class Example(xmlrpc.XMLRPC): > """An example object to be published.""" > > def xmlrpc_add(self, a, b): > """Return sum of arguments.""" > return a + b def xmlrpc_whatIsMyAddress(self): return self.transport.getPeer().host > > if __name__ == '__main__': > from twisted.internet import reactor > r = Example() > reactor.listenTCP(7080, server.Site(r)) > reactor.run() > > I want to be able to get the address of the client that calls the xmlrpc > method can anyone help me with this? This solution didn't work because 'transport' isn't a property of the Example class. I'm currently in the process of changing from a customized SimpleXMLRPCServer to a twisted XMLRPC server solution and I need to insert the client IP into the attributes passed to the called xmlrpc method. Anyone who knows the answer and is willing to share the info? Regards, Remy

3 3

[Twisted-Python] Reconnecting PB Client
by Don Smith Sept. 12, 2007

Sept. 12, 2007

Does anyone have some example code for building a reconnecting PB client? This is what I have now: from twisted.spread import pb from twisted.internet import reactor from twisted.python import util events = pb.PBClientFactory() reactor.connectTCP("localhost", 8789, events) queue = events.getRootObject()

3 2

[Twisted-Python] twisted pysnmp status
by Brendan Simon April 18, 2007

April 18, 2007

Hi, I just wanted to know what the latest status is with twisted and pysnmp. Is pysnmp 4.x supported? Is full SNMPv3 supported? What is the roadmap and/or plans for twisted and pysnmp ??? I want to write and SNMPv3 GUI manager in python :) I'm not sure if the complexity and dependency of twisted is worth the effort? Then again on the other hand it may provide lots of things I would probably need to write anyway ? Given I would also like SSH and maybe even WBEM and/or WS-Management protocols too, then maybe twisted provides a better framework ??? Cheers, Brendan.

4 7

[Twisted-Python] exceptions.TypeError: ('Could not adapt', <twisted.spread.pb._PortalAuthChallenger instance at 0x013E3760>, <InterfaceClass twisted.cred.credentials.IUsernamePassword>)
by Justin Johnson March 30, 2007

March 30, 2007

Hi, I'm running Python 2.5 and Twisted 2.5.0 on Windows 2003 Server. I have a web site (Twisted/Nevow) where users login with a username and password. I save the username and password once they login because the same username and password are used to connect to various Perspective Broker servers, and I don't want to make users reauthenticate for every server that is connected to behind the scenes. Since the password cannot be hashed when passed to PB I store it in plaintext. Up to this point I have been using a realm and file password db as listed at the end of this email. When I recently upgraded from Twisted 2.4.0 to 2.5.0 and from Python 2.4 to 2.5 I get the following error. 2007/03/28 10:23 -0500 [Broker,0,IP_ADDRESS] Peer will receive following PB traceback: 2007/03/28 10:23 -0500 [Broker,0,IP_ADDRESS] Unhandled Error Traceback (most recent call last): File "D:\Python25\lib\site-packages\twisted\spread\pb.py", line 847, in _recvMessage netResult = object.remoteMessageReceived(self, message, netArgs, netKw) File "D:\Python25\lib\site-packages\twisted\spread\flavors.py", line 119, in remoteMessageReceived state = method(*args, **kw) File "D:\Python25\lib\site-packages\twisted\spread\pb.py", line 1217, in remote_respond d = self.portalWrapper.portal.login(self, mind, IPerspective) File "D:\Python25\lib\site-packages\twisted\cred\portal.py", line 109, in login return maybeDeferred(c.requestAvatarId, credentials --- <exception caught here> --- File "D:\Python25\lib\site-packages\twisted\internet\defer.py", line 107, in maybeDeferred result = f(*args, **kw) File "d:\python25\lib\site-packages\ratcontrol\login.py", line 195, in requestAvatarId up = credentials.IUsernamePassword(c) exceptions.TypeError: ('Could not adapt', <twisted.spread.pb._PortalAuthChallenger instance at 0x013E3760>, <InterfaceClass twisted.cred.credentials.IUsernamePassword>) 2007/03/28 10:23 -0500 [Broker,client] Unhandled error in Deferred: 2007/03/28 10:23 -0500 [Broker,client] Unhandled Error Traceback from remote host -- Traceback unavailable I looked at the Subversion log for twisted.spread.pb.py and am not sure what change is causing this failure. The contents of ratcontrol\login.py are below. Does anyone have any pointers on what I should be looking for here? Is there any way I can accomplish my goal in a more maintainable way? Thanks, Justin ---------------- from twisted.cred import portal, checkers, credentials, error from twisted.web import resource from twisted.spread import pb from twisted.python import log, components from twisted.internet.defer import maybeDeferred from twisted.internet import defer from twisted.python import failure, reflect, components from zope.interface import implements from nevow import inevow from ratcontrol.web import pages from ratcontrol import perspectives from ratcontrol import services def noLogout(): return None class RatControlRealm: """A simple implementor of cred's IRealm. For web, this gives us the LoggedIn page. For PB, this gives us a reference to the perspective. """ implements(portal.IRealm) def __init__(self): self.clients = [] self.serviceTypes = [] def addServiceType(self, serviceType): self.serviceTypes.append(serviceType) def requestAvatar(self, avatarIdAndPassword, mind, *interfaces): ## If the login is anonymous, only one item will be passed. ## If it is an authorized login, both the id and password ## will be passed. if len(avatarIdAndPassword) == 2: avatarId, avatarPassword = avatarIdAndPassword else: avatarId = avatarIdAndPassword for iface in interfaces: if iface is inevow.IResource: if avatarId is checkers.ANONYMOUS: resc = pages.NotLoggedIn() return (resource.IResource, resc, noLogout) else: resc = pages.LoggedIn(avatarId, avatarPassword, self) return resource.IResource, resc, noLogout elif iface is pb.IPerspective: if avatarId is checkers.ANONYMOUS: return (pb.IPerspective, None, None) else: log.msg("User %s logged in" % avatarId) #s = services.RatControlService() s = services.getServiceForTypes(self.serviceTypes) self.p = perspectives.IPerspectiveRatControl(s) self.p.loggedIn = 1 return (pb.IPerspective, self.p, self.p.logout) raise NotImplementedError("Can't support that interface.") class FilePasswordDB: """A file-based, text-based username/password database. Records in the datafile for this class are delimited by a particular string. The username appears in a fixed field of the columns delimited by this string, as does the password. Both fields are specifiable. If the passwords are not stored plaintext, a hash function must be supplied to convert plaintext passwords to the form stored on disk and this CredentialsChecker will only be able to check IUsernamePassword credentials. If the passwords are stored plaintext, IUsernameHashedPassword credentials will be checkable as well. """ implements(checkers.ICredentialsChecker) def __init__(self, filename, delim=':', usernameField=0, passwordField=1, caseSensitive=True, hash=None): """ @type filename: C{str} @param filename: The name of the file from which to read username and password information. @type delim: C{str} @param delim: The field delimiter used in the file. @type usernameField: C{int} @param usernameField: The index of the username after splitting a line on the delimiter. @type caseSensitive: C{bool} @param caseSensitive: If true, consider the case of the username when performing a lookup. Ignore it otherwise. @type passwordField: C{int} @param passwordField: The index of the password after splitting a line on the delimiter. @type hash: Three-argument callable. @param hash: A function used to transform the plaintext password received over the network to a format suitable for comparison against the version stored on disk. The arguments to the callable are the username, the network-supplied password, and the in-file version of the password. """ self.filename = filename self.delim = delim self.ufield = usernameField self.pfield = passwordField self.caseSensitive = caseSensitive self.hash = hash if self.hash is None: # The passwords are stored plaintext. We can support both # plaintext and hashed passwords received over the network. self.credentialInterfaces = ( credentials.IUsernamePassword, credentials.IUsernameHashedPassword ) else: # The passwords are hashed on disk. We can support only # plaintext passwords received over the network. self.credentialInterfaces = ( credentials.IUsernamePassword, ) def _cbPasswordMatch(self, matched, usernameAndPassword): ## Return both the username and password so we can save the password ## for connecting to other services once we are authenticated on one. if matched: return usernameAndPassword else: return failure.Failure(error.UnauthorizedLogin()) def getUser(self, username): try: f = file(self.filename) except: log.err() raise error.UnauthorizedLogin() else: if not self.caseSensitive: username = username.lower() for line in f: line = line.rstrip() parts = line.split(self.delim) if self.ufield >= len(parts) or self.pfield >= len(parts): continue if self.caseSensitive: if parts[self.ufield] != username: continue elif parts[self.ufield].lower() != username: continue return parts[self.ufield], parts[self.pfield] raise KeyError(username) def requestAvatarId(self, c): try: u, p = self.getUser(c.username) except KeyError: return failure.Failure(error.UnauthorizedLogin()) else: up = credentials.IUsernamePassword(c) if self.hash: if up is not None: h = self.hash(up.username, up.password, p) if h == p: return (u, p) return failure.Failure(error.UnauthorizedLogin()) else: ## Note that we pass both the username and the password return defer.maybeDeferred(c.checkPassword, p ).addCallback(self._cbPasswordMatch, (u,p))

2 2

[Twisted-Python] Why classes in twisted.protocols.basic use "static member" to store states?
by 甜瓜 March 30, 2007

March 30, 2007

Hi, I'm a newbie. After went through the source code in twisted.protocols.basic, I found it's really strange that all classes in twisted.protocols.basic use "static member" to store states. Take class StatefulStringProtocol as the example. Some Factory will creates a group of StatefulStringProtocol instances which share the static member "state". But why all instances MUST have the same state? In my opinion, instance A may be in the "handshaking" state, while instance B may be in the "data sending" state. The static member can not represent this case. Therefore, I think protocols should use "member variable" rather that "static member": class StatefulStringProtocol: def __init__(self): self.state = 'init' ... Could you plz give me some clue? I am a little confused. Most possibly, I misunderstanding the design about protocols-factory. Thank you. ShenLei

5 6

[Twisted-Python] Simultaneous Blocking Operations and Twisted
by Matthew Glubb March 30, 2007

March 30, 2007

Hi All, I am new to twisted and python. I am evaluating the possibility of using it for an application that I am writing. The framework looks very comprehensive (I am also enjoying the python language :) but I have some questions regarding deferred objects and the twisted web server itself. My application essentially comprises the following: 1. HTTP request is received by the server. 2. Server creates a listening socket on an arbitrary port and waits for n seconds for incoming data 3. Data (or timeout message) is returned to the client by way of the HTTP response. This is a relatively simple application and I have created a prototype in PHP with fastcgi. I am well aware of the limitations of PHP when it some to asynchronous events. The application needs to service thousands of clients simultaneously. In terms of CPU power, this is not a problem, the logic is simple and there is little overhead having thousands of listening sockets waiting for data (I'd prefer to use epoll but select or poll will do). However, there is a massive memory overhead because it is necessary to ensure that there is always one, idle process for any incoming request. Given that the request may wait for up to n seconds before returning a response, the number of simultaneous PHP fastcgi processes that are required is unreasonably high. I have toyed with the idea of writing a small (low memory) binary that will accommodate my needs, but my C isn't great, hence twisted. I was wondering whether it is feasible to use the twisted framework for this application. I have been playing with the framework and I can't see a way to service what are essentially thousands of requests that perform blocking operations simultaneously. I have noted the twisted documentation on threads but I was hoping, for the sake of my sanity, that I might be able to manage this using an asynchronous approach. Essentially, what I think I am looking for is a waitFor() on a deferred callback. As an aside, is it possible to compile platform specific python applications as small binaries? Or does the interpreter (is it required?) mean the memory consumption is prohibitive? Can anybody offer any advice on the matter? Thanks a lot :) Matthew Glubb

3 4

[Twisted-Python] anyway to know client address in XMLRPC?
by wang wei March 29, 2007

March 29, 2007

hello, I need to know the client address connected to my Twisted XMLRPC server, is there any way? the fellow code couldn't work in my machine. class Examle(xmlrpc.XMLRPC): """An example object to be published.""" def _getPeerAddress(self): return self.request.getClientIP() def render(self, request): self.request = request xmlrpc.XMLRPC.render(self, request) the result is said: "C:\Python24\lib\site-packages\twisted\web\server.py:272: UserWarning: Warning! r equest.finish called twice. http.Request.finish(self)" then how can I get client address in twisted?

2 2

[Twisted-Python] 2 threads questions
by Jonathan Vanasco March 28, 2007

March 28, 2007

i have a threaded daemon which occasionally hangs because of some stalling in the threadpool a- whats the best way to shut it down ? attempts to ctl-c / kill it seem to be interpreted by a random thread. the only way to kill it is a kill -9 the python threading docs said that was the expected behavior, and that it could be avoided by using signal. I figured twisted might have something in it already to handle shutting down a reactor with active threads. the threads docs for twisted are a little light. b- i'm trying to figure out the cause of the stalling in the threadpool from the status messages i'm printing, I'm seeing the main python code firing fine, but the threadpool is just stuck - nothing dispatched to it seems to clean up i wonder if it might be related to a few tickets i've seen http://twistedmatrix.com/trac/ticket/2448 http://trac.edgewall.org/ticket/3923 i'm trying to figure out a way to reliably test this. i've got a ssh manhole running via conch ( and instructions from the twisted book ) can anyone suggest a good way to test and try to debug what is going on? i'm extremely uninformed when it comes to python threads.

2 2

[Twisted-Python] problem with a tutorial
by Rob Smith March 28, 2007

March 28, 2007

I'm hoping this is just a simple configuration problem I have. I followed the tutorial for Using the Twisted Application Framework and created the .tac file shown below: from twisted.application import internet, service from qotdfactory import QOTDFactory port = 7001 factory = QOTDFactory() application = service.Application("qotd") echoService = internet.TCPServer(port, factory) echoService.setServiceParent(application) As soon as I run it with twistd on my winXP laptop it fails with an Import error "no module named qotdfactory". qotdfactory.py exists in the same directory as the .tac file. If I take the code in the .py file and put it directly in the .tac file, the program runs without trouble. I've tried adding the directory both files are in to the windows path variable, and tried adding a __init__.py file to the directory. Neither made a difference. Any thoughts on what I'm missing? Thanks, Rob Smith ____________________________________________________________________________________ Sucker-punch spam with award-winning protection. Try the free Yahoo! Mail Beta. http://advision.webevents.yahoo.com/mailbeta/features_spam.html

3 3

[Twisted-Python] Can I still run (most of) Twisted on Python 2.2?
by Erik Forsberg March 27, 2007

March 27, 2007

Hi! In TwistedCore/setup.py, I find that python 2.3 is required for Twisted. I'd like to use Twisted for a project that currently requires only Python 2.2, which raises the question - which parts of Twisted requires 2.3, and how much of it can I run on Python 2.2? On the same matter - are there any plans on when to stop supporting Python 2.3, Python 2.4, etc? Thanks, \EF -- Erik Forsberg OpenSource-based Thin Client Technology Systems Analyst/Developer Phone: +46-13-21 46 00 Cendio AB Web: http://www.cendio.com

6 7

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com March 25, 2007

March 25, 2007

Bug summary ______________________ Summary for 2007-03-18 through 2007-03-25 Bugs opened: 11 Bugs closed: 9 Total open bugs: 881 (+2) |== Type Changes |== Priority Changes |== Component Changes |Defect: -3 |Highest: -1 |Conch: +0 |Enhancement: +3 |High: -3 |Core: +2 |Task: +2 |Normal: +4 |Mail: -1 |Low: +2 |Pydoctor: +1 |Release Management: +0 |Web: -1 |Web2: +1 New / Reopened Bugs ______________________ ===== Highest ===== [#2528] t.conch.ssh.userauth.ssh_USERAUTH_REQUEST creates a Deferred and doesn't return it (opened by jml) (CLOSED, fixed) defect conch http://twistedmatrix.com/trac/ticket/2528 [#2530] Remove some of the warnings of process-docs (opened by therve) (CLOSED, fixed) defect core http://twistedmatrix.com/trac/ticket/2530 ===== Normal ===== [#2529] AMP streaming protocol extension (opened by glyph) enhancement core http://twistedmatrix.com/trac/ticket/2529 [#2531] twisted.conch.stdio.runWithProtocol fails in Eclipse/pyDev console (opened by neurophyre) (CLOSED, invalid) defect conch http://twistedmatrix.com/trac/ticket/2531 [#2532] How is a source listing generated (opened by duncanwebb) (CLOSED, fixed) task pydoctor http://twistedmatrix.com/trac/ticket/2532 [#2533] Pydoctor SVN and TortoiseSVN don't work (opened by duncanwebb) enhancement pydoctor http://twistedmatrix.com/trac/ticket/2533 [#2535] Allow Twisted to work without SIGCHLD on posix (opened by spiv) enhancement core http://twistedmatrix.com/trac/ticket/2535 [#2536] Deprecated TimeoutQueue so we can remove it (opened by exarkun) task core http://twistedmatrix.com/trac/ticket/2536 [#2538] twisted.web2.client.http has no unittests for pipelining support (opened by dreid) enhancement web2 http://twistedmatrix.com/trac/ticket/2538 ===== Low ===== [#2534] investigate event dispatch optimization (opened by exarkun) task core http://twistedmatrix.com/trac/ticket/2534 [#2537] doc/current/api source links point to SVN HEAD (opened by mwh) defect release management http://twistedmatrix.com/trac/ticket/2537 Closed Bugs ______________________ ===== Highest ===== [#2528] t.conch.ssh.userauth.ssh_USERAUTH_REQUEST creates a Deferred and doesn't return it (opened by jml, closed by jml, fixed) defect conch http://twistedmatrix.com/trac/ticket/2528 [#2530] Remove some of the warnings of process-docs (opened by therve, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/2530 [#2521] SMTP authentication tracebacks when receiving an empty line (opened by washort, closed by washort, fixed) defect mail http://twistedmatrix.com/trac/ticket/2521 ===== High ===== [#1272] selectreactor's addReader unexpectedly assumes a logPrefix attr (opened by marienz, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/1272 [#1366] TwistedQuotes documentation references external source files (opened by exarkun, closed by exarkun, duplicate) defect core http://twistedmatrix.com/trac/ticket/1366 [#927 ] Add description of Twisted Web to its project site (opened by hypatia, closed by oubiwann, fixed) defect web http://twistedmatrix.com/trac/ticket/927 ===== Normal ===== [#1126] Package Twisted using Eggs/EasyInstall (opened by glyph, closed by therve, duplicate) enhancement release management http://twistedmatrix.com/trac/ticket/1126 [#2531] twisted.conch.stdio.runWithProtocol fails in Eclipse/pyDev console (opened by neurophyre, closed by glyph, invalid) defect conch http://twistedmatrix.com/trac/ticket/2531 [#2532] How is a source listing generated (opened by duncanwebb, closed by duncanwebb, fixed) task pydoctor http://twistedmatrix.com/trac/ticket/2532

1 0

[Twisted-Python] thread pool question
by Rutt, Benjamin March 23, 2007

March 23, 2007

From http://twistedmatrix.com/projects/core/documentation/howto/threading.htm l: I copied the code below and ran it as: $ cat twisted-threadpool.py #!/usr/bin/env python from twisted.internet import reactor def aSillyBlockingMethod(x): import time time.sleep(2) print x # run method in thread reactor.callInThread(aSillyBlockingMethod, "2 seconds have passed") $ ./twisted-threadpool.py 2 seconds have passed But I note that the program never stops. I infer that least one thread is getting set as a non-daemon thread, according to python's thread rules on program termination: `thread.setDaemon(daemonic)' Set the thread's daemon flag to the Boolean value DAEMONIC. This must be called before `start()' is called. The initial value is inherited from the creating thread. The entire Python program exits when no active non-daemon threads are left. Is there a way to set this program up such that the program will exit when all of the threads registered via callInThread, and the main thread, have exited? Thanks, -- Benjamin Rutt

4 7

[Twisted-Python] Twisted friendly hosting services?
by Vincent Cate March 23, 2007

March 23, 2007

I am working on an application that runs using twisted.web. I want to host it on a machine with plenty of bandwidth as cheaply as possible. Is there a hosting service that uses Twisted.web so that I can just add my resource to the namespace? Or will I have to get a "virtual dedicated server" or a real "dedicated server"? If so, does anyone know a virtual one that uses Ubuntu? Thanks, -- Vince

4 3

[Twisted-Python] Using twisted.conch
by pico March 22, 2007

March 22, 2007

Hi all. I'm struggling a bit with twisted.conch. I've read the howto at http://twistedmatrix.com/projects/conch/documentation/howto/conch_client.ht…. I can understand this example fine. For a simple application, it works fine to subclass SSHChannel and implement all the code in that classes methods. However I would like to be able to implement something as follows: class SSHWrapper: def connect(self,hostname,username,password): #Create SshTransport object def runCommand(self,command) #Use SshChannel object to run command, return a deferred def disconnect(self): #Cleanup The problem is that, due to the ClientFactory creating instances of the relevant objects for you (SSHClientTransport,SSHUserAuthClient,etc), you are left without a reference to the SSHChannel object,required to run the command. Is there any way to get the SSHChannel object from the SSHClientTransport instance. Any advice with this matter would be appreciated. Thanks Pico

2 1

[Twisted-Python] Twistd memory usage
by Jarek Zgoda March 22, 2007

March 22, 2007

I have an application, ran as twistd plugin. If I run it as daemon, twistd consumes all available memory (of 4GB available), then after few seconds of swapping twistd is killed by kernel's OOM killer. I cann't reproduce this when I run this application in no-daemon mode (twistd -n), the memory footprint is constant and does not exceed 150MB. Is this normal behaviour? -- Jarek Zgoda "We read Knuth so you don't have to."

2 2

[Twisted-Python] hardcoded max in threadpool or connection pool ?
by Jonathan Vanasco March 21, 2007

March 21, 2007

forgive my ignorance this week - i'm rusty on twisted. I seem to be hitting a limit on threads / db connection ( not sure which one , as db could block connection ) at 11 The only code i have influencing the sizes are as follows: ======= class MY_dbConnectionPool( adbapi.ConnectionPool ): def __init__( self ,): adbapi.ConnectionPool.__init__( self, 'psycopg2' , host="localhost", database="xxxxx", user="xxxxx", password="xxxxx", cp_min= 16, cp_max= 16 ) ======= reactor.suggestThreadPoolSize( 17 ) reactor.run() ======= i never see more than 11 connections - in fact, i always see 11 connections -- 10 for the db the thread pool, and 1 for the db in the main i'm fairly sure if there's any error here it's going to be my fault -- can someone point me in the right direction ?

1 0

[Twisted-Python] suggestion for tracking down cause of 2 bugs sought: possibly threads / defer.DeferredSemaphore ?
by Jonathan Vanasco March 21, 2007

March 21, 2007

I've got 2 bugs in a twisted app that I'm trying to narrow in on. Overview of the app: A twisted service connects to a db and requests about 500 web pages to download and analyze. The service than uses defer.DeferredSemaphore service to handle dispatching each request to a thread via deferToThread . defer.DeferredSemaphore( tokens= MAX_DOWNLOADERS ) reactor.suggestThreadPoolSize( MAX_DOWNLOADERS + 1 ) Each thread runs a python script that is not written in Twisted and has a lot of blocking. And to pre-empt many comments that will inevitably pop up: It would be great to convert all the code to twisted , but 90% of the code + underlying modules are used by command line maintenance routines that were already in place. I just don't have the time or resources to rewrite it all in twisted. One day I'd love to have the funding to hire someone to manage them, but I just don't have that time myself. Overview of the bugs: 1) The query for figuring out which pages to download can take quite some time , so I run the getNextDownloads routine in a new thread , so the pool always remains full. It works perfect, except when the number of items queued up to defer.DeferredSemaphore reaches ~ 1000. After ~1000, I seem to only be running with 1-2 semaphores in place not 15 Question: Can anyone suggest a way to test this to see where the error is? Off the top of my head, the following seem plausible: a- hitting 1000 just breaks something in defer.DeferredSemaphore . the module is listed as unstable, so I'm guessing this could be the cause. b- the defer.DeferredSemaphore connection is just coincidental -- its really an issue with threads locking up c- something entirely different Unfortunately, I'm at a loss on trying to figure out whats going on. 2) I'm experiencing an obscene amount of process growth. I'm fairly certain that its from constant errors in urllib2 , which has been nothing of a nightmare with countless known bugs and patches ( downloading takes place via Mechanize , which is wrapped via urllib2 ), and makes me consider just wrapping wget or curl. but this could very well be threading related. Question: Is there any way to profile memory usage by twisted in a granular fashion ? I'd like to figure out where this memory usage is ( it grows steadily to ~1.7gb after 8 hours ). Thanks, // Jonathan Vanasco | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | SyndiClick.com | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | FindMeOn.com - The cure for Multiple Web Personality Disorder | Web Identity Management and 3D Social Networking | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | RoadSound.com - Tools For Bands, Stuff For Fans | Collaborative Online Management And Syndication Tools | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

2 2

[Twisted-Python] Twisted.Names assistance with auth and additional sections
by Sean Leach March 21, 2007

March 21, 2007

Hey all, I have a client I have written using twisted.names that will send a DNS request. Here is the gist of the relevant code: (tfp, tfile) = tempfile.mkstemp() os.write(tfp, 'nameserver %s' % ip) os.close(tfp) resolver = client.createResolver(resolvconf=tfile) deferred = resolver.lookupAddress(somevalue) deferred.addCallback(resultHandler) hacky way to set the IP, but I am having issues with passing in the "servers" list I don't want to troubleshoot until I figure out this problem. Anyways, as you can see, this code calls the lookupAddress method for the name I want. The problem I am having is it is only returning the answer section of the response. I need the additional and authority sections as well (and I know for sure those sections exist in the response). I have looked at the docs and the code, and I don't know what I am missing. I am sure it's something silly, but I am missing it right now. So long question short, what do I need to change in my code so I can get the additional sections and authority sections in the response so I can get to those values? Thanks, Sean

4 9

[Twisted-Python] question about thread and threadpool
by wang wei March 21, 2007

March 21, 2007

the document said: # run method in thread reactor.callInThread(aSillyBlockingMethod, "2 seconds have passed") but after above code, Is that Thread dead? and how can I re-use this thread? and if I create a thread using python thread module, can I use reactor to re-use it? there is a threadpool in twisted, is there any example of how to use it? thanks for any advices.

2 2

[Twisted-Python] using stdio with Twisted
by Neurophyre March 20, 2007

March 20, 2007

Hello, I'm building an old-school BBS client using Twisted. It needs to take user input, parse it, send messages over the wire and get messages back and display them.. the usual. I'm just now getting to writing the UI and have run into a couple problems. First, whether using protocol.Protocol or protocols.basic.LineReceiver (in raw mode) with internet.stdio.StandardIO, I have the same problem: the dataReceived (or rawDataReceived in the case of LineReceiver in raw mode) functions are only called after one types some text and hits return. I need to process input on a keystroke level, since there will be prompts at which a single keystroke will do things. Second, I was looking at twisted.conch.insults.insults.TerminalProtocol after some Googling -- does anyone know if I can just drop this in in place of protocols.basic.LineReceiver (or whatever) and have it work properly, or does it require some special invocation and/or grab stdio by itself? Thanks!

2 4

[Twisted-Python] Inheriting t.a.service.Service and super
by Sébastien LELONG March 20, 2007

March 20, 2007

Hi, I've been using Twisted for several months, and have frequently encountered the following problem. In my Twisted apps, I declare services (t.a.service.Service), which then I need to subclass. The problem is I can't call super() on such subclass, since the class Service in twisted does not inherit the built-in 'object' type ("TypeError: super() argument 1 must be type, not classobj"). For now, I can use other form of superclass calling, but that can be error prone. Is there any particular reason why class Service isn't a subclass of object ? It's kind of surprising since Service is designed to be subclassed. Thanks in advance for your help/replies. Cheers, Seb -- Sébastien LELONG sebastien.lelong[at]sirloon.net http://www.sirloon.net

2 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com March 18, 2007

March 18, 2007

Bug summary ______________________ Summary for 2007-03-11 through 2007-03-18 Bugs opened: 12 Bugs closed: 11 Total open bugs: 879 (+1) |== Type Changes |== Priority Changes |== Component Changes |Defect: +0 |Highest: -4 |Conch: +1 |Enhancement: +2 |High: +3 |Core: -4 |Normal: +2 |Lore: +3 |Lowest: +1 |Mail: +1 |Trial: +0 |Web: +1 |Web2: +1 |Words: -1 New / Reopened Bugs ______________________ ===== Highest ===== [#2521] SMTP authentication tracebacks when receiving an empty line (opened by washort) defect mail http://twistedmatrix.com/trac/ticket/2521 [#2523] portal can't check for inherited interfaces (opened by therve) (CLOSED, fixed) defect core http://twistedmatrix.com/trac/ticket/2523 [#2430] web.xmlrpc returns response with "text/html" content-type when xmlrpc method does not exist. (opened by Remi) (CLOSED, fixed) defect web http://twistedmatrix.com/trac/ticket/2430 ===== High ===== [#2525] Improve Lore's automated test coverage to UQDS standards (opened by glyph) enhancement lore http://twistedmatrix.com/trac/ticket/2525 [#2526] Integrate Lore patches which do index generation for ThinkCSPy (opened by glyph) enhancement lore http://twistedmatrix.com/trac/ticket/2526 [#2527] Lore Index output for LaTeX (opened by glyph) enhancement lore http://twistedmatrix.com/trac/ticket/2527 ===== Normal ===== [#2517] Xish XPath event observers not tested (opened by exarkun) defect words http://twistedmatrix.com/trac/ticket/2517 [#2518] t.w.xmlrpc.QueryProtocol does not close the connection after handling the request (opened by karlb) defect web http://twistedmatrix.com/trac/ticket/2518 [#2519] Couple of conch bug-fixes (opened by johnnypops) defect conch http://twistedmatrix.com/trac/ticket/2519 [#2520] twisted.mail.pop3client.POP3Client does not allow plain text login (opened by yasusii) (CLOSED, fixed) defect mail http://twistedmatrix.com/trac/ticket/2520 [#2522] HTTPAuthResource: 'ValueError: too many values to unpack' when IRealm.requestAvatarId returns a logout callable (opened by dreid) defect web2 http://twistedmatrix.com/trac/ticket/2522 ===== Lowest ===== [#2524] Support for triggering trial's waiting for Deferreds from within doctests (opened by strank) enhancement trial http://twistedmatrix.com/trac/ticket/2524 Closed Bugs ______________________ ===== Highest ===== [#1652] xish' EventDispatcher grinds to a halt when using many different one-time observers (opened by ralphm, closed by ralphm, fixed) defect words http://twistedmatrix.com/trac/ticket/1652 [#2476] local variable 'cbl' referenced before assignment (opened by jack, closed by ralphm, fixed) defect words http://twistedmatrix.com/trac/ticket/2476 [#2419] t.i.fdesc improvements (opened by itamarst, closed by therve, fixed) enhancement core http://twistedmatrix.com/trac/ticket/2419 [#2005] trial twisted/mail/ traces back, omitting trailing / does not. (opened by jerub, closed by therve, fixed) defect trial http://twistedmatrix.com/trac/ticket/2005 [#2430] web.xmlrpc returns response with "text/html" content-type when xmlrpc method does not exist. (opened by Remi, closed by therve, fixed) defect web http://twistedmatrix.com/trac/ticket/2430 [#2430] web.xmlrpc returns response with "text/html" content-type when xmlrpc method does not exist. (opened by Remi, closed by therve, fixed) defect web http://twistedmatrix.com/trac/ticket/2430 [#2523] portal can't check for inherited interfaces (opened by therve, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/2523 [#2463] add tests for logging of unhandled deferred errors (opened by antoine, closed by therve, fixed) enhancement core http://twistedmatrix.com/trac/ticket/2463 ===== Normal ===== [#2515] "python setup.py build" fails for Twisted (opened by edwards, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/2515 [#2520] twisted.mail.pop3client.POP3Client does not allow plain text login (opened by yasusii, closed by yasusii, fixed) defect mail http://twistedmatrix.com/trac/ticket/2520 [#1739] (patch) Removing system event triggers while firing does the wrong thing. (opened by antony, closed by glyph, duplicate) defect core http://twistedmatrix.com/trac/ticket/1739

1 0

[Twisted-Python] Game using twisted: Castle-Combat
by Karl Bartel March 17, 2007

March 17, 2007

Hi! I've written a game which could be added to http://twistedmatrix.com/ trac/wiki/ProjectsUsingTwisted, if it qualifies as successful. Name: Castle-Combat URL: http://linux-games.com/castle-combat Desc: a multiplayer game designed after the arcade game "Rampart" Thanks, Karl

2 1

[Twisted-Python] Foolscap-0.1.0 released
by Brian Warner March 16, 2007

March 16, 2007

I made a new release of Foolscap last night, available in the usual place at http://twistedmatrix.com/trac/wiki/FoolsCap . This release makes a few usability improvements for Foolscap-based servers, and fixes a bug that made it impossible to use a Tub right after creating it for the first time (if that system's clock was faster than the system being connected to). The Foolscap-0.1.0 release tarball is signed by my GPG public key (0x1514A7BD) and available from: http://twistedmatrix.com/~warner/Foolscap/foolscap-0.1.0.tar.gz http://twistedmatrix.com/~warner/Foolscap/foolscap-0.1.0.tar.gz.sig There are no protocol changes between 0.1.0 and the previous 0.0.7 release, just bugfixes and API additions. There are still protocol changes in the works, though. Why call this version 0.1.0? Because this release is actually kind of useful for something. What is Foolscap? From the README: This is a ground-up rewrite of Perspective Broker, which itself is Twisted's native RPC/RMI protocol (Remote Procedure Call / Remote Method Invocation). If you have control of both ends of the wire, and are thus not constrained to use some other protocol like HTTP/XMLRPC/CORBA/etc, you might consider using Foolscap. Fundamentally, Foolscap allows you to make a python object in one process available to code in other processes, which means you can invoke its methods remotely. This includes a data serialization layer to convey the object graphs for the arguments and the eventual response, and an object reference system to keep track of which objects you are connecting to. It uses a capability-based security model, such that once you create a non-public object, it is only accessible to clients to whom you've given the (unguessable) PB-URL. You can of course publish world-visible objects that have well-known PB-URLs. The complete release notes for 0.1.0 are attached below. share and enjoy, -Brian * Release 0.1.0 (15 Mar 2007) ** usability improvements *** Tubs now have a certFile= argument A certFile= argument has been added to the Tub constructor to allow the Tub to manage its own certificates. This argument provides a filename where the Tub should read or write its certificate. If the file exists, the Tub will read the certificate data from there. If not, the Tub will generate a new certificate and write it to the file. The idea is that you can point certFile= at a persistent location on disk, perhaps in the application's configuration or preferences subdirectory, and then not need to distinguish between the first time the Tub has been created and later invocations. This allows the Tub's identity (derived from the certificate) to remain stable from one invocation to the next. The related problem of how to make (unguessable) object names persistent from one program run to the next is still outstanding, but I expect to implement something similar in the future (some sort of file to which object names are written and read later). certFile= is meant to be used somewhat like this: where = os.path.expanduser("~/.myapp.cert") t = Tub(certFile=where) t.registerReference(obj) # ... *** All eventual-sends are retired on each reactor tick, not just one. Applications which make extensive use of the eventual-send operations (in foolscap.eventual) will probably run more smoothly now. In previous releases, the _SimpleCallQueue class would only execute a single eventual-send call per tick, then take care of all pending IO (and any pending timers) before servicing the next eventual-send. This could probably lead to starvation, as those eventual-sends might generate more work (and cause more network IO), which could cause the event queue to grow without bound. The new approach finishes as much eventual-send work as possible before accepting any IO. Any new eventual-sends which are queued during the current tick will be put off until the next tick, but everything which was queued before the current tick will be retired in the current tick. ** bug fixes *** Tub certificates can now be used the moment they are created In previous releases, Tubs were only willing to accept SSL certificates that created before the moment of checking. If two systems A and B had unsynchronized clocks, and a Foolscap-using application on A was run for the first time to connect to B (thus creating a new SSL certificate), system B might reject the certificate because it looks like it comes from the future. This problem is endemic in systems which attempt to use the passage of time as a form of revocation. For now at least, to resolve the practical problem of certificates generated on demand and used by systems with unsynchronized clocks, Foolscap does not use certificate lifetimes, and will ignore timestamps on the certificates it examines.

1 0