Mailman 3 December 2005 - Twisted

[Twisted-Python] [patch] issue 630: timezone in twisted log
by Alexey Shamrin 05 Dec '05

05 Dec '05

Hello! Nearly half a year ago I took issue 630 from twisted bug database [1] and tried to fix it. This bug is good to fix, because it makes the experience of looking at the console after issuing "twistd -noy some.tac" much better. But after adding several comments on the issue and attaching patches, I recieved ZERO response... :-( [1] http://twistedmatrix.com/bugs/issue630 -- Alexey

1 0

[Twisted-Python] Question on deferreds
by Frank Millman 04 Dec '05

04 Dec '05

Hi all, I am trying to convert my app to use Twisted. The original version handles a gui, some business logic, and a database connection, all in one program. I have split it into a 'client' program, that handles the gui; and a server program, that handles the business logic and the database connection, and communicates with the client using pb. The original program has some logic to validate user input. The data to be validated is passed up through 3 or 4 classes. Here is some very simplified pseudo code - class1(): def check1(data): perform test if test failed: return False if not class2.check2(data): return False return True class2(): def check2(data): perform test if test failed: return False if not class3.check3(data): return False return True class3(): def check3(data): perform test if test failed: return False if not class4.check4(data): return False return True class4(): def check4(data): perform test if test failed: return False return True In my new program, classes 1-3 run on the client, class4 runs on the server. Therefore I tried to change class3 as follows - class3(): def check3(data): avatar.callRemote(check4,data).addCallback(afterCheck) def afterCheck(result): return result This does not work, as check1 and check2 never receive the result. I would like to avoid going through my entire program and changing each occurrence of check1 and check2 into a deferred and a callback. I do not particularly mind if the client program blocks, as I do not want the user to proceed until the result is known. Is there any way to accomplish this? Thanks for any advice Frank Millman

1 0

[Twisted-Python] Request for ideas
by Ken Whitesell 04 Dec '05

04 Dec '05

Warning - fairly long post follows for what I hope is a simple question that only needs (possibly) a short answer. (I'm not looking for any code, just one of a couple possible ideas.) Environment: A LAN with a server and multiple PCs. Full control is available over all the PCs. I have the ability to run whatever I want on each - in this case it means Python 2.4 / Twisted 2. Background / Use Case: Imagine a simultaneous silent auction. (More like an RFP-type situation.) Any of the PCs can put an item up for bid. Other PCs can respond (or ignore!) that item. There are two timeout requirements - an "acknowledgement" timeout, where the PC has a certain period of time to respond "I want to make a bid", and a response timeout that restricts the clients to how much time they have to make the bid. These timeouts can change depending on the item going up for bid. (How and 'what' goes up for bid is irrelevant to my question.) What I've done so far: I've created a class called "CommTrans" to manage the auction. Each time an item goes up for bid, an instance of CommTrans is created. CommTrans creates two objects, one each to manage the acknowlegements and the other to manage the replies (AckMgr, RepMgr). The RepMgr is a Viewable that can be called by the client to submit its bid. (RepMgr is passed to the client in the callRemote call.) CommTrans issues the callRemote, passing RepMgr to the client. Then it sets a function in the AckMgr as the callback. The client receives the request, and decides whether or not to respond to the request. If yes, it issues a callLater on the function to determine the bid and submit it to the RepMgr. Either way, the original function returns either "Yes" or "No", triggering the callback on the CommTrans. Both the AckMgr and RepMgr issue callLater to schedule the timeout. If all possible machines reply before the timeout occurs, the timeout is cancelled. If the timeout occurs first, a flag is set to have the objects reject all further information from the clients. (It's simply ignored, nothing special occurs.) QUESTION #1: Did I miss something in the Twisted API that would handle this type of situation? I don't think a Deferred list would work in this situation, because it's not going to fire until all deferrs complete - something which might not occur. But this isn't the problem - the code that I have written for these functions all work. The specific problem occurs when objects are reused. I've had situations where a client fails to reply for significantly longer than the timeout period. In that interval, the original CommTrans, AckMgr and RepMgr have been deleted and replaced by new instances. The client finally replies to the original request, and the reply is caught by those new objects! (There is what I think is a very small, but non-zero probability that I'm misinterpreting what I'm seeing.) From what I can see in the source code, the internal ID is what's set by the builtin 'id' function. I'm printing these at key times and I can see where they're being duplicated. The only time I see this problem occur is when a new object has been created with the same id as a previous object. So this leads me to QUESTION #2: Anyone have ideas on how to resolve this? (Note: I can't channel everything through a single CommTrans object, because multiple bids might be happening concurrently with overlapping timeouts.) About the only thought I've had so far is to generate a GUID-like object for each CommTrans, and include that in every communication to and from the clients. The server would check each ack and each reply to ensure that the ack and reply are for the proper auction. Sending another message to the client isn't going to be a good solution, because the client might be tied up doing other things - starving the reactor - so by the time the message is processed, it may be too late. This also brings up QUESTION #3, although it hasn't happened yet: Server does a callRemote to the client from a commTrans object. Before the remote function returns (client is _extremely_ busy) the commTrans object goes away. What will happen when the function exits? (I know, I ought to generate a test-case for this - I'm more wondering if anyone knows right off-hand.) All thoughts, ideas, comments, recommendations are encouraged. Thanks to any/everyone who managed to get this far through my writing. Finally, I'm most pleased to announce that my proposal to provide a "Getting Started with Twisted" tutorial at PyCon has been accepted! I know I can't turn people into Twisted gurus in 3 hours, but I hope I can get a few over that initial hurdle. Thanks, Ken

2 1

[Twisted-Python] Twisted Words: throttling server output
by Seventh Holy Scripture 04 Dec '05

04 Dec '05

I've been writing an IRC bot using Twisted Words, and am having difficulty keeping it from flooding itself off the server. Currently my output subroutines look like this: def queueMsg(self, channel, msg): self.chanQueue.append(channel) self.msgQueue.append(msg) def emptyMsgQueue(self): for msg in self.msgQueue: channel = self.chanQueue.pop(0) self.msg(channel, msg) if channel[0] == "#": self.logger.log("<%s> %s" % (self.nickname, msg)) time.sleep(0.5) self.msgQueue = [] This ought to work as far as I know... but the bot waits until after all the sleep statements have executed to spew the output to the IRC server, which (if there's a lot of it) results in a forced disconnect due to "Excess Flood". What am I doing wrong here? My guess is that twisted has its own message queue, and waits until after all host program subroutines have ended to process it... I'm not good at python, though, so it could be a stupid mistake on my part. In any case, is there another way to introduce a delay between the messages sent to server that will work?

3 2

[Twisted-Python] Non-list expression received, 'Server' object has no attribute 'socket'
by Alex Couper 02 Dec '05

02 Dec '05

I have an application that requires each node using it to update each other, and to fetch updates automatically. Each node, therefore, has a server and a client on it. I am receiving different errors each time I run the program, and they are coming from within the twisted packages, (shown below) Error 1: File "C:\Python24\lib\site-packages\twisted\internet\tcp.py", line 89, in _closeSocket skt = self.socket exceptions.AttributeError: 'Server' object has no attribute 'socket' Error 2: File "C:\Python24\lib\site-packages\twisted\spread\pb.py", line 572, in expressionReceived raise ProtocolError("Non-list expression received.") twisted.spread.pb.ProtocolError: Non-list expression received. I suspect that they are caused by the way that I am running my server. On occasion, reactor.run() finishes, and thus there is no longer a server running in effect. I therefore use a while loop to always be running the server. The server is launched as a thread. def run(self): while 1: application = service.Application("copy_receiver") internet.TCPServer(8800, pb.PBServerFactory (self)).setServiceParent( service.IServiceCollection(application)) self.listener = reactor.listenTCP(8800, pb.PBServerFactory (self)) #installSignalHandlers=0 allows the reactor to run in a non-main thread reactor.run(installSignalHandlers=0) Is this known to cause problems? If not, any ideas of why i might be getting the error messages? or any ideas how I can keep the server running so that it doesn't exit? Alex Couper

2 1

[Twisted-Python] How can a tcp client connect with multi servers?
by Xu Ryan 02 Dec '05

02 Dec '05

Here is the situation: a tcp control client need connect to multi-agent. Is there a light way to connect with server , instead make a factory to do each connect?

2 18

[Twisted-Python] Help needed with SmartHostSMTPRelayingManager
by James Tanis 02 Dec '05

02 Dec '05

Right now I've got this SMTP server that I've slowly been piecing together using Twisted 2.1. I was able to get everything working as far as local delivery, but after adding a relaymanager.Queue the server was able to receive and queue messages from a mail client as well as deliver to local accounts. The problem comes in with relaymanager.SmartHostSMTPRelayingManager. Here is the log with traceback: 2005/12/01 21:24 EST [-] Log opened. 2005/12/01 21:24 EST [-] twistd 2.1.0 (/usr/local/bin/python 2.4.2) starting up 2005/12/01 21:24 EST [-] reactor class: twisted.internet.selectreactor.SelectReactor 2005/12/01 21:24 EST [-] Loading ./h2smtp.py... 2005/12/01 21:24 EST [-] Set 71757_1133487291.33_0_138982188 waiting 2005/12/01 21:24 EST [-] Set 71757_1133487291.33_0_138982188 waiting 2005/12/01 21:24 EST [-] /etc/resolv.conf changed, reparsing 2005/12/01 21:24 EST [-] Resolver added ('24.197.160.17', 53) to server list 2005/12/01 21:24 EST [-] Resolver added ('24.197.160.18', 53) to server list 2005/12/01 21:24 EST [-] twisted.names.dns.DNSDatagramProtocol starting on 62457 2005/12/01 21:24 EST [-] Loaded. 2005/12/01 21:24 EST [-] twisted.mail.protocols.ESMTPFactory starting on 2500 2005/12/01 21:24 EST [-] Starting factory <twisted.mail.protocols.ESMTPFactory instance at 0x85b04ec> 2005/12/01 21:24 EST [-] twisted.mail.protocols.POP3Factory starting on 1100 2005/12/01 21:24 EST [-] Starting factory <twisted.mail.protocols.POP3Factory instance at 0x85b052c> 2005/12/01 21:24 EST [twisted.names.dns.DNSDatagramProtocol (UDP)] Starting factory <twisted.mail.relaymanager.SMTPManagedRelayerFactory instance at 0x85fff0c> 2005/12/01 21:24 EST [Uninitialized] Traceback (most recent call last): File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 58, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 43, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/local/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/local/lib/python2.4/site-packages/twisted/internet/tcp.py", line 542, in doConnect self._connectDone() File "/usr/local/lib/python2.4/site-packages/twisted/internet/tcp.py", line 545, in _connectDone self.protocol = self.connector.buildProtocol(self.getPeer()) File "/usr/local/lib/python2.4/site-packages/twisted/internet/base.py", line 669, in buildProtocol return self.factory.buildProtocol(addr) File "/usr/local/lib/python2.4/site-packages/twisted/mail/relaymanager.py", line 103, in buildProtocol protocol = self.protocol(self.messages, self.manager, *self.pArgs, File "/usr/local/lib/python2.4/site-packages/twisted/mail/relaymanager.py", line 79, in __init__ relay.SMTPRelayer.__init__(self, messages, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/mail/relay.py", line 108, in __init__ smtp.SMTPClient.__init__(self, *args, **kw) exceptions.TypeError: __init__() takes at least 2 arguments (1 given) 2005/12/01 21:24 EST [Uninitialized] Backing off on delivery of ['71757_1133487291.33_0_138982188'] 2005/12/01 21:24 EST [Uninitialized] Stopping factory <twisted.mail.relaymanager.SMTPManagedRelayerFactory instance at 0x85fff0c> >From what I can tell smtp.SMTPClient isn't getting passed identity. Since this error is generated with a call to the relaying manager's .checkState(self) function which only takes self as a argument.. well lets just say I can't figure out what I'm missing from my own tracing through the code in the relay/relaymanager/smtp modules. You should find my code attached, the error should be easily reproducible if you want to run it. -- James Tanis jtanis(a)pycoder.org http://pycoder.org

1 1

[Twisted-Python] Need a SmartHost(E)SMTPRelayingManager guru
by James Tanis 02 Dec '05

02 Dec '05

Previously my only experience with Twisted was with 1.3 when I was developing a protocol from the ground up. Right now I've got this SMTP server that I've slowly been piecing together using Twisted 2.1. I was able to get everything working but after experiencing the learning curve firsthand I decided to buy the shiny new Twisted: Network Programming Essentials -- which is a great book I might add. Although the book did open my eyes to things I had missed as well as help me to improve my own code for local delivery it does not seem to cover (E)SMTP relaying which is a valuable component to me. Anyway, after adding a relaymanager.Queue the server was able to receive and queue messages from a mail client as well as deliver to local accounts. The problem comes in with relaymanager.SMTPRelayingManager (relaymanager.ESMTPRelayingManager gave me errors even earlier so I dropped it for the time being). Here is the log with traceback: 2005/12/01 21:24 EST [-] Log opened. 2005/12/01 21:24 EST [-] twistd 2.1.0 (/usr/local/bin/python 2.4.2) starting up 2005/12/01 21:24 EST [-] reactor class: twisted.internet.selectreactor.SelectReactor 2005/12/01 21:24 EST [-] Loading ./h2smtp.py... 2005/12/01 21:24 EST [-] Set 71757_1133487291.33_0_138982188 waiting 2005/12/01 21:24 EST [-] Set 71757_1133487291.33_0_138982188 waiting 2005/12/01 21:24 EST [-] /etc/resolv.conf changed, reparsing 2005/12/01 21:24 EST [-] Resolver added ('24.197.160.17', 53) to server list 2005/12/01 21:24 EST [-] Resolver added ('24.197.160.18', 53) to server list 2005/12/01 21:24 EST [-] twisted.names.dns.DNSDatagramProtocol starting on 62457 2005/12/01 21:24 EST [-] Loaded. 2005/12/01 21:24 EST [-] twisted.mail.protocols.ESMTPFactory starting on 2500 2005/12/01 21:24 EST [-] Starting factory <twisted.mail.protocols.ESMTPFactory instance at 0x85b04ec> 2005/12/01 21:24 EST [-] twisted.mail.protocols.POP3Factory starting on 1100 2005/12/01 21:24 EST [-] Starting factory <twisted.mail.protocols.POP3Factory instance at 0x85b052c> 2005/12/01 21:24 EST [twisted.names.dns.DNSDatagramProtocol (UDP)] Starting factory <twisted.mail.relaymanager.SMTPManagedRelayerFactory instance at 0x85fff0c> 2005/12/01 21:24 EST [Uninitialized] Traceback (most recent call last): File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 58, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 43, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/local/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/local/lib/python2.4/site-packages/twisted/internet/tcp.py", line 542, in doConnect self._connectDone() File "/usr/local/lib/python2.4/site-packages/twisted/internet/tcp.py", line 545, in _connectDone self.protocol = self.connector.buildProtocol(self.getPeer()) File "/usr/local/lib/python2.4/site-packages/twisted/internet/base.py", line 669, in buildProtocol return self.factory.buildProtocol(addr) File "/usr/local/lib/python2.4/site-packages/twisted/mail/relaymanager.py", line 103, in buildProtocol protocol = self.protocol(self.messages, self.manager, *self.pArgs, File "/usr/local/lib/python2.4/site-packages/twisted/mail/relaymanager.py", line 79, in __init__ relay.SMTPRelayer.__init__(self, messages, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/mail/relay.py", line 108, in __init__ smtp.SMTPClient.__init__(self, *args, **kw) exceptions.TypeError: __init__() takes at least 2 arguments (1 given) 2005/12/01 21:24 EST [Uninitialized] Backing off on delivery of ['71757_1133487291.33_0_138982188'] 2005/12/01 21:24 EST [Uninitialized] Stopping factory <twisted.mail.relaymanager.SMTPManagedRelayerFactory instance at 0x85fff0c> >From what I can tell smtp.SMTPClient isn't getting passed identity. Since this error is generated with a call to the relaying manager's .checkState(self) function which only takes self as a argument.. well lets just say I can't figure out what I'm missing from my own tracing through the code in the relay/relaymanager/smtp modules. You should find my code attached, the error should be easily reproducible if you want to run it. -- James Tanis jtanis(a)pycoder.org http://pycoder.org

1 0

[Twisted-Python] Re: [Twisted-commits] r15274 - Significantly improve twisted.python.filepath, adding several new features and tests.
by Jonathan Lange 02 Dec '05

02 Dec '05

On 02/12/05, Glyph Lefkowitz <glyph(a)wolfwood.twistedmatrix.com> wrote: > Author: glyph > Date: Thu Dec 1 14:47:17 2005 > New Revision: 15274 > > Modified: > trunk/twisted/python/filepath.py > trunk/twisted/test/test_paths.py > Log: > Significantly improve twisted.python.filepath, adding several new features and tests. <snip> > Modified: trunk/twisted/python/filepath.py > ============================================================================== > --- trunk/twisted/python/filepath.py (original) > +++ trunk/twisted/python/filepath.py Thu Dec 1 14:47:17 2005 > @@ -5,6 +5,10 @@ > from __future__ import generators > > import os > +import errno > +import base64 > +import sha > + > from os.path import isabs, exists, normpath, abspath, splitext > from os.path import basename, dirname > from os.path import join as joinpath > @@ -25,8 +29,13 @@ > class InsecurePath(Exception): > pass > > +def _secureEnoughString(): > + """ > + Create a pseudorandom, 16-character string for use in secure filenames. > + """ > + return base64.urlsafe_b64encode(sha.new(os.urandom(64)).digest())[:16] > + <snip> This breaks Twisted. base64.urlsafe_b64encode is not available in Python before version 2.4. cheers, jml

1 0

[Twisted-Python] Re: user interaction & the reactor
by Lenny G Arbage 02 Dec '05

02 Dec '05

Okay, I think I've got it now. This: http://twistedmatrix.com/projects/core/documentation/howto/threading.html combined with this: http://mail.python.org/pipermail/python-list/2004-February/209632.html contained some details I was woefully ignorant of. --- Lenny G Arbage <alengarbage(a)yahoo.com> wrote: > Where can I find examples or instructions on > combining > the reactor with [command-line] user interaction? > I've been searching every resource I can, and know > it > must be possible based on threads like this one: > http://twistedmatrix.com/pipermail/twisted-python/2005-June/010625.html > . I /can/ block on a call made from within the > reactor while waiting for user's raw_input, but that > doesn't allow me to give live feedback to the user > from incoming status messages, and might cause > problems if a large number of status messages arrive > while the user is 'away'. All the example clients I > can find inside of twisted are not [user-] > interactive > (ftpclient, echoclient, simpleclient, etc). Am I > looking in the wrong place? Have I missed a howto? > > Specifically I have a server that takes commands via > a > protocol I define and a client that sends commands > to > the server. The commands on the server take some > time > (on the order of 20 seconds to many minutes) to > complete or error out. Once they do, status is sent > back to the client, i.e., the user can issue many > commands that are concurrently 'pending', completing > when the server sends back status. > > I've started by running the client reactor in a > thread, and then using the main thread to get input > from the user. Am I making this more difficult than > it needs to be? Is there some way to integrate user > action (keyboard input) as events into the reactor's > event model? Or is there a better approach? __________________________________________ Yahoo! DSL Something to write home about. Just $16.99/mo. or less. dsl.yahoo.com

1 0