Mailman 3 July 2005 - Twisted

[Twisted-Python] communication idioms with Perspective Broker
by Antony Kummel 26 Jul '05

26 Jul '05

Hello, I am using Twisted for a distributed application, and currently I am designing the basic objects of the application that will be transferred from process to process via PB. I ran into some conceptual problems, and I wonder if there are either general solutions (patterns, etc.) or Twisted-specific ones, or general ones already adapted to Twisted, etc. Problems: 1. Synchronization of events and state/data changes: I want clients to be able to receive notifications from a server when certain events occur. When such events are related to changes in state or data of the server that are also accessible to the user, I want the clients interface for retrieving the data and getting notified of the event to be coherent. Example of the problem: Lets say I have a server that can be in a state of being associated with a user, and that I want the server both to provide notifications to clients when it becomes associated or disassociated with a user, and to provide an isAssociated method that says if it is currently associated or not. If I provide the isAssociated functionality using a remote method call (with a referenceable), and the notification functionality also using a remote method call from the server to the client, then it is possible, for example, for a client to call twice the isAssociated method, and get a different result, without having received the notification (it may simply have not yet arrived), or also it is possible for the client to call isAssociated, receive a notification that the state has changed, and only then get the result from the deferred returned by isAssociated, which reflects the state prior to the change. The solution I came up with is to always expose associated events and data using a single cacheable object, so that when there is a change in state/data that should also make an event fire, the data change is propagated to the remote cache, and the remote cache fires an event locally (at the clients side). This way, the clients representation of the servers state is always coherent. 2. Synchronization of remote commands and remote events/data changes: I want clients to be able to issue commands to a server that make its state or associated data change, and also to have an up-to-date representation of the state or data (and possible events issued by changes in the state/data). The problem is how to synchronize the firing of the deferred returned by the remote method call with the change in the clients representation of the servers state/data (i.e. to make sure that when the deferred fires, the clients representation of the state is coherent with the command, i.e. changed). Presently I have no solution for this, but Im pretty sure that it requires some combination of referenceable and cacheable that will insure one coherent interface to associated data, events and commands. I am thinking to achieve this with a copyable that will contain both a referenceable and a cacheable. 3. Wrapping remote objects for non-online representation. I want to be able to have objects that can be used locally, but may also provide an interface to one or more servers that have to do with the state of the object. Another reason for these wrappers is that I want to be able to pass them freely from process to process, without being dependent on the connection or on a specific server. For example, there could be a User object that may have a username and password, etc. and also an event that fires when the user connects to the system, and a remote method for sending him messages. The username and password may be required by a process regardless of being connected to a server that can provide the other functions (for example, a process may want to display to a user a list of all of the users of the system, but may also not be connected to a presence server that provides the other functionality). I would say that what is needed is an anti-Avatar an object that represents some hyper-service entity that may be connected to servers that enable some of its functions and may also not. The question is, how to represent this to the user of the object. Whether to allow access to cached remote data and subscription to events when not online, whether to return a deferred and try to connect in case we dont have the data or throw an exception, etc. I suppose this is mostly a matter of style, but if anyone has done something like this before maybe they would have some insight Cheers, Antony Kummel __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 3

[Twisted-Python] Re: [Twisted-commits] r14219 - Add LowLevel HTTP Client code (based on the OSAF zanshin), with support for Basic and Digest authentication. Documentations and unittests to come.
by Jp Calderone 25 Jul '05

25 Jul '05

On Mon, 25 Jul 2005 14:02:51 -0600, David Reid <dreid(a)wolfwood.twistedmatrix.com> wrote: >Author: dreid >Date: Mon Jul 25 14:02:50 2005 >New Revision: 14219 > >Added: > trunk/twisted/web2/client/ > trunk/twisted/web2/client/__init__.py > trunk/twisted/web2/client/auth.py > trunk/twisted/web2/client/http.py >Modified: > trunk/twisted/web2/http_headers.py >Log: >Add LowLevel HTTP Client code (based on the OSAF zanshin), with support for Basic and Digest authentication. Documentations and unittests to come. I've reverted this, since it broke a couple of the existing web2 unit tests. Just as a reminder, current Twisted development policy dictates that: new feature development should occur in a branch, not in trunk; branches should be reviewed by at least one other core developer before being merged into trunk; and all new features should come with unit tests. If anyone needs any help figuring out the technical details (how to make or merge branches in svn, etc), #twisted and this mailing list are always around to help. Not following this policy (or introducing test suite regressions) makes it highly likely that the commit will be reverted. Thanks, Jp

1 0

[Twisted-Python] Defgen explanation
by Paolino 23 Jul '05

23 Jul '05

Hello all, I have the next running script which cause me some questions. I thought deferredGenerators tech could be easily reconverted to Deferreds ,but I'm having no success in doing it on the piece of code in Indexer.do method. Particularly if I substitute the generator with DeferredList([indexing() for indexing in indexings]).addCallback(self.do) I fall in recursion.But the rescheduling has to happen really at the end of the indexing. I think I miss something :( Thanks for help from twisted.internet import reactor from twisted.internet.defer import waitForDeferred as wait,deferredGenerator as async,Deferred,DeferredList import time def deferredReady(value=False,later=0): deferred=Deferred() reactor.callLater(later,deferred.callback,value) return deferred coop=lambda:wait(deferredReady()) class Indexer(object): def do(self,*_): indexings=self.indexings self.indexings=[] # a new queue ##### how to substitute this with non-generator deferring ? def _(): yield wait(DeferredList([indexing() for indexing in indexings])) deferredReady().addCallback(self.do) # rescheduling async(_)() ####### end of substitution def __init__(self): self.indexings=[] # a queue self.do() # first scheduling def index(self,uri):# the interface for triggers def _(): print 'indexing ',uri for i in range(10000): #adjust this higher to see the queue growing if your machine is faster yield coop() print 'indexed',uri self.indexings.append(async(_)) print 'Queued indexings',len(self.indexings) indexer=Indexer() from twisted.internet import task task.LoopingCall(lambda :indexer.index('ciao')).start(3) task.LoopingCall(lambda :indexer.index('mamma')).start(4) task.LoopingCall(lambda :indexer.index('papa')).start(5) reactor.run()

1 0

[Twisted-Python] listen to a process and react to events
by Daniel Isenegger 23 Jul '05

23 Jul '05

Hi, is it possible within the TwistedMatrix framework to listen to processes (e.g. running on local HDs) like you can listen to a port? For example, i have a GUI firing events. Now i want to listen from twistedmatrix to this process, catch the output of the process and process them. can anybody provide me a code example? A somehow similar example is quotes.py from the TwistedMatrix Homepage howto's (http://twistedmatrix.com/projects/core/documentation/howto/process.html). But (as i understand it) there the reactor listen to a port, not to the process fortune itself. Thanks dani

1 0

[Twisted-Python] PB: New credentials checker?
by Steve Freitas 23 Jul '05

23 Jul '05

Hi all, I'm trying to invent my own checkers, with associated credentials, and still use the pbClientFactory's login() function. I saw, however, that the source in pb.py notes that, for now, login() only works with IUsernamePassword: "Currently only credentials implementing IUsernamePassword are supported." (twisted/spread/pb.py, line 1611). So I started by subclassing the pbClientFactory, making my own login() function, and _cbSendUsername() and _cbResponse(), and then as I realized PB is using its own challenge response mechanism, it just started getting more complicated than I thought it probably should. So what's my best option -- continue on the path I'm on, or just serialize the data I need into the username and password fields in the standard checkers? Or is there an easier way I'm missing? Thanks! Steve

1 0

Re: [Twisted-Python] PB waitingForAnswers KeyError
by Bob Ippolito 23 Jul '05

23 Jul '05

On Jul 22, 2005, at 1:32 PM, William Waites wrote: > On Fri, Jul 22, 2005 at 09:19:50AM -1000, Bob Ippolito wrote: > >> >> I expect that you're using TwistedManager incorrectly. There doesn't >> appear to be anything wrong with the code you attached, but it also >> doesn't do anything on its own. >> > > I'm trying to put together a minimal example that is supposed to > work and > doesn't. > > How the bit driven by calls works; the embedded Python creates a > thread > and does something like: > > d = routeServer.callRemote("lookup", "somesource", "somedest") > rset = manager.getDeferred(d) That's your problem, you can't call ANY Twisted code from anything but the reactor thread (the one that called interleave), unless explicitly stated that it's allowed to do so (callFromThread, for example). You've misunderstood what the example is trying to demonstrate, it is not a magical "invert twisted" box. It demonstrates how you can create a simple ostensibly single-threaded application using Twisted that has a blocking interface on top of Deferreds. It's similar to how you would use any of the networking code in the standard library, only that there's asynchronous goodness somewhere behind the scenes making everything tick. -bob

3 4

[Twisted-Python] PB waitingForAnswers KeyError
by William Waites 22 Jul '05

22 Jul '05

This is with the Twisted Sumo tar, with threadedselectreactor dropped in: File "/usr/lib/python2.3/site-packages/twisted/internet/threadedselectreactor.py", line 252, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/lib/python2.3/site-packages/twisted/internet/tcp.py", line 351, in doRead return self.protocol.dataReceived(data) File "/usr/lib/python2.3/site-packages/twisted/spread/banana.py", line 184, in dataReceived gotItem(item) File "/usr/lib/python2.3/site-packages/twisted/spread/banana.py", line 114, in gotItem self.callExpressionReceived(item) File "/usr/lib/python2.3/site-packages/twisted/spread/banana.py", line 81, in callExpressionReceived self.expressionReceived(obj) File "/usr/lib/python2.3/site-packages/twisted/spread/pb.py", line 561, in expressionReceived method(*sexp[1:]) File "/usr/lib/python2.3/site-packages/twisted/spread/pb.py", line 919, in proto_answer d = self.waitingForAnswers[requestID] exceptions.KeyError: 19 What am I doing to make this happen? I am using a class derived from the threadedselect blocking demo. This implementation is attached, and has a blocking run() method and a blocking getDeferred() method. It takes interleaved' callbacks and puts them in a queue. The run() method takes them out of the queue in a blocking way and calls them. getDeferred() makes a threading.Event(), causes a CB to be queued, and then waits on the event to return the result. This way there can be many threads using the blocking getDeferred(), and many threads that just use the normal twisted stuff. This is running inside Asterisk and implements a PB client. This error appears under load (30-40 PB calls per second, 10-20 threads using getDeferred()). The server works just fine, and I have verified with a non-threadedselectreactor dummy client. A single threaded client is able to get several hundred PB calls per second through, and running more than one single threaded client works as expected. Looking at the spread/pb.py, I can't see how this KeyError should happen. waitingForAnswers[requestID] always gets set before the remote call, so I don't see a race condition where we could get a response before the dictionary is populated. Help? -w

2 1

[Twisted-Python] Weekly Bug Summary
by James Y Knight 21 Jul '05

21 Jul '05

Bug summary ______________________ Since 2005-07-12 14:40:23.400000: Bugs opened: 15 Bugs closed: 4 Total open bugs: 412 (+11) Urgent: 9 (+0) Bug: 179 (+8) Feature: 147 (+3) Wish: 77 (+0) New / Reopened Bugs ______________________ ==== Bug ==== [WEB] twisted.web.proxy doesn't reverse-map redirects like ProxyPassReverse (6 days ago) http://twistedmatrix.com/bugs/issue1109 opened by kragen OS-X vs PTYProcess, OSError vs IOError (yesterday) http://twistedmatrix.com/bugs/issue1120 opened by warner [FTP] FTPClient does not fire current commands errback on connectionLost (1 week ago) http://twistedmatrix.com/bugs/issue1107 opened by orbitz roundup exception when clicking "confirm password" link more than once (4 months ago) CLOSED http://twistedmatrix.com/bugs/issue919 opened by zooko [WEB] twisted.web.proxy.ReverseProxyResource incorrectly sends host header with no port (3 days ago) http://twistedmatrix.com/bugs/issue1117 opened by kragen [WIN32] win32 spawnProcess should return something (3 days ago) http://twistedmatrix.com/bugs/issue1116 opened by warner [CORE] intermittent test failure in test_policies (5 days ago) CLOSED http://twistedmatrix.com/bugs/issue1114 opened by warner [TRIAL] Trial fails with "signal number out of range" for signal 64 (6 days ago) CLOSED http://twistedmatrix.com/bugs/issue1110 opened by justinj [CORE] intermittent test failures in threads (5 days ago) http://twistedmatrix.com/bugs/issue1115 opened by warner [CONCH] intermittent test failures in conch (5 days ago) http://twistedmatrix.com/bugs/issue1111 opened by warner [CORE] intermittent test failure in testCallLater (5 days ago) http://twistedmatrix.com/bugs/issue1113 opened by warner [FLOW] intermittent test failure in flow: testThreadedImmediate (5 days ago) http://twistedmatrix.com/bugs/issue1112 opened by warner ==== Feature ==== [DOCUMENTATION] Document Services available in Twisted (2 days ago) http://twistedmatrix.com/bugs/issue1119 opened by spiv [DOCUMENTATION] Document how to write a Service correctly (2 days ago) http://twistedmatrix.com/bugs/issue1118 opened by spiv [WEB] web2 needs SOAP support (7 days ago) http://twistedmatrix.com/bugs/issue1108 opened by jblocksom Bugs Closed ______________________ ==== Bug ==== Stupid code needs removing from ftp.py (1 month ago) http://twistedmatrix.com/bugs/issue1075 opened by orbitz, closed by exarkun roundup exception when clicking "confirm password" link more than once (4 months ago) http://twistedmatrix.com/bugs/issue919 opened by zooko, closed by kragen [CORE] intermittent test failure in test_policies (5 days ago) http://twistedmatrix.com/bugs/issue1114 opened by warner, closed by exarkun [TRIAL] Trial fails with "signal number out of range" for signal 64 (6 days ago) http://twistedmatrix.com/bugs/issue1110 opened by justinj, closed by exarkun

1 0

[Twisted-Python] timers
by Andrzej Leszczynski 21 Jul '05

21 Jul '05

Hi folks, short question, I implement a very custom statefull protocol which need to have some timers in it. Can someone point me to examples how to do that? A.

4 7

[Twisted-Python] subclassing and class methods returning deferreds
by Thomas Vander Stichele 21 Jul '05

21 Jul '05

Hi, I wanted to have a class hierarchy where .stop methods would make sure everything is cleaned up properly before reactors get stopped and such. The obvious way to do that was to return a deferred that gets fired when everything else is done, and make sure that deferred is handled correctly everywhere. So I wrote up an example that does this: http://thomas.apestaart.org/download/tmp/subclass-deferred.py I'd appreciate it if someone who's interested would take a look at it to suggest some improvements, as I'm sure I might have overthought things a little, or maybe there are better ways to do something like this. Thanks, Thomas Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> Welcome to Hits City, Jeff K - Population: you <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/

2 2