I am using Twisted for a distributed application, and currently I am designing the basic objects of the application that will be transferred from process to process via PB. I ran into some conceptual problems, and I wonder if there are either general solutions (patterns, etc.) or Twisted-specific ones, or general ones already adapted to Twisted, etc.
1. Synchronization of events and state/data changes: I want clients to be able to receive notifications from a server when certain events occur. When such events are related to changes in state or data of the server that are also accessible to the user, I want the clients interface for retrieving the data and getting notified of the event to be coherent.
Example of the problem: Lets say I have a server that can be in a state of being associated with a user, and that I want the server both to provide notifications to clients when it becomes associated or disassociated with a user, and to provide an isAssociated method that says if it is currently associated or not. If I provide the isAssociated functionality using a remote method call (with a referenceable), and the notification functionality also using a remote method call from the server to the client, then it is possible, for example, for a client to call twice the isAssociated method, and get a different result, without having received the notification (it may simply have not yet arrived), or also it is possible for the client to call isAssociated, receive a notification that the state has changed, and only then get the result from the deferred returned by isAssociated, which reflects the state prior to the change.
The solution I came up with is to always expose associated events and data using a single cacheable object, so that when there is a change in state/data that should also make an event fire, the data change is propagated to the remote cache, and the remote cache fires an event locally (at the clients side). This way, the clients representation of the servers state is always coherent.
2. Synchronization of remote commands and remote events/data changes: I want clients to be able to issue commands to a server that make its state or associated data change, and also to have an up-to-date representation of the state or data (and possible events issued by changes in the state/data). The problem is how to synchronize the firing of the deferred returned by the remote method call with the change in the clients representation of the servers state/data (i.e. to make sure that when the deferred fires, the clients representation of the state is coherent with the command, i.e. changed).
Presently I have no solution for this, but Im pretty sure that it requires some combination of referenceable and cacheable that will insure one coherent interface to associated data, events and commands. I am thinking to achieve this with a copyable that will contain both a referenceable and a cacheable.
3. Wrapping remote objects for non-online representation. I want to be able to have objects that can be used locally, but may also provide an interface to one or more servers that have to do with the state of the object. Another reason for these wrappers is that I want to be able to pass them freely from process to process, without being dependent on the connection or on a specific server.
For example, there could be a User object that may have a username and password, etc. and also an event that fires when the user connects to the system, and a remote method for sending him messages. The username and password may be required by a process regardless of being connected to a server that can provide the other functions (for example, a process may want to display to a user a list of all of the users of the system, but may also not be connected to a presence server that provides the other functionality). I would say that what is needed is an anti-Avatar an object that represents some hyper-service entity that may be connected to servers that enable some of its functions and may also not.
The question is, how to represent this to the user of the object. Whether to allow access to cached remote data and subscription to events when not online, whether to return a deferred and try to connect in case we dont have the data or throw an exception, etc. I suppose this is mostly a matter of style, but if anyone has done something like this before maybe they would have some insight
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
On Mon, 25 Jul 2005 14:02:51 -0600, David Reid <dreid(a)wolfwood.twistedmatrix.com> wrote:
>Date: Mon Jul 25 14:02:50 2005
>New Revision: 14219
>Add LowLevel HTTP Client code (based on the OSAF zanshin), with support for Basic and Digest authentication. Documentations and unittests to come.
I've reverted this, since it broke a couple of the existing web2 unit tests.
Just as a reminder, current Twisted development policy dictates that: new feature development should occur in a branch, not in trunk; branches should be reviewed by at least one other core developer before being merged into trunk; and all new features should come with unit tests.
If anyone needs any help figuring out the technical details (how to make or merge branches in svn, etc), #twisted and this mailing list are always around to help.
Not following this policy (or introducing test suite regressions) makes it highly likely that the commit will be reverted.
I have the next running script which cause me some questions.
I thought deferredGenerators tech could be easily reconverted to
Deferreds ,but I'm having no success in doing it on the piece of code in
Particularly if I substitute the generator with
DeferredList([indexing() for indexing in
indexings]).addCallback(self.do) I fall in recursion.But the
rescheduling has to happen really at the end of the indexing.
I think I miss something :(
Thanks for help
from twisted.internet import reactor
from twisted.internet.defer import waitForDeferred as
wait,deferredGenerator as async,Deferred,DeferredList
self.indexings= # a new queue
##### how to substitute this with non-generator deferring ?
yield wait(DeferredList([indexing() for indexing in indexings]))
deferredReady().addCallback(self.do) # rescheduling
####### end of substitution
self.indexings= # a queue
self.do() # first scheduling
def index(self,uri):# the interface for triggers
print 'indexing ',uri
for i in range(10000): #adjust this higher to see the queue
growing if your machine is faster
print 'Queued indexings',len(self.indexings)
from twisted.internet import task
is it possible within the TwistedMatrix framework to listen to processes
(e.g. running on local HDs) like you can listen to a port?
For example, i have a GUI firing events. Now i want to listen from
twistedmatrix to this process, catch the output of the process and
can anybody provide me a code example?
A somehow similar example is quotes.py from the TwistedMatrix Homepage
But (as i understand it) there the reactor listen to a port, not to the
process fortune itself.
I'm trying to invent my own checkers, with associated credentials, and
still use the pbClientFactory's login() function. I saw, however, that
the source in pb.py notes that, for now, login() only works with
IUsernamePassword: "Currently only credentials implementing
IUsernamePassword are supported." (twisted/spread/pb.py, line 1611).
So I started by subclassing the pbClientFactory, making my own login()
function, and _cbSendUsername() and _cbResponse(), and then as I
realized PB is using its own challenge response mechanism, it just
started getting more complicated than I thought it probably should.
So what's my best option -- continue on the path I'm on, or just
serialize the data I need into the username and password fields in the
standard checkers? Or is there an easier way I'm missing?
On Jul 22, 2005, at 1:32 PM, William Waites wrote:
> On Fri, Jul 22, 2005 at 09:19:50AM -1000, Bob Ippolito wrote:
>> I expect that you're using TwistedManager incorrectly. There doesn't
>> appear to be anything wrong with the code you attached, but it also
>> doesn't do anything on its own.
> I'm trying to put together a minimal example that is supposed to
> work and
> How the bit driven by calls works; the embedded Python creates a
> and does something like:
> d = routeServer.callRemote("lookup", "somesource", "somedest")
> rset = manager.getDeferred(d)
That's your problem, you can't call ANY Twisted code from anything
but the reactor thread (the one that called interleave), unless
explicitly stated that it's allowed to do so (callFromThread, for
You've misunderstood what the example is trying to demonstrate, it is
not a magical "invert twisted" box. It demonstrates how you can
create a simple ostensibly single-threaded application using Twisted
that has a blocking interface on top of Deferreds. It's similar to
how you would use any of the networking code in the standard library,
only that there's asynchronous goodness somewhere behind the scenes
making everything tick.
This is with the Twisted Sumo tar, with threadedselectreactor dropped in:
File "/usr/lib/python2.3/site-packages/twisted/internet/threadedselectreactor.py", line 252, in _doReadOrWrite
why = getattr(selectable, method)()
File "/usr/lib/python2.3/site-packages/twisted/internet/tcp.py", line 351, in doRead
File "/usr/lib/python2.3/site-packages/twisted/spread/banana.py", line 184, in dataReceived
File "/usr/lib/python2.3/site-packages/twisted/spread/banana.py", line 114, in gotItem
File "/usr/lib/python2.3/site-packages/twisted/spread/banana.py", line 81, in callExpressionReceived
File "/usr/lib/python2.3/site-packages/twisted/spread/pb.py", line 561, in expressionReceived
File "/usr/lib/python2.3/site-packages/twisted/spread/pb.py", line 919, in proto_answer
d = self.waitingForAnswers[requestID]
What am I doing to make this happen? I am using a class derived from the threadedselect
blocking demo. This implementation is attached, and has a blocking run() method and
a blocking getDeferred() method. It takes interleaved' callbacks and puts them in a
queue. The run() method takes them out of the queue in a blocking way and calls them.
getDeferred() makes a threading.Event(), causes a CB to be queued, and then waits
on the event to return the result. This way there can be many threads using the blocking
getDeferred(), and many threads that just use the normal twisted stuff.
This is running inside Asterisk and implements a PB client. This error appears under
load (30-40 PB calls per second, 10-20 threads using getDeferred()).
The server works just fine, and I have verified with a non-threadedselectreactor dummy
client. A single threaded client is able to get several hundred PB calls per second
through, and running more than one single threaded client works as expected.
Looking at the spread/pb.py, I can't see how this KeyError should happen.
waitingForAnswers[requestID] always gets set before the remote call, so I don't see
a race condition where we could get a response before the dictionary is populated.
I wanted to have a class hierarchy where .stop methods would make sure
everything is cleaned up properly before reactors get stopped and such.
The obvious way to do that was to return a deferred that gets fired when
everything else is done, and make sure that deferred is handled
So I wrote up an example that does this:
I'd appreciate it if someone who's interested would take a look at it to
suggest some improvements, as I'm sure I might have overthought things a
little, or maybe there are better ways to do something like this.
Dave/Dina : future TV today ! - http://www.davedina.org/
<-*- thomas (dot) apestaart (dot) org -*->
Welcome to Hits City, Jeff K - Population: you
<-*- thomas (at) apestaart (dot) org -*->
URGent, best radio on the net - 24/7 ! - http://urgent.fm/