[Twisted-Python] communication idioms with Perspective Broker
Hello, I am using Twisted for a distributed application, and currently I am designing the basic objects of the application that will be transferred from process to process via PB. I ran into some conceptual problems, and I wonder if there are either general solutions (patterns, etc.) or Twisted-specific ones, or general ones already adapted to Twisted, etc. Problems: 1. Synchronization of events and state/data changes: I want clients to be able to receive notifications from a server when certain events occur. When such events are related to changes in state or data of the server that are also accessible to the user, I want the clients interface for retrieving the data and getting notified of the event to be coherent. Example of the problem: Lets say I have a server that can be in a state of being associated with a user, and that I want the server both to provide notifications to clients when it becomes associated or disassociated with a user, and to provide an isAssociated method that says if it is currently associated or not. If I provide the isAssociated functionality using a remote method call (with a referenceable), and the notification functionality also using a remote method call from the server to the client, then it is possible, for example, for a client to call twice the isAssociated method, and get a different result, without having received the notification (it may simply have not yet arrived), or also it is possible for the client to call isAssociated, receive a notification that the state has changed, and only then get the result from the deferred returned by isAssociated, which reflects the state prior to the change. The solution I came up with is to always expose associated events and data using a single cacheable object, so that when there is a change in state/data that should also make an event fire, the data change is propagated to the remote cache, and the remote cache fires an event locally (at the clients side). This way, the clients representation of the servers state is always coherent. 2. Synchronization of remote commands and remote events/data changes: I want clients to be able to issue commands to a server that make its state or associated data change, and also to have an up-to-date representation of the state or data (and possible events issued by changes in the state/data). The problem is how to synchronize the firing of the deferred returned by the remote method call with the change in the clients representation of the servers state/data (i.e. to make sure that when the deferred fires, the clients representation of the state is coherent with the command, i.e. changed). Presently I have no solution for this, but Im pretty sure that it requires some combination of referenceable and cacheable that will insure one coherent interface to associated data, events and commands. I am thinking to achieve this with a copyable that will contain both a referenceable and a cacheable. 3. Wrapping remote objects for non-online representation. I want to be able to have objects that can be used locally, but may also provide an interface to one or more servers that have to do with the state of the object. Another reason for these wrappers is that I want to be able to pass them freely from process to process, without being dependent on the connection or on a specific server. For example, there could be a User object that may have a username and password, etc. and also an event that fires when the user connects to the system, and a remote method for sending him messages. The username and password may be required by a process regardless of being connected to a server that can provide the other functions (for example, a process may want to display to a user a list of all of the users of the system, but may also not be connected to a presence server that provides the other functionality). I would say that what is needed is an anti-Avatar an object that represents some hyper-service entity that may be connected to servers that enable some of its functions and may also not. The question is, how to represent this to the user of the object. Whether to allow access to cached remote data and subscription to events when not online, whether to return a deferred and try to connect in case we dont have the data or throw an exception, etc. I suppose this is mostly a matter of style, but if anyone has done something like this before maybe they would have some insight Cheers, Antony Kummel __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Antony Kummel <antonykummel@yahoo.com> writes: As a general comment, I would suggest that you may wish to consider if what appears to be your goal of keeping asynchronous activities completely in sync is really a constraint you must meet? That is, your points seem concerned with ensuring that asynchronous actions or state changes somehow are reflected to users of that state in some atomic manner or on a transactional basis (no changes exposed until everything is complete). This is a non-trivial problem in a distributed form, particularly if you start to consider network failures that can interfere with communication mid-operation. It would appear you're looking for a ACID-like transactional layer. Instead of trying to meet such a tough constraint, it may be better to see if you can design your system to accept, from the beginning, that there will be a period of latency between actions taken in the system and resultant changes to state being apparent to users of that state. Or, that different parts of the system may be operating with older (but still valid) data at various points in time, and take more explicit control of when they update to the latest data. This has its own form of complexity, but can yield a much more resiliant system.
1. Synchronization of events and state/data changes: I want clients to be able to receive notifications from a server when certain events occur. When such events are related to changes in state or data of the server that are also accessible to the user, I want the client's interface for retrieving the data and getting notified of the event to be coherent.
(...)
The solution I came up with is to always expose associated events and data using a single cacheable object, so that when there is a change in state/data that should also make an event fire, the data change is propagated to the remote cache, and the remote cache fires an event locally (at the client's side). This way, the client's representation of the server's state is always coherent.
This seems reasonable to me, or at least as close as you can come to your requirement of keeping a remote client's data/events in sync. You would need to ensure that your state object managed changes to itself such that it only reflected changes down the PB channel to any cacheable observers at appropriate times when its own state was consistent. E.g., you wouldn't necessarily update on each field or attribute change. What you're effectively doing here is using the cacheable object as your transaction manager, controlling what clients see. Note that to ensure consistency between clients and server, you'd want to have everyone working from cacheables of this object and nobody (even on the server itself) working from the object itself. Otherwise, you'd need a mechanism (such as with point 2) You still have an issue of network outages preventing updates from making it to clients, but in that case you'd only have to deal with a disconnected client being out of date with respect to the server and other clients, but in a consistent way, and upon reconnecting it would receive a new, but again consistent, set of state information.
2. Synchronization of remote commands and remote events/data changes: I want clients to be able to issue commands to a server that make its state or associated data change, and also to have an up-to-date representation of the state or data (and possible events issued by changes in the state/data). The problem is how to synchronize the firing of the deferred returned by the remote method call with the change in the client's representation of the server's state/data (i.e. to make sure that when the deferred fires, the client's representation of the state is coherent with the command, i.e. changed).
(...)
Presently I have no solution for this, but I'm pretty sure that it requires some combination of referenceable and cacheable that will insure one coherent interface to associated data, events and commands. I am thinking to achieve this with a copyable that will contain both a referenceable and a cacheable.
I think you're going to have an uphill battle here to try to synchronize what is inherently distributed and asynchronous behavior. If I absolutely had to do this, it's probably along the lines of what your considering - I'd consider having a transaction object that encapsulated change requests and state updates under a single umbrella. Rather than a simple cacheable though, you'd probably need to implement a two-phase commit protocol so that the state only changed on both client/server or not at all. If you didn't do that, you'd leave yourself open to the operation occurring on the server, but the network preventing the new state information from getting down to the client, and the client not knowing what state the server was in. But I'd much rather just assume that the client needed to work properly with the state as it currently knew, and respond properly to any state updates as they occurred, whether due to its own operation or some other client's operation. E.g., formally decouple the request to perform an operation, from the state changes that would occur as that operation was performed. In other words, try to stick more to a model/controller approach, where the model state is monitored by clients and they simply react to its changes, but actions taken always flow through a distinct controller path that is associated with but decoupled from the model.
3. Wrapping remote objects for non-online representation. I want to be able to have objects that can be used locally, but may also provide an interface to one or more servers that have to do with the state of the object. Another reason for these wrappers is that I want to be able to pass them freely from process to process, without being dependent on the connection or on a specific server.
(...)
The question is, how to represent this to the user of the object. Whether to allow access to cached remote data and subscription to events when not online, whether to return a deferred and try to connect in case we don't have the data or throw an exception, etc. I suppose this is mostly a matter of style, but if anyone has done something like this before maybe they would have some insight
What we've done in a system of ours that is designed to be an distributed data system is treat the core data objects in the system as pure state objects as much as possible, generally falling into two classes: * Pure instance data (data objects) of which multiple copies may exist simultaneously throughout the system, but which operate locally (Copyable in twisted). The only methods such objects have are to manipulate the local representation (also handled by direct attribute access). * Shared state objects (model objects) of which multiple observers throughout the system may exist (Cacheable in twisted). While these objects may have similar method and attribute access as the above instance data, they are only ever manipulated by controller objects, for which the original instance is only the same "node" as the original instance of the state object. Any client needing to make changes must use a reference to that controller (a twisted Referenceable) and not the model itself, even on the local node where the model/controller are instantiated. The choice between the two object types is not a hard and fast rule - we've tended to use more of the former so far. We then constructed a framework of "manager" objects which are designed to provide access and manipulation of the above data objects. The key attribute of a manager object is that it is both referenceable (twisted Referenceable) and all of its methods are deferrable interfaces - even if used locally. Although I found I had a desire to try to somehow just always pass object references (referenceable) around to everything and let PB handle everything transparently - in practical terms I didn't find that workable. There are just too many nooks and crannies you can get into as things get distributed that taking some more explicit control became necessary to ensure robustness. And you just can't always assume that making changes to state on what is ostensibly a shared object will magically get reflected everywhere reliably. But that's actually where we found PB to be just at the right level as it was easy enough to wrap to behave how we wanted. We provided an extra layer of wrapping for the networking, both for the basic connection as well as our referenceables. A Client object encapsulates making a connection to a server, along with reconnecting as necessary and generating local signals (we use pydispatcher) on connection state changes. A matching Server object provides client access to local managers - through a simple Registry object - upon a client's connection. A general purpose wrapper object is used to wrap each manager referenceable so that they appear to be local (it uses the manager's interface definition to automatically translate method calls into callRemote), as well as to automatically re-establish contact with the remote manager if needed, by listening for the Client signals and on a reconnect, re-obtaining the remote Registry handle and refetching remote references to the manager it had previously wrapped. The wrapping may be multi-layer. This allows us for example, to have a remote site have a master server, which maintains the client link to a central server. That site server, therefore, has what it considers a local Registry with a whole set of local managers - all of which are technically wrappers around the twisted referenceable to the main central server. But then other machines at the site themselves become clients of the site server, with their own references to the site server's references. So when the other site machines make requests they flow to the site server, and then up to the central server and back down. But the same source code works whether running on the central server, or at any level of the site servers, without knowing the difference (since all registry and manager interfaces are deferrable anyway). While the wrapper isolates users from need to worry about reconnecting, we don't attempt to hide the fact that an outage is occurring. Attempts to make a call on a wrapped manager during an outage generates normal PB exceptions, with one change that we modified Twisted to always return exceptions up the deferred change (even for a dead reference) so clients wouldn't have to deal with both local exceptions and errbacks. In practice, the client applications generally have some application level object that is also listening to the Client objects connection signals, and either blocking access to the user when the network is (with an appropriate message), or adjusting behavior accordingly. So, in operation, client code works something like: * Instantiate a Client object, give it connection info and start the connection. Request the registry from the client object (which is deferrable and only fires once the overall connection cycle is complete). * Using the Registry object (which itself is a remote wrapper version on the client side), query references for any manager objects needed. * Using the manager objects, retrieve any data objects needed. Changes to model objects occur through their controllers, while changes to data objects are performed locally and changes updated via explicit "save" calls to the managers. The last point is where we run into similar issues as yourself, I think. By choosing this route we do not provide for other clients of that same data object to automatically see changes made by other clients. They would continue to run with the copy they had previously received, although any subsequent retrieval would get a new copy with the new data. To handle crossing state changes, the originator (actual manager object on whatever node it exists on) of the data object maintains an internal tag (we use a UUID but it could also be a hash of the contents) in the object representing its unique state and will raise a SaveConflict exception of our own if someone else attempts to store changes to an outdated copy. It is up to clients to handle such issues, should they occur (typically by requerying the information and then re-applying their changes), although in practice we really don't have scenarios where this happens yet due to typical usage patterns. Some of this could change if we moved a data object to a model object, but then we're requiring that even simple users of the data object maintain a remote cacheable reference to the object, which is relatively heavyweight. Thus my comment about it being a grey area above as to which sort of object we decide to place such state in. In our environment, our User object (which contains identifying and control information about users) is just a data object, as the need for simultaneous manipulation and monitoring of it is reasonably low. We do expect to have many copies of it around, but mostly on a read-only basis. In your context, I would think that the user object itself need not be something that constantly updates, but the state of which users were currently online would fit better as a model (and the controller to feed it would have methods for a given user to go online or offline). In our structure, we would separate out the concept of generating a system message - probably into a messaging manager - which would then receive requests to transmit messages to identified users. But I don't think I'd try to tie those three things (current User object contents, currently online user set, generating a message) into any sort of guaranteed state ... I'd leave them very loosely coupled. On the issue of distribute events, that's the area we're currently working on, and to us the hardest part is how to handle events that may be generated during outages for which the disconnected clients have subscriptions. If it's just for changes to state objects (such as cacheables) that's not so bad, since the reconnection process will re-query the current state information. But if it's for more general notifications (we might have our own bit for "user updated" like your "user came online") you have a question of how long do you queue up such events for clients that might never show up again. Currently we are targetting such events being handled by a signal or event manager, which will maintain an ongoing history of such events. Subscribers to the event manager will get copies of appropriate events. When a client connects it's local wrapper for the remote event manager will actually handle all local subscriptions, maintaining a single remote set of subscriptions to minimize network I/O. It will also track the delivery of any events. Upon being disconnected/reconnecting (per the standard mechanisms), the client event manager wrapper will request any signals that may have been generated since the last event seen prior to the disconnect. We'll have to bound this somehow for prolonged outages. But a key point is still decoupling the event handling from other operations, and we won't be trying to force everything to stay in sync with other clients and/or servers at all times. If you've put up with me until here, I hope that this at least gives you some other approaches to think about, even if some or all of it isn't directly applicable to your problem domain. -- David
Hi David, Thanks a lot for your response. It seems that your solution fulfills most of what I wanted to achieve, so it was a great help. This is how I understand the registry/manager/wrappers system: The meaning of the wrappers is that referenceables are transferable to third parties who get their flow pass through the middle process, and that the wrappers get reconnected automatically If I understand correctly, the only purpose of the registry is to provide an interface to enable the re-connection of the wrappers. The purpose of managers is simply to dispense data and state objects. Questions: Do managers only dispense state and data, or do they also provide state control? Do cacheables (state objects) re-connect? Is there any reason why they shouldnt? Regarding multi-layer wrapping, how do cacheables go from the original server to the final client without becoming unjellyable in the middle? Thanks, Antony Kummel P.S. There are some things I am considering to do differently, and it would be interesting to hear your opinion. The system I had in mind: I like and want to adhere to the data/state distinction you made. Events will be handled locally by remote caches, based on changes in the cached data (this may be accomplished degenerately, by not exposing anything other than the event). Differences from your system: I would like all of my referenceables and cacheables in my system to be re-connecting. This to some extent cancels the need for managers, because any dynamically changing object is re-connecting. Instead of (or possibly in addition to) manager objects, I want to have what I call Seed objects, which represent a combination of state, data and referenceables (all optional). These seeds will be copyable, and will include the knowledge required to retrieve their components. The seed objects will represent the basic objects of the system (like users), and will provide control and access to their entire data/state. The main reasons for seeds are: I want state, data and control to be provided by the same object for clarity, and not have each of them require an individual query. For example, a user will have Name, email address, etc. as data, online/offline as state, and a send_message method. I want state objects, referenceables, and possibly data associated with a Seed to be retrievable from a Server different from the one who dispensed the Seed (for example, the database may provide a seed and the associated user-changeable data, but the state may be kept by a different server). For example, the users data may be stored in a database, his online/offline state retrieved from a presence server, and sending him a message may require connecting to his workstation. --------------------------------- Start your day with Yahoo! - make it your home page
Antony Kummel <antonykummel@yahoo.com> writes:
This is how I understand the registry/manager/wrappers system:
The meaning of the wrappers is that referenceables are transferable to third parties who get their flow pass through the middle process, and that the wrappers get reconnected automatically
Close (IMO) - the referenceable to the original object (which is created as a result of passing that object through PB to a third party) is placed in a wrapper which is then given to client code. For example (in ASCII): +---------+ +---Wrapper-----+ | Manager |---[ PB transport ]---|[Referenceable]|<---- Client code +---------+ +---------------+ So the only reference most of the client code maintains is to the wrapper object, which can remain consistent across outages. It handles reconnections to the original object when needed, which will technically create a new referenceable (since PB referenceables can't continue to be used across a disconnect/reconnect), but that is transparent to the client code.
If I understand correctly, the only purpose of the registry is to provide an interface to enable the re-connection of the wrappers. The purpose of managers is simply to dispense data and state objects.
The primary reason for the registry in our system is to act as a single management object to retrieve references to our registerable objects (such as managers), whether the request to locate a given registerable object is coming locally or remotely. Much as any other central registry, it permits us to pass a single object reference around to various parts in the system (including remote clients) through which access to other official entry points can be retrieved. But yes, it also simplifies the remote connection process since all we need to do is provide a remote reference to a registry and through that remote (wrapped) references to objects such as managers may be retrieved through the same code that would be used if the registry was local. And yes, as I've described managers are largely data management objects. We do also have higher order registerables (we call them packages) which implement high level functionality - generally to simplify common operations that would otherwise need to interact with several managers simultaneously.
Questions:
Do managers only dispense state and data, or do they also provide state control?
I suppose it depends on what you would consider covered by "state control," but the general answer would be there's no single rule. Some managers are almost entirely pure data storage/retrieval, while others provide for the retrieval of objects that themselves are fairly complex (such as our cacheable models/controllers).
Do cacheables (state objects) re-connect? Is there any reason why they shouldnt?
Yes, they can be wrapped as well. To the server side instance of the cacheable, a reconnection is just another "new" observer.
Regarding multi-layer wrapping, how do cacheables go from the original server to the final client without becoming unjellyable in the middle?
(warning - this got very long after I started writing it...) I'm not sure if you meant copyable here instead of cacheable since a cacheable controls it's own transmission of state to the client, as opposed to a copyable which has to be directly jellyable. For the cacheable, as long as it implements the Cacheable support, it controls what gets transmitted to any observer, so whether it's the original instance or a client reference to the original instance, it's transmitting the same data. But we have to date handled cacheables with an additional layer. Since we use cacheables typically for models for which users need to monitor changes, we needed something that works the same locally and remotely. We tend to use pydispatcher for signals (or some of our older objects handle the observer pattern directly), and implement our models using that, so all monitoring is technically local. We then have a generic server side wrapper that is a pb.Cacheable, and can observe any such model as its data for the cacheable clients. This might also work by just having the models be directly cacheable, but it's the way the system has grown to date. The key to most of this is that we built a structure where the remote wrapped instance of an object uses the same class definition (directly through inheritance) as the original instance, just with a wrapper mixed-in. Not only does the client side wrapped object "work" like the local object, with the use of callRemote hidden behind the normal interface, but it then can be remotely referenced itself and behave just like the original reference. To try to strip down to a simple example, we were able to encapsulate pretty much everything about the distributed processing part of the system into two package modules - remoteable.py for a server side support, and remote.py for client side. remoteable is thin - we've have copyable/referenceable/cacheable subclasses just to isolate some custom code (lets classes define some fields that should automatically pickle to avoid PB not knowing how to transmit them) and for future expansion. This also houses the server side observer/cacheable wrapper I mentioned above. remote handles the client side. It defines the key wrapper classes (for client side copyable copies, referenceable references, and cacheable caches :-)). These wrapper classes implement reconnections (cacheable/referenceable) and other custom support (like unpickling for copies). They also themselves inherit from the remoteable classes so they can also be passed over a PB session. remote then defines classes that multiply inherit from each of the original classes for those classes that may be distributed, as well as the appropriate wrapper class. In most cases these definitions are simply "pass" but they sometimes define slightly custom functionality for the client side. The only really detailed one is the remote.Registry, which has the knowledge to automatically wrap any retrieved object in the appropriate wrapper. An example may help. Assuming the following classes in remoteable/remote as mentioned above: - - - - - - - - - - - - - - - - - - - - - - - - - remoteable.Copyable, Cacheable, Referenceable - subclasses of pb.* remoteable.ModelCache - wraps an model as a cacheable. We have subclasses of this for each model (so we can register the unjellying) remote.CopyObject - mirror on the remote side for remoteable.Copyable. Is itself also a remoteable.Copyable remote.RemoteWrapper - remote side wrapper for a Referenceable. Is itself also a remoteable.Referenceable. - - - - - - - - - - - - - - - - - - - - - - - - - Then, if in a core module in our package (call it aurora.User) in the system we defined some user related objects (that are meant to be distributable), it might look like: - - - - - - - - - - - - - - - - - - - - - - - - - class User(remoteable.Copyable): """A typical data object""" # Attributes and simple methods for manipulating as needed pass class UserModel(remoteable.Cacheable): """A typical cached model""" # Attributes and signal support for notification on changes pass class UserManager(interfaces.IUserManager, remotable.Referenceable): """A typical manager. IUserManager is an interface definition for the public API""" # Methods for accessing/changing User and UserModel objects # Assume that getUser retrieves user and getModel retrievs a UserModel - - - - - - - - - - - - - - - - - - - - - - - - - As it stands above, the user objects would be fully usable in a local context. Access to the UserManager would be through a Registry in which it had been registered, and the UserManager would provide access to either User or UserModel objects. To permit distribution, we'd first add appropriate remote_* (or view_*) entry points to the UserManager. Most would simply mirror their original methods (leaving it up to pb to construct the references). But any methods that returned models would be adjusted so that instead of just returning the model, they wrapped that model in an appropriate remoteable.ModelCache subclass and returned that instead. So something like: - - - - - - - - - - - - - - - - - - - - - - - - - remote_getUser = getUser def remote_getModel(self, *args, **kwargs): return remoteable.UserModel(self.getModel(*args, **kwargs)) - - - - - - - - - - - - - - - - - - - - - - - - - That's the extent to which original objects need to be touched. The only remote entry points are in managers (our referenceables), with data objects being handled by PB as copyable or cacheable. Then in the remote.py module we'd add the following: - - - - - - - - - - - - - - - - - - - - - - - - - class User(aurora.User.User, CopyObject): # CopyObject is our own mirror to remoteable.Copyable pass pb.setUnjellyableForClass(aurora.User.User, User) # Note that a remote copy can be a copy of itself (this handles hops 2+) pb.setUnjellyableForClass(User, User) class UserModel(aurora.User.UserModel, pb.RemoteCache): # Depending on how the model detects state changes, you may need to # do some processing in setCopyableState or you may not. pass pb.setUnjellyableForClass(remoteable.UserModel, UserModel) class UserManager(RemoteWrapper, interfaces.IUserManager): exclude = "remote_getModel" # We still need to locally wrap as a cacheable for hops 2+ def remote_getModel(self, *args, **kwargs): return remoteable.UserModel(self.getModel(*args, **kwargs)) - - - - - - - - - - - - - - - - - - - - - - - - - The last one could probably use some explaining. Our RemoteWrapper class intercepts attribute lookups, and based on any superclass that is one of our interfaces, uses the interface definition to reflect method calls (as well as remote_* versions of them) over callRemote. We permit certain methods to be excluded from the wrapping (via an "exclude" attribute) which lets us handle them locally in the wrapper. In this case, just as the original user object did, we need to wrap the local cache of a UserModel in the cacheable before trying to return to any further remote callers. (This is where having our remote.UserModel be directly a pb.Cacheable might simplify things). But the getUser method is basically for free, since PB will handle making a copyable of the original user object which will end up coming across to the client wrapped as a remote.User object. Overall, we don't do that much overriding of the remote methods. One case where we do is for the remote.Registry since it's responsible for always wrapping returned managers in the right remote class. Since our registry lookup method is given an interface to find a manager for, the remote.Registry looks in the local module (remote) for a class definition inheriting from the same interface and then uses that to wrap the returned referenceable, thus more or less transparently making the returned referenceable look just like the original object. These remote.* objects are all themselves copy/cache/referenceable since they also inherit from their remoteable counterparts (or are wrapped by such as in the getModel call). So this can go on for many hops. Now let me see if I can put this together with a few other components. For example, in a two hop setup, you'd get: Server [<--A-->] Client 1 [<---B--->] Client 2 (a) Registry <------ remote.Registry <------ remote.Registry (b) UserManager <--- remote.UserManager <--- remote.UserManager (c) User <---------- remote.User <---------- remote.User (d) UserModel <----- remote.UserModel <----- remote.UserModel (etc...) The connections "A" and "B" are actually paired Server and Client objects of our own (that I mentioned in my last note). During a startup sequence, Server creates the master registry (including instantiating and registering any managers). It then establishes a Server object that provides access to the Registry for network clients. Simultaneously the Registry may be used by local processing. At some point, Client 1 uses its Client object to connect to Server's Server object and retrieve a reference to Registry (a), which is wrapped in a remote.Registry by the Client object. That remote.Registry can then be published by Client 1's Server object (the Server object just knows it has a registry, but can't or needn't distinguish between Registry and remote.Registry), which can be retrieved by Client 2's Client object. Client 2 also gets a remote.Registry, but it's an extra "hop" removed from the original Registry instance. Now sticking with 2 hops, say Client 2 needs some information. First, it'll ask its registry for a reference to the UserManager. The call is reflected by Client 2's remote.Registry up to Client 1, whose remote.Registry reflects it up to Server's Registry. That Registry returns a reference to UserManager which PB sends as a referenceable (shared only between Server and Client 1). The remote.Registry on Client 1 wraps that as a remote.UserManager and then returns it to Client 2, which again causes PB to send a referenceable (shared only between Client 1 and Client 2), which Client 2's remote.Registry again wraps as a remote.UserManager. Now, Client 2 asks its UserManager for a User object. The call reflects up to the Server the same way, but the response this time is a copyable, so PB copies it across Server->Client 1 (which instantiates a remote.User), which is then copied by PB from Client 1->Client 2 (creating another remote.User). And perhaps now Client 2 wants a UserModel (asking the UserManager). Call again reflects up to Server, but the remote entry point on the main UserManager wraps the UserModel in a remoteable.UserModel to return to PB, which then treats it as a cacheable down to Client 1, which instantiates it as remote.UserModel. Client 1's remote.UserManager then wraps it in a local remoteable.UserModel to return (as a cacheable) to Client 2, which gets a remote.UserModel. From Server's perspective there is a remoteable.UserModel instance (which is watching signals on the original UserModel) which has Client 1 as a PB observer, and from Client 1's perspective there is a remoteable.UserModel instance (which is watching signals on the local remote.UserModel) which has Client 2 as a PB observer. Still with me? :-) Now let's say there's an outage - say between Server and Client 1. Whatever the next attempt is to use callRemote in any wrapped object will detect the problem and emit a disconnected signal. We also have a periodic Client->Server object "ping" that will pick up an outage in the absence of other calls, which occurs periodically or is triggered automatically upon receiving the disconnected signal from any wrapper object. Upon detection by the Client object of the outage, it then emits its own disconnected signal, upon which various application level operations may take place, officially disconnects the PB socket, and starts attempting to reconnect. Any operations on wrapped objects past this point will generate the normal PB DeadReferenceError exception since we shut down the connection. When Client 1's Client object manages to reconnect, it will immediately re-query the registry from the Server's Server object. Once it has successfully retrieved the new registry, it then emits a newly connected signal which includes the new registry reference. Our remote.Registry object (along with other application level stuff) listens for this signal and upon receipt, updates its internal wrapped reference, and automatically issues a requery to that reference for any managers that had previously been queried through it. When it gets new references to them it updates its internal information, as well as any wrappers that it had previously handed out (it keeps a cache). Once this final step is completed, any application code that had been attempting to use those wrapped references will be working again. The remote copyables don't need any special support since they are still legitimate copies. But remote cacheables also need to be re-connected, and are trickier since it's harder to come up with a single way to retrieve new cacheables, since they are less regular than manager references retrieved through the registry. To date we've handled this on a case by case basis either through the wrapper of the responsible manager, or via application level support for re-retrieving the model upon receipt of the reconnection signal.
P.S.
(...)
The system I had in mind:
I like and want to adhere to the data/state distinction you made. Events will be handled locally by remote caches, based on changes in the cached data (this may be accomplished degenerately, by not exposing anything other than the event).
As mentioned above, in our case we make use of pydispatcher for signals/events within each local application space, using the PB cacheable setup (with wrappers on each end) to reflect the data. This lets client code be written as if it was handling local signals regardless of whether the model object is a cache of a remote object or truly the local instance.
Differences from your system:
I would like all of my referenceables and cacheables in my system to be re-connecting. This to some extent cancels the need for managers, because any dynamically changing object is re-connecting.
We're pretty much auto-reconnecting (as above). I think you'll probably need something akin to a manager, or at least a registry to perform the reconnection though, or else you won't have a well-defined point at the original to which you can re-issue the original request to get a new referenceable/cacheable on the reconnecting client.
Instead of (or possibly in addition to) manager objects, I want to have what I call Seed objects, which represent a combination of state, data and referenceables (all optional). These seeds will be copyable, and will include the knowledge required to retrieve their components.
Sounds reasonable. I still think you'll need a separate construct to "own" access to these Seed objects, or else what is the remote seed reference going to issue a query against in order to rebuild its remote references following an outage? (...)
The main reasons for seeds are:
I want state, data and control to be provided by the same object for clarity, and not have each of them require an individual query. For example, a user will have Name, email address, etc. as data, online/offline as state, and a send_message method.
One thing to consider is the creation of the information/state that the seeds are encapsulating. One of the reasons we ended up going more heavily towards copyable objects (rather than references) is that we can end up creating such objects at various points within the distributed system. So it's very convenient to be able to instantiate a local object instance (say of a user object) in order to begin the process of creating a user, and populating its information, without bringing in the rest of the baggage of the remote connection until it comes time for the "store" operation. Likewise we found it much easier to manage reconnections upon "active" objects with well-defined APIs as opposed to the data objects such as a user record. So even if you have the construct of a seed object to encapsulate remote handling, you might want to consider separating out the data object components into their own class for simpler manipulation, prior to assigning that data to a seed object to become part of the distributed system.
I want state objects, referenceables, and possibly data associated with a Seed to be retrievable from a Server different from the one who dispensed the Seed (for example, the database may provide a seed and the associated user-changeable data, but the state may be kept by a different server). For example, the users data may be stored in a database, his online/offline state retrieved from a presence server, and sending him a message may require connecting to his workstation.
This sounds like more of a reason to split some of the functionality into separable entities than trying to combine them all into a single seed object, although I could probably see some argument for combining in order to hide the origin of the data from the end user. But then you're going to have to keep a lot of information in that seed object about where each of its information pieces originally came from and be able to reconstitute the references when needed. And handle what happens if you lose contact with the owner of one piece of the information but not another. To a large extent, permitting this sort of breakout is where we headed with our registry/manager structure. To a client, it only has a registry reference, and asks it for managers in order to retrieve/manipulate state. But it doesn't know how the registry locates managers nor how the managers locate their state. So when I ask my "local" registry for a user manager, for all I know that request is replicated across 5 hosts and I eventually get what appears to be a local user manager but is a remote reference to an object instance 5 hosts away. At the same time that same registry when asked for a session manager, might return me a local object from my own local process. The decision about where the managers are located is up to top level application code that instantiates the registry and makes it available on the network (and we have various registry variants for different ways of combining local and remote managers). This lets each "hop" along the way make some of its own decisions independent of other parts of the system, with a given node running a registry in control of what any nodes "behind" it sees, or even what managers are available. This can certainly be incorporated into a single seed object, but I think you'll have to make some decisions about how the original data sources are configured (and does that itself need to be capable of being distributed). If you can own that configuration amongst various centrally maintained servers, and you're operating from primarily a hub and spoke system it'll probably work well. But if you might end up with independently operating clusters of nodes or want to distribute administrative domains over various sorts of data, it might be more of a challenge. Hope this has spurred some more thoughts. Best of luck with your project! -- David
participants (2)
-
Antony Kummel -
David Bolen