Antony Kummel antonykummel@yahoo.com writes:

This is how I understand the registry/manager/wrappers system:

The meaning of the wrappers is that referenceables are transferable to third parties who get their flow pass through the middle process, and that the wrappers get reconnected automatically

Close (IMO) - the referenceable to the original object (which is created as a result of passing that object through PB to a third party) is placed in a wrapper which is then given to client code. For example (in ASCII):

+---------+ +---Wrapper-----+ | Manager |---[ PB transport ]---|[Referenceable]|<---- Client code +---------+ +---------------+

So the only reference most of the client code maintains is to the wrapper object, which can remain consistent across outages. It handles reconnections to the original object when needed, which will technically create a new referenceable (since PB referenceables can't continue to be used across a disconnect/reconnect), but that is transparent to the client code.

If I understand correctly, the only purpose of the registry is to provide an interface to enable the re-connection of the wrappers. The purpose of managers is simply to dispense data and state objects.

The primary reason for the registry in our system is to act as a single management object to retrieve references to our registerable objects (such as managers), whether the request to locate a given registerable object is coming locally or remotely. Much as any other central registry, it permits us to pass a single object reference around to various parts in the system (including remote clients) through which access to other official entry points can be retrieved.

But yes, it also simplifies the remote connection process since all we need to do is provide a remote reference to a registry and through that remote (wrapped) references to objects such as managers may be retrieved through the same code that would be used if the registry was local.

And yes, as I've described managers are largely data management objects. We do also have higher order registerables (we call them packages) which implement high level functionality - generally to simplify common operations that would otherwise need to interact with several managers simultaneously.

Questions:

Do managers only dispense state and data, or do they also provide state control?

I suppose it depends on what you would consider covered by "state control," but the general answer would be there's no single rule. Some managers are almost entirely pure data storage/retrieval, while others provide for the retrieval of objects that themselves are fairly complex (such as our cacheable models/controllers).

Do cacheables (state objects) re-connect? Is there any reason why they shouldn’t?

Yes, they can be wrapped as well. To the server side instance of the cacheable, a reconnection is just another "new" observer.

Regarding multi-layer wrapping, how do cacheables go from the original server to the final client without becoming unjellyable in the middle?

(warning - this got very long after I started writing it...)

I'm not sure if you meant copyable here instead of cacheable since a cacheable controls it's own transmission of state to the client, as opposed to a copyable which has to be directly jellyable.

For the cacheable, as long as it implements the Cacheable support, it controls what gets transmitted to any observer, so whether it's the original instance or a client reference to the original instance, it's transmitting the same data.

But we have to date handled cacheables with an additional layer. Since we use cacheables typically for models for which users need to monitor changes, we needed something that works the same locally and remotely. We tend to use pydispatcher for signals (or some of our older objects handle the observer pattern directly), and implement our models using that, so all monitoring is technically local. We then have a generic server side wrapper that is a pb.Cacheable, and can observe any such model as its data for the cacheable clients. This might also work by just having the models be directly cacheable, but it's the way the system has grown to date.

The key to most of this is that we built a structure where the remote wrapped instance of an object uses the same class definition (directly through inheritance) as the original instance, just with a wrapper mixed-in. Not only does the client side wrapped object "work" like the local object, with the use of callRemote hidden behind the normal interface, but it then can be remotely referenced itself and behave just like the original reference.

To try to strip down to a simple example, we were able to encapsulate pretty much everything about the distributed processing part of the system into two package modules - remoteable.py for a server side support, and remote.py for client side.

remoteable is thin - we've have copyable/referenceable/cacheable subclasses just to isolate some custom code (lets classes define some fields that should automatically pickle to avoid PB not knowing how to transmit them) and for future expansion. This also houses the server side observer/cacheable wrapper I mentioned above.

remote handles the client side. It defines the key wrapper classes (for client side copyable copies, referenceable references, and cacheable caches :-)). These wrapper classes implement reconnections (cacheable/referenceable) and other custom support (like unpickling for copies). They also themselves inherit from the remoteable classes so they can also be passed over a PB session.

remote then defines classes that multiply inherit from each of the original classes for those classes that may be distributed, as well as the appropriate wrapper class. In most cases these definitions are simply "pass" but they sometimes define slightly custom functionality for the client side. The only really detailed one is the remote.Registry, which has the knowledge to automatically wrap any retrieved object in the appropriate wrapper.

An example may help. Assuming the following classes in remoteable/remote as mentioned above:

- - - - - - - - - - - - - - - - - - - - - - - - - remoteable.Copyable, Cacheable, Referenceable - subclasses of pb.* remoteable.ModelCache - wraps an model as a cacheable. We have subclasses of this for each model (so we can register the unjellying)

remote.CopyObject - mirror on the remote side for remoteable.Copyable. Is itself also a remoteable.Copyable remote.RemoteWrapper - remote side wrapper for a Referenceable. Is itself also a remoteable.Referenceable. - - - - - - - - - - - - - - - - - - - - - - - - -

Then, if in a core module in our package (call it aurora.User) in the system we defined some user related objects (that are meant to be distributable), it might look like:

- - - - - - - - - - - - - - - - - - - - - - - - - class User(remoteable.Copyable): """A typical data object""" # Attributes and simple methods for manipulating as needed pass

class UserModel(remoteable.Cacheable): """A typical cached model""" # Attributes and signal support for notification on changes pass

class UserManager(interfaces.IUserManager, remotable.Referenceable): """A typical manager. IUserManager is an interface definition for the public API""" # Methods for accessing/changing User and UserModel objects # Assume that getUser retrieves user and getModel retrievs a UserModel - - - - - - - - - - - - - - - - - - - - - - - - -

As it stands above, the user objects would be fully usable in a local context. Access to the UserManager would be through a Registry in which it had been registered, and the UserManager would provide access to either User or UserModel objects.

To permit distribution, we'd first add appropriate remote_* (or view_*) entry points to the UserManager. Most would simply mirror their original methods (leaving it up to pb to construct the references). But any methods that returned models would be adjusted so that instead of just returning the model, they wrapped that model in an appropriate remoteable.ModelCache subclass and returned that instead. So something like:

- - - - - - - - - - - - - - - - - - - - - - - - - remote_getUser = getUser

def remote_getModel(self, *args, **kwargs): return remoteable.UserModel(self.getModel(*args, **kwargs)) - - - - - - - - - - - - - - - - - - - - - - - - -

That's the extent to which original objects need to be touched. The only remote entry points are in managers (our referenceables), with data objects being handled by PB as copyable or cacheable.

Then in the remote.py module we'd add the following:

- - - - - - - - - - - - - - - - - - - - - - - - - class User(aurora.User.User, CopyObject): # CopyObject is our own mirror to remoteable.Copyable pass

pb.setUnjellyableForClass(aurora.User.User, User) # Note that a remote copy can be a copy of itself (this handles hops 2+) pb.setUnjellyableForClass(User, User)

class UserModel(aurora.User.UserModel, pb.RemoteCache): # Depending on how the model detects state changes, you may need to # do some processing in setCopyableState or you may not. pass

pb.setUnjellyableForClass(remoteable.UserModel, UserModel)

class UserManager(RemoteWrapper, interfaces.IUserManager): exclude = "remote_getModel"

# We still need to locally wrap as a cacheable for hops 2+ def remote_getModel(self, *args, **kwargs): return remoteable.UserModel(self.getModel(*args, **kwargs)) - - - - - - - - - - - - - - - - - - - - - - - - -

The last one could probably use some explaining. Our RemoteWrapper class intercepts attribute lookups, and based on any superclass that is one of our interfaces, uses the interface definition to reflect method calls (as well as remote_* versions of them) over callRemote. We permit certain methods to be excluded from the wrapping (via an "exclude" attribute) which lets us handle them locally in the wrapper. In this case, just as the original user object did, we need to wrap the local cache of a UserModel in the cacheable before trying to return to any further remote callers. (This is where having our remote.UserModel be directly a pb.Cacheable might simplify things). But the getUser method is basically for free, since PB will handle making a copyable of the original user object which will end up coming across to the client wrapped as a remote.User object.

Overall, we don't do that much overriding of the remote methods. One case where we do is for the remote.Registry since it's responsible for always wrapping returned managers in the right remote class. Since our registry lookup method is given an interface to find a manager for, the remote.Registry looks in the local module (remote) for a class definition inheriting from the same interface and then uses that to wrap the returned referenceable, thus more or less transparently making the returned referenceable look just like the original object.

These remote.* objects are all themselves copy/cache/referenceable since they also inherit from their remoteable counterparts (or are wrapped by such as in the getModel call). So this can go on for many hops.

Now let me see if I can put this together with a few other components. For example, in a two hop setup, you'd get:

Server [<--A-->] Client 1 [<---B--->] Client 2 (a) Registry <------ remote.Registry <------ remote.Registry (b) UserManager <--- remote.UserManager <--- remote.UserManager (c) User <---------- remote.User <---------- remote.User (d) UserModel <----- remote.UserModel <----- remote.UserModel (etc...)

The connections "A" and "B" are actually paired Server and Client objects of our own (that I mentioned in my last note).

During a startup sequence, Server creates the master registry (including instantiating and registering any managers). It then establishes a Server object that provides access to the Registry for network clients. Simultaneously the Registry may be used by local processing.

At some point, Client 1 uses its Client object to connect to Server's Server object and retrieve a reference to Registry (a), which is wrapped in a remote.Registry by the Client object. That remote.Registry can then be published by Client 1's Server object (the Server object just knows it has a registry, but can't or needn't distinguish between Registry and remote.Registry), which can be retrieved by Client 2's Client object. Client 2 also gets a remote.Registry, but it's an extra "hop" removed from the original Registry instance.

Now sticking with 2 hops, say Client 2 needs some information. First, it'll ask its registry for a reference to the UserManager. The call is reflected by Client 2's remote.Registry up to Client 1, whose remote.Registry reflects it up to Server's Registry. That Registry returns a reference to UserManager which PB sends as a referenceable (shared only between Server and Client 1). The remote.Registry on Client 1 wraps that as a remote.UserManager and then returns it to Client 2, which again causes PB to send a referenceable (shared only between Client 1 and Client 2), which Client 2's remote.Registry again wraps as a remote.UserManager.

Now, Client 2 asks its UserManager for a User object. The call reflects up to the Server the same way, but the response this time is a copyable, so PB copies it across Server->Client 1 (which instantiates a remote.User), which is then copied by PB from Client 1->Client 2 (creating another remote.User).

And perhaps now Client 2 wants a UserModel (asking the UserManager). Call again reflects up to Server, but the remote entry point on the main UserManager wraps the UserModel in a remoteable.UserModel to return to PB, which then treats it as a cacheable down to Client 1, which instantiates it as remote.UserModel. Client 1's remote.UserManager then wraps it in a local remoteable.UserModel to return (as a cacheable) to Client 2, which gets a remote.UserModel. From Server's perspective there is a remoteable.UserModel instance (which is watching signals on the original UserModel) which has Client 1 as a PB observer, and from Client 1's perspective there is a remoteable.UserModel instance (which is watching signals on the local remote.UserModel) which has Client 2 as a PB observer.

Still with me? :-)

Now let's say there's an outage - say between Server and Client 1. Whatever the next attempt is to use callRemote in any wrapped object will detect the problem and emit a disconnected signal. We also have a periodic Client->Server object "ping" that will pick up an outage in the absence of other calls, which occurs periodically or is triggered automatically upon receiving the disconnected signal from any wrapper object.

Upon detection by the Client object of the outage, it then emits its own disconnected signal, upon which various application level operations may take place, officially disconnects the PB socket, and starts attempting to reconnect.

Any operations on wrapped objects past this point will generate the normal PB DeadReferenceError exception since we shut down the connection.

When Client 1's Client object manages to reconnect, it will immediately re-query the registry from the Server's Server object. Once it has successfully retrieved the new registry, it then emits a newly connected signal which includes the new registry reference.

Our remote.Registry object (along with other application level stuff) listens for this signal and upon receipt, updates its internal wrapped reference, and automatically issues a requery to that reference for any managers that had previously been queried through it. When it gets new references to them it updates its internal information, as well as any wrappers that it had previously handed out (it keeps a cache). Once this final step is completed, any application code that had been attempting to use those wrapped references will be working again.

The remote copyables don't need any special support since they are still legitimate copies. But remote cacheables also need to be re-connected, and are trickier since it's harder to come up with a single way to retrieve new cacheables, since they are less regular than manager references retrieved through the registry. To date we've handled this on a case by case basis either through the wrapper of the responsible manager, or via application level support for re-retrieving the model upon receipt of the reconnection signal.

P.S.

(...)

The system I had in mind:

I like and want to adhere to the data/state distinction you made. Events will be handled locally by remote caches, based on changes in the cached data (this may be accomplished degenerately, by not exposing anything other than the event).

As mentioned above, in our case we make use of pydispatcher for signals/events within each local application space, using the PB cacheable setup (with wrappers on each end) to reflect the data. This lets client code be written as if it was handling local signals regardless of whether the model object is a cache of a remote object or truly the local instance.

Differences from your system:

I would like all of my referenceables and cacheables in my system to be re-connecting. This to some extent cancels the need for managers, because any dynamically changing object is re-connecting.

We're pretty much auto-reconnecting (as above). I think you'll probably need something akin to a manager, or at least a registry to perform the reconnection though, or else you won't have a well-defined point at the original to which you can re-issue the original request to get a new referenceable/cacheable on the reconnecting client.

Instead of (or possibly in addition to) manager objects, I want to have what I call Seed objects, which represent a combination of state, data and referenceables (all optional). These seeds will be copyable, and will include the knowledge required to retrieve their components.

Sounds reasonable. I still think you'll need a separate construct to "own" access to these Seed objects, or else what is the remote seed reference going to issue a query against in order to rebuild its remote references following an outage?

(...)

The main reasons for seeds are:

I want state, data and control to be provided by the same object for clarity, and not have each of them require an individual query. For example, a user will have Name, email address, etc. as data, online/offline as state, and a send_message method.

One thing to consider is the creation of the information/state that the seeds are encapsulating. One of the reasons we ended up going more heavily towards copyable objects (rather than references) is that we can end up creating such objects at various points within the distributed system. So it's very convenient to be able to instantiate a local object instance (say of a user object) in order to begin the process of creating a user, and populating its information, without bringing in the rest of the baggage of the remote connection until it comes time for the "store" operation. Likewise we found it much easier to manage reconnections upon "active" objects with well-defined APIs as opposed to the data objects such as a user record.

So even if you have the construct of a seed object to encapsulate remote handling, you might want to consider separating out the data object components into their own class for simpler manipulation, prior to assigning that data to a seed object to become part of the distributed system.

I want state objects, referenceables, and possibly data associated with a Seed to be retrievable from a Server different from the one who dispensed the Seed (for example, the database may provide a seed and the associated user-changeable data, but the state may be kept by a different server). For example, the user’s data may be stored in a database, his online/offline state retrieved from a presence server, and sending him a message may require connecting to his workstation.

This sounds like more of a reason to split some of the functionality into separable entities than trying to combine them all into a single seed object, although I could probably see some argument for combining in order to hide the origin of the data from the end user. But then you're going to have to keep a lot of information in that seed object about where each of its information pieces originally came from and be able to reconstitute the references when needed. And handle what happens if you lose contact with the owner of one piece of the information but not another.

To a large extent, permitting this sort of breakout is where we headed with our registry/manager structure. To a client, it only has a registry reference, and asks it for managers in order to retrieve/manipulate state. But it doesn't know how the registry locates managers nor how the managers locate their state. So when I ask my "local" registry for a user manager, for all I know that request is replicated across 5 hosts and I eventually get what appears to be a local user manager but is a remote reference to an object instance 5 hosts away. At the same time that same registry when asked for a session manager, might return me a local object from my own local process.

The decision about where the managers are located is up to top level application code that instantiates the registry and makes it available on the network (and we have various registry variants for different ways of combining local and remote managers). This lets each "hop" along the way make some of its own decisions independent of other parts of the system, with a given node running a registry in control of what any nodes "behind" it sees, or even what managers are available.

This can certainly be incorporated into a single seed object, but I think you'll have to make some decisions about how the original data sources are configured (and does that itself need to be capable of being distributed). If you can own that configuration amongst various centrally maintained servers, and you're operating from primarily a hub and spoke system it'll probably work well. But if you might end up with independently operating clusters of nodes or want to distribute administrative domains over various sorts of data, it might be more of a challenge.

Hope this has spurred some more thoughts. Best of luck with your project!

-- David