[Twisted-Python] keeping client data in sync with Twisted, mirroring some Axiom Items on the client-side
Divmod and Twisted lists, I think I've asked something similar before on the Divmod list, but I'd like to ask again for some advice about using Divmod Axiom (an object-relational DB) in a multi-user Twisted GUI application I'm writing. This has as much to do with Twisted as it does Axiom, so I felt it best to send to both lists. I apologise for the noise. The problem I'm having is mainly architectural. I have about 15 Axiom Item classes that live on a server-side database, and these are Twisted pb.Copyable, so the clients end up with a pb.RemoteCopy which has all of the attributes from the server-side Item. I now have about 1000 lines of code for the pb.Copyable Axiom Items and 500 lines of code for all these pb.RemoteCopy objects, but essentially all they are classes with the same attribute names as the Axiom Items. Apart from being a pain to maintain, it feels like this is really wasted code. There's also a lot of code to notify other clients when data in these objects changes. Basically, my app is starting to feel really bloated, and I think I'm going about this the wrong way. Someone mentioned earlier that what I really want is Zope Enterprise Objects, and they are right. A distributed database that's simple keep all clients in sync is exactly what I want, but now that I've invested so much time and energy into Twisted I'd hate to throw it all away and begin the long arduous task of learning and switching to Zope. My basic goals are this: 1) Have a database which several clients can share and modify. For example, if this is a contacts database, and I update someone's phone number, I want all the other clients to get the new number. Perhaps this can be implemented using pb.Cacheables? I already do something like this using one Cacheable which I send pb.Copyable objects to it's client-side RemoteCache as they are modified, and it updates a dict of objects indexed by storeID.... but my implementation is far from elegant :) 2) I'd like clients to be able to possibly one day "work offline". Say for example they don't have network access, they could, instead of logging in to the server, use their local cache of contacts. add/ edit/delete then and then have these changes reflected when they next logon. I assume this would probably take some effort to implement regardless of whether I used Twisted or ZEO, but my main stumbling block at the moment is that I can't send Axiom Items down the wire, since they have a Store reference and storeID etc. What I would really like to do is send Axiom Items to clients and have them create an Axiom Item *with the same storeID and details* as the one they received. Sort of like mirroring the data, but not necessarily the whole DB, just the objects they require. I'm starting to wonder now if I should replace Twisted with ZEO in my application. The main reason I didn't go with ZEO in the beginning is I couldn't find any examples of it being used for mutli-user GUI database-apps, only for mirroring/scaling Zope instances, so figured it wasn't designed with this in mind. I do like Twisted, and I'm very happy with it as a framework, I'm just struggling with how to deal with this problem. Thanks again all, Robert
On Tuesday 24 October 2006 03:32, Robert Gravina wrote:
I'm starting to wonder now if I should replace Twisted with ZEO in my application.
The good news is that zeo+zodb can be used without pulling in the rest of Zope (thank god) and you probably could marry it with the rest of your twisted app. The bad news is that zodb has completely different usage semantics and tuning assumptions than axiom, and even if you find a way to deal with zodb asynchronously (which may be a bigger problem than I just indicated now that I give it a passing though), you may find that such a change ends up leaking up a few layers of abstraction, depending on your use cases. I'd be happy to hash it out off-list if you decide to go that way. Regardless of these challenges, if your app is write-heavy, zodb is probably not the way to go. It does seem like what you are trying to do with Axiom would be a fairly common case, but it's young so it might not have been addressed. I'm eager to hear what the smart people have to say. I think I've heard noises about Axiom starting to support other databases beyond sqllite, so perhaps it is not beyond the realm of possibility that you could connect a bunch of axiom-wielding processes to the same database. For that matter, sqlite has been talking about providing a client/server interface. Either way, I think you'd be on the bleeding edge. Boy, I'm a big help, eh? Mike.
On 2006/10/25, at 1:27, Mike Pelletier wrote:
I started off doing this actually, and decided to switch to Axiom because it was object-relational. I've had some no-so-fun experiences with Plone and ZODB madness, but it probably has more to do with not understanding what was going on. For the most part, as long as all the database-related work goes on on the server side, whether I use ZODB or Axiom probably doesn't make much of a difference to the rest of the application.
Thanks. Actually most of my database access so far is along the lines of the client sending an object to the server (along with its child objects), and the server persists it (and add/edits/removes the children to suit). Your typical CRUD GUI.
I have heard this before yes. Actually I there will be more reading going on than writing. The application is your basic database-with- reports type application, so I'd say it would not be write heavy.
As am I :) What I'm trying to I believe can be done with Twisted PB. Calling remote methods on all clients who have an object which is pb.Cacheable is quite easy to do. I am just unsure about how to go about it in a generic way. Also I'm sure there's a lot more to creating a distributed object system than meets the eye, but then again Twisted PB handles much of the heavy lifting. Originally, I planned to do much of the work on the server, and one thing I liked about Twisted was that I could easy decide what work was done on the client, and what on the server. Now that I'm a little further in developmentI realise that the clients would really benifit from having much of that data locally, so that can search/make PDF reports etc. easily. Perhaps I'm just using Twisted wrong and I should be doing this on the server? I'm really I'm happy for the clients do *all* the work, which is why I'm back to considering ZEO again. The server just needs to handle authentication and access control, ensuring everyone has the latest copy of an object, make sure two users aren't allowed to open a edit GUI for some object at the same time, etc. One important point, is that with ZEO I just code as if the database was a local ZODB. I like that because it saves literally hundreds of lines of code dealing with sending objects back and forth, notifying clients etc.
If I were to go this way, then I wouldn't really be using Twisted right? In this case, I may as well use ZEO. Perhaps I'm taking the wrong approach - maybe all I need is a database server which clients connect to, and do away with the whole of Twisted cred/PB etc. that I'm using?
Boy, I'm a big help, eh?
Actually you were thanks! Robert
On Tuesday 24 October 2006 03:32, Robert Gravina wrote:
I'm starting to wonder now if I should replace Twisted with ZEO in my application.
The good news is that zeo+zodb can be used without pulling in the rest of Zope (thank god) and you probably could marry it with the rest of your twisted app. The bad news is that zodb has completely different usage semantics and tuning assumptions than axiom, and even if you find a way to deal with zodb asynchronously (which may be a bigger problem than I just indicated now that I give it a passing though), you may find that such a change ends up leaking up a few layers of abstraction, depending on your use cases. I'd be happy to hash it out off-list if you decide to go that way. Regardless of these challenges, if your app is write-heavy, zodb is probably not the way to go. It does seem like what you are trying to do with Axiom would be a fairly common case, but it's young so it might not have been addressed. I'm eager to hear what the smart people have to say. I think I've heard noises about Axiom starting to support other databases beyond sqllite, so perhaps it is not beyond the realm of possibility that you could connect a bunch of axiom-wielding processes to the same database. For that matter, sqlite has been talking about providing a client/server interface. Either way, I think you'd be on the bleeding edge. Boy, I'm a big help, eh? Mike.
On 2006/10/25, at 1:27, Mike Pelletier wrote:
I started off doing this actually, and decided to switch to Axiom because it was object-relational. I've had some no-so-fun experiences with Plone and ZODB madness, but it probably has more to do with not understanding what was going on. For the most part, as long as all the database-related work goes on on the server side, whether I use ZODB or Axiom probably doesn't make much of a difference to the rest of the application.
Thanks. Actually most of my database access so far is along the lines of the client sending an object to the server (along with its child objects), and the server persists it (and add/edits/removes the children to suit). Your typical CRUD GUI.
I have heard this before yes. Actually I there will be more reading going on than writing. The application is your basic database-with- reports type application, so I'd say it would not be write heavy.
As am I :) What I'm trying to I believe can be done with Twisted PB. Calling remote methods on all clients who have an object which is pb.Cacheable is quite easy to do. I am just unsure about how to go about it in a generic way. Also I'm sure there's a lot more to creating a distributed object system than meets the eye, but then again Twisted PB handles much of the heavy lifting. Originally, I planned to do much of the work on the server, and one thing I liked about Twisted was that I could easy decide what work was done on the client, and what on the server. Now that I'm a little further in developmentI realise that the clients would really benifit from having much of that data locally, so that can search/make PDF reports etc. easily. Perhaps I'm just using Twisted wrong and I should be doing this on the server? I'm really I'm happy for the clients do *all* the work, which is why I'm back to considering ZEO again. The server just needs to handle authentication and access control, ensuring everyone has the latest copy of an object, make sure two users aren't allowed to open a edit GUI for some object at the same time, etc. One important point, is that with ZEO I just code as if the database was a local ZODB. I like that because it saves literally hundreds of lines of code dealing with sending objects back and forth, notifying clients etc.
If I were to go this way, then I wouldn't really be using Twisted right? In this case, I may as well use ZEO. Perhaps I'm taking the wrong approach - maybe all I need is a database server which clients connect to, and do away with the whole of Twisted cred/PB etc. that I'm using?
Boy, I'm a big help, eh?
Actually you were thanks! Robert
participants (2)
-
Mike Pelletier -
Robert Gravina