On 2006/05/05, at 22:49, Colin McMillen wrote:

...

Am I checking for equality in the wrong way? Or am I approaching the problem in the wrong way?

I'm not sure if this is the Right Way of doing things, but I got around this in my own application by making my Copyable objects have a unique ID. Keep your objects in a dict that maps from id -> object, and then just say something like:

objDict[obj.objID] = obj

whenever you get an object back. Possibly you want to check to see if the object is already in the dict, depending on your application-level constraints -- in my particular application, it would be considered an error to send back an object with an ID that wasn't already known by the server. (Also, you may or may not want to set __eq__, __lt__, etc. to compare these objects by their ID.)

Thanks Colin, this would actually work fine for what I need to do. I just figured that I was doing something wrong since it's mentioned in a few places that Twisted should be able to do this. Perhaps I am mistaking Copyable with Referenceable however... The example from the docs I was thinking of where an object is send on a round trip then compared using '==' is actually a pb.Referenceable (I think Copyable is a subclass though)? Anyway, your solution should work and I might use that for the time being. All objects that could possibly go out to clients and come back again should be stored in the ZODB, so I might be able to use whatever ID it uses, or generate one myself otherwise. If anyone knows the "Twisted Way" do deal with several clients sharing lists of objects (or even just complex objects that refer to other objects) please let me know. Thanks all for the fast responses!

- Colin

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

As I said, I'm not sure that my way of doing things is the Right Way (or considered a best practice), so if anyone can confirm or deny, that would possibly be useful. :) It seems like a hack, but it works for me.

Yeah actually I think my problem is that I shouldn't be using Copyable since they copy-by-value and what I really want to do is copy-by-reference so that when clients edit attributes on the object the server (and other clients) can find out about it. I should be using Cacheable, but not sure having every object in my application as a Cacheable with a list of observers is the best way to go about it. Guess I'll just have to think this through a little more. Although I understand whether to use Referenceable/Copyable/Cachable depends on your specific application, If anyone could suggest a good general way to approach keeping clients in sync who are sharing many objects that would be of great help. I've already got one "cache" type object (a pb.Cacheable) that keeps a list of observers and all adds/updates/deletes go through this... but up until now I have been using Copyable for the objects that are stored in lists in this cache, but found I can't update after a client modifies using Copyables (because I have no way of telling which object was sent back to the server). Using Referencable would be tricky because the clients need to access all the classes attributes to display in the UI, and that would require remote_get* methods for all of them) and using Cachables again would require *another* list of observers to be maintained - I'm just not sure how all this should work. Does anyone have any suggestions of how to tackle this problem? Robert

It's possible to make each element in the set being edited a cacheable. If you think that would introduce too much overhead, then you can add a method to the "address book" object which updates one of the entries in this address book, and uses an id to identify the entry within the address book. So the address book is the dict. That's what I did in a similar case. I'm no expert though.

HTH

Micky

Yeah. I've just realised that after saving an object in the ZODB it should have an ID attribute called "_p_oid". I can can compare copyables that come back to objects in the database this way. I think this will be much easier to implement than having around 20 classes which are all cachables, although I can see some problems arising here because the "_p_oid" only has a useful value after the object has been persisted. Perhaps someone knows of an open source app which has tackled this problem (i.e. keeping an object hierarchy in sync across multiple clients) that I could look at and get general strategies from? It seems all my solutions have a "hack" feel too them, and I can't wrap my head around how to do this using a mix of cacheables/copyables/ referencables. For now, I just have one top-level cache that get it's observe_* methods called (e.g. observe_addFoo(foo)) and then just look to see if there is an Foo with the same _p_oid in my list of Foo's and update (by just calling setCopyableState) if so, or append otherwise. Something just doesn't feel right about this approach. Robert

Only thing I can think of is generate your own id. Doesn't have to be random, you can simply increament a counter every time you create a new instance. If you're worried about the id growing too long you can search the database for unused ids that are lower than the current value of the counter. Perhaps somebody else has a better idea. On 24/06/06, Robert Gravina wrote:

This is an old thread, but I am finally tackling the round trip editing problem and unfortunately getting nowhere. Basically, I want to be able to create some object on one machine, then send it off to another (who edits some of the attributes) then when the other machine calls a method on the original machine and passes it the updated object, I can identify it simply by comparing for quality. Think of your basic client/server database application.

The Twisted howtos make the claim that " Copyable objects return unchanged from a round trip", and can be compared for quality like (obj == obj) but in all my attempts I can get this to work (they are never equal). Does someone know of some sample code where this is done successfully? It is done successfully with pb.Referenceables in the howtos (look for the pb2client.py and pb2server.py listings). http://twistedmatrix.com/projects/core/documentation/howto/pb-usage.html

Since I'm using the ZODB, I've tried using ZODB's _p_oid attribute to identify objects that come back to me but the _p_oid is None even after a transaction.commit() (since the object hasn't been persisted yet probably).

I could also create my own ID attribute and attempt to generate a random ID and compare that, but this is most definately a hack.

I am really having trouble progressing with my application because of this problem. I'd really appreciate some insight on how to go about solving this. Admittedly I'm not all that experienced with Twisted, but I thought this kind of thing was supposed to be straightforward. What am I doing wrong?

Thanks,

Robert

On 2006/05/07, at 18:55, Micky Latowicki wrote:

...

It's possible to make each element in the set being edited a cacheable. If you think that would introduce too much overhead, then you can add a method to the "address book" object which updates one of the entries in this address book, and uses an id to identify the entry within the address book. So the address book is the dict. That's what I did in a similar case. I'm no expert though.

HTH

Micky _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Ah sorry everyone... it seems I've misunderstood a very basic concept: Copyables are copy-by-value Referenceables are copy-by-reference It says so here: http://www.lothar.com/tech/papers/PyCon-2003/pb-pycon/pb.html So basically I need to use either Referenceable or Viewable! OK now I can get back to work. Robert On 2006/06/25, at 9:18, Robert Gravina wrote:

...

Only thing I can think of is generate your own id. Doesn't have to be random, you can simply increament a counter every time you create a new instance. If you're worried about the id growing too long you can search the database for unused ids that are lower than the current value of the counter.

While I appreciate your suggestions, given that PB is really quite powerful and pb.Referenceables can be identified after a round trip, I figure there must be a better way of doing this. I don't want to use pb.Referenceables because I need to access the objects attributes and call methods on the remote end. I'm not 100% sure that your method is bug-free, although I haven't had a good think about it. Thanks anyhow.

...

Perhaps somebody else has a better idea.

I really hope so. I'm stuck for ideas, short of turning everything into pb.Cacheables.

Robert

...

On 24/06/06, Robert Gravina wrote: This is an old thread, but I am finally tackling the round trip editing problem and unfortunately getting nowhere. Basically, I want to be able to create some object on one machine, then send it off to another (who edits some of the attributes) then when the other machine calls a method on the original machine and passes it the updated object, I can identify it simply by comparing for quality. Think of your basic client/server database application.

The Twisted howtos make the claim that " Copyable objects return unchanged from a round trip", and can be compared for quality like (obj == obj) but in all my attempts I can get this to work (they are never equal). Does someone know of some sample code where this is done successfully? It is done successfully with pb.Referenceables in the howtos (look for the pb2client.py and pb2server.py listings). http://twistedmatrix.com/projects/core/documentation/howto/pb- usage.html

Since I'm using the ZODB, I've tried using ZODB's _p_oid attribute to identify objects that come back to me but the _p_oid is None even after a transaction.commit() (since the object hasn't been persisted yet probably).

I could also create my own ID attribute and attempt to generate a random ID and compare that, but this is most definately a hack.

I am really having trouble progressing with my application because of this problem. I'd really appreciate some insight on how to go about solving this. Admittedly I'm not all that experienced with Twisted, but I thought this kind of thing was supposed to be straightforward. What am I doing wrong?

Thanks,

Robert

On 2006/05/07, at 18:55, Micky Latowicki wrote:

...

It's possible to make each element in the set being edited a cacheable. If you think that would introduce too much overhead, then you can add a method to the "address book" object which updates one of the entries in this address book, and uses an id to identify the entry within the address book. So the address book is the dict. That's what I did in a similar case. I'm no expert though.

HTH

Micky _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On 2006/06/27, at 1:56, David Bolen wrote:

Robert Gravina writes:

...

Ah sorry everyone... it seems I've misunderstood a very basic concept:

Copyables are copy-by-value Referenceables are copy-by-reference

It says so here: http://www.lothar.com/tech/papers/PyCon-2003/pb-pycon/pb.html

So basically I need to use either Referenceable or Viewable! OK now I can get back to work.

Well, or just define your own comparison semantics in your copyable, as you might for any other object. We use copyables all the time for precisely this pattern (return object remotely, edit it, re-use it in a subsequent call back to the server) and it works fine as long as you control the comparison and don't fall back to default Python rules, which for class instances is effectively just an id() comparison.

Thanks, that is exactly what I am trying to do (return object remotely, edit it, re-use it in a subsequent call back to the server). What kind of comparison do you do then? Do you compare all attributes? By the way, what is an id() comparison? As far as I know Python compares to see if the instances (i.e. memory locations) are equal.

As it turns out, we do also embed an "_id" attribute within the objects (an auto-generated UUID) so we can detect collisions (two clients attempting to modify the same object), but for general comparison purposes outside of that collision check, we exclude the _id attribute.

I've been looking around trying to find some good code for generating IDs, but most snippets I can find are accompanied by post after post on how that generation technique may fail under certain conditions. I've been using the ZODB, but for the live of me I can't seem to get anything out of the _p_oid attribute (supposedly the object ID, but it's always None!). This prompted me today to experiment with Axiom (thanks Divmod guys!) and I like what I see so far, but I'm having problems with that also (see here if you're interested ). I'd really like to stick with Copyables and the ZODB if possible. What I'm not sure how to do is when I get my copyable back, not screw up my references to this object in the ZODB because what I'm really getting back is a whole new object. Thanks for your advice, Robert

-- David

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Thanks Mike and others.. I just figured out how I can get this to work (with a bit of help from the ZODB mailing list too)! I am using the ZODB, and up until now I though that the _p_oid (the attribute that stores a unique object ID) was None all the time, even after I had persisted an object. I tried loading up a persisted object iteractively and found:

...

...

p._p_oid '\x00\x00\x00\x00\x00\x00\x00\x08' print p._p_oid p._p_oid is None False

... that it prints out as nothing, but it really is something (hence missing it in all my debug printouts)! (Those nutcase Zope developers!)... anyhow, it now appears I can compare the _p_oid of a Copyable coming back and find the object that represents it on the server, update it accordingly, commit the transaction and do a victory dance. Thanks all for you help. I thought I'd post this so anyone else using Twisted and ZODB won't spend days on this like I did. Robert On 2006/06/27, at 3:55, Mike Pelletier wrote:

On Monday 26 June 2006 13:55, Robert Gravina wrote:

...

By the way, what is an id() comparison? As far as I know Python compares to see if the instances (i.e. memory locations) are equal.

You got your chocolate in my peanutbutter!

An "id()" comparison would be, "id(this) == id(that)". In most (all?) implementations, 'id()' returns the memory location, but this is an implementation detail that is not guaranteed and shouldn't be depended on. An object's python id is to be treated as an opaque identifier.

Mike.

Robert Gravina writes:

Thanks, that is exactly what I am trying to do (return object remotely, edit it, re-use it in a subsequent call back to the server). What kind of comparison do you do then? Do you compare all attributes? By the way, what is an id() comparison? As far as I know Python compares to see if the instances (i.e. memory locations) are equal.

Precisely what comparison we do depends on the object in question. Yes, in many cases it's more or less a straight __dict__ comparison (sans the "_id" attribute), but in others there are only a few "identity" attributes that we consider important for comparison purposes, or that need special processing. Such as a username object equality test ignoring case of the username, or an object with date fields that might round trip through a database so we round to a precision that we consider equal. So for example, our Password object contains, for equality testing: def __hash__(self): return hash(self._lvalue) def __eq__(self,other): if isinstance(other,Password): return self._lvalue == other._lvalue elif isinstance(other,types.StringTypes): return self._lvalue == other.lower() else: return False def __ne__(self,other): return not self.__eq__(other) (_lvalue is an internally lowercased version of the current value) Note that we also chose to permit a direct string comparison with a password object, due to how it is often used to validate input, but most other objects can only compare equally to others of their type. A more blind __dict__ comparison (here for a FanClub object of ours) might go like: def __eq__(self,other): if isinstance(other,FanClub): d1 = self.__dict__.copy() d2 = other.__dict__.copy() try: del d1['_id'] except: pass try: del d2['_id'] except: pass return d1 == d2 else: raise NotImplementedError def __ne__(self,other): return not self.__eq__(other) In general, just implement the rules as appropriate for your object. Don't forget that if you are imposing rules on the equality - such as our case insensitive username object above - you want to match that with an appropriate __hash__ too, unless the object will never be used as a dictionary key.

...

As it turns out, we do also embed an "_id" attribute within the objects (an auto-generated UUID) so we can detect collisions (two clients attempting to modify the same object), but for general comparison purposes outside of that collision check, we exclude the _id attribute.

I've been looking around trying to find some good code for generating IDs, but most snippets I can find are accompanied by post after post on how that generation technique may fail under certain conditions.

I ran into a similar issue, and we ended up pretty much rolling our own that at least had our own acceptable conditions. It uses ctypes to wrap either the native Win32 call on Windows or libuuid on *nix (we've used it under FreeBSD and Linux). So it needs ctypes, and libuuid available for *nix. Note that the Win32 call pre-Win2K generates an address based UUID, while 2K+ defaults to random. We've used it for Python 2.2+. It falls back to an ASPN-based recipe worst case, but that generates UUIDs that aren't DCE conforming, so we warn to stderr in that scenario. I'll attach the generator class below if it helps. Since we wrote ours, there have been some other recent UUID generation efforts in Python (e.g., http://zesty.ca/python/uuid.py) that seem quite functional, but we didn't have incentive to move. In terms of synchronizing the re-transmitted copyable with an existing ZODB reference though, that still sounds like it may require some extra work or index keeping, unless you already have a way to locate the current internal copy of the object on the server side based on the copy supplied in the client request. Once you've got the server copy then the above discussion can be used to determine if a change has been made. -- David - - - - - - - - - - - - - - - - - - - - - - - - - class _UUIDGenerator(object): """Internal class used to contain state for UUID generation. Selects the appropriate platform specific approach to UUID generation, preferring to use appropriate external UUID routines if available, otherwise falling back to a non-standard internal mechanism. An instance of this class generates UUIDs by being called - returning the 16-byte buffer containing the UUID as raw data.""" def __init__(self): # We attempt to reference an appropriate platform routine for # UUID generation, and create a generation function to use it, # referencing that function as the "_generator" attribute of this # instance. If anything goes wrong during the process, we fall # back to a pure Python implementation try: import ctypes, struct if sys.platform == 'win32': uuidgen = ctypes.windll.rpcrt4.UuidCreate else: uuidgen = ctypes.cdll.LoadLibrary('libuuid.so').uuid_generate # Both of these platform functions fill in a 16-byte block, so # we create a single buffer to be used on each generation. Keep # in the instance so it will be available when accessed by the # generator function later. self.buffer = ctypes.c_buffer(16) # The generation function calls the platform routine to fill in # the buffer. In the Windows case, the buffer is structured as # separate fields (the UUID struct) in native byte order, so we # use struct to unpack/repack the fields into network order to # conform to the opaque buffer UUID will be expecting. def _generator(): uuidgen(self.buffer) if sys.platform == 'win32': return struct.pack('>LHH8s', *struct.unpack('LHH8s',self.buffer)) else: return self.buffer.raw except: # If we fallback to this method, generate a warning for now # (which will occur at import time) just as a head's up. sys.stderr.write('Warning: ' 'Falling back to internal UUID generation\n') def _generator(): # The following code was borrowed from ASPN, adjusted to not # use any additional option arguments. Note that this does # not generate UUIDs conforming to the DCE 1.1 specification. import time, random, md5 t = long( time.time() * 1000 ) r = long( random.random()*100000000000000000L ) try: a = socket.gethostbyname( socket.gethostname() ) except: # if we can't get a network address, just imagine one a = random.random()*100000000000000000L data = str(t)+' '+str(r)+' '+str(a) return md5.md5(data).digest() self._generator = _generator def __call__(self): """Return a buffer containing a newly generated UUID""" return self._generator()