[Twisted-Python] Foolscap-0.1.1 released
I've just released Foolscap-0.1.1, the next-generation-of-PB RPC library, available in the usual place at: http://twistedmatrix.com/trac/wiki/FoolsCap http://twistedmatrix.com/~warner/Foolscap/ This release enhances many of the "constraints" (a form of explicit typechecking that can be applied to messages sent over the wire), fixes a long-standing bug in Tub.stopService() which makes it easier to write trial unit tests for code which uses Foolscap (and which was hopefully responsible for many of the spurious test failures we've seen in the past), and adds a one-way no-response 'callRemoteOnly' method. Full details are in the release notes, attached below. Due to the implementation of callRemoteOnly, this release is *not* wire-protocol compatible with the previous 0.1.0 release. Fortunately, it knows this, and the version-negotiation code will refuse to connect to an incompatible peer. Many thanks to my employer, AllMyData.com, for supporting development of this release. We're starting to use it for an internal project, and we're discovering all sorts of usability improvements that need to be made. The next batch will probably be centered around the needs of long-running server programs: specifically persistent mapping from externally-visible (but generally unguessable) names to internal handler objects, such that each time the process gets restarted, the same name maps to the next incarnation of the handler object. We already have this facility for the private key (and thus the TubID), but now it would be nice to have it for individual objects. have a well-connected day, -Brian * Release 0.1.1 (03 Apr 2007) ** Incompatibility Warning Because of the technique used to implement callRemoteOnly() (specifically the commandeering of reqID=0), this release is not compatible with the previous release. The protocol negotiation version numbers have been bumped to avoid confusion, meaning that 0.1.0 Tubs will refuse to connect to 0.1.1 Tubs, and vice versa. Be aware that the errors reported when this occurs may not be ideal, in particular I think the "reconnector" (tub.connectTo) might not log this sort of connection failure in a very useful way. ** changes to Constraints Method specifications inside RemoteInterfaces can now accept or return 'Referenceable' to indicate that they will accept a Referenceable of any sort. Likewise, they can use something like 'RIFoo' to indicate that they want a Referenceable or RemoteReference that implements RIFoo. Note that this restriction does not quite nail down the directionality: in particular there is not yet a way to specify that the method will only accept a Referenceable and not a RemoteReference. I'm waiting to see if such a thing is actually useful before implementing it. As an example: class RIUser(RemoteInterface): def get_age(): return int class RIUserListing(RemoteInterface): def get_user(name=str): """Get the User object for a given name.""" return RIUser In addition, several constraints have been enhanced. StringConstraint and ListConstraint now accept a minLength= argument, and StringConstraint also takes a regular expression to apply to the string it inspects (the regexp can either be passed as a string or as the output of re.compile()). There is a new SetConstraint object, with 'SetOf' as a short alias. Some examples: HexIdConstraint = StringConstraint(minLength=20, maxLength=20, regexp=r'[\dA-Fa-f]+') class RITable(RemoteInterface): def get_users_by_id(id=HexIdConstraint): """Get a set of User objects; all will have the same ID number.""" return SetOf(RIUser, maxLength=200) These constraints should be imported from foolscap.schema . Once the constraint interface is stabilized and documented, these classes will probably be moved into foolscap/__init__.py so that you can just do 'from foolscap import SetOf', etc. *** UnconstrainedMethod To disable schema checking for a specific method, use UnconstrainedMethod in the RemoteInterface definition: from foolscap.remoteinterface import UnconstrainedMethod class RIUse(RemoteInterface): def set_phone_number(area_code=int, number=int): return bool set_arbitrary_data = UnconstrainedMethod The schema-checking code will allow any sorts of arguments through to this remote method, and allow any return value. This is like schema.Any(), but for entire methods instead of just specific values. Obviously, using this defeats te whole purpose of schema checking, but in some circumstances it might be preferable to allow one or two unconstrained methods rather than resorting to leaving the entire class left unconstrained (by not declaring a RemoteInterface at all). *** internal schema implementation changes Constraints underwent a massive internal refactoring in this release, to avoid a number of messy circular imports. The new way to convert a "shorthand" description (like 'str') into an actual constraint object (like StringConstraint) is to adapt it to IConstraint. In addition, all constraints were moved closer to their associated slicer/unslicer definitions. For example, SetConstraint is defined in foolscap.slicers.set, right next to SetSlicer and SetUnslicer. The constraints for basic tokens (like lists and ints) live in foolscap.constraint . ** callRemoteOnly A new "fire and forget" API was added to tell Foolscap that you want to send a message to the remote end, but do not care when or even whether it arrives. These messages are guaranteed to not fire an errback if the connection is already lost (DeadReferenceError) or if the connection is lost before the message is delivered or the response comes back (ConnectionLost). At present, this no-error philosophy is so strong that even schema Violation exceptions are suppressed, and the callRemoteOnly() method always returns None instead of a Deferred. This last part might change in the future. This is most useful for messages that are tightly coupled to the connection itself, such that if the connection is lost, then it won't matter whether the message was received or not. If the only state that the message modifies is both scoped to the connection (i.e. not used anywhere else in the receiving application) and only affects *inbound* data, then callRemoteOnly might be useful. It may involve less error-checking code on the senders side, and it may involve fewer round trips (since no response will be generated when the message is delivered). As a contrived example, a message which informs the far end that all subsequent messages on this connection will sent entirely in uppercase (such that the recipient should apply some sort of filter to them) would be suitable for callRemoteOnly. The sender does not need to know exactly when the message has been received, since Foolscap guarantees that all subsequently sent messages will be delivered *after* the 'SetUpperCase' message. And, the sender does not need to know whether the connection was lost before or after the receipt of the message, since the establishment of a new connection will reset this 'uppercase' flag back to some known initial-contact state. rref.callRemoteOnly("set_uppercase", True) # returns None! This method is intended to parallel the 'deliverOnly' method used in E's CapTP protocol. It is also used (or will be used) in some internal Foolscap messages to reduce unnecessary network traffic. ** new Slicers: builtin set/frozenset Code has been added to allow Foolscap to handle the built-in 'set' and 'frozenset' types that were introduced in python-2.4 . The wire protocol does not distinguish between 'set' and 'sets.Set', nor between 'frozenset' and 'sets.ImmutableSet'. For the sake of compatibility, everything that comes out of the deserializer uses the pre-2.4 'sets' module. Unfortunately that means that a 'set' sent into a Foolscap connection will come back out as a 'sets.Set'. 'set' and 'sets.Set' are not entirely interoperable, and concise things like 'added = new_things - old_things' will not work if the objects are of different types (but note that things like 'added = new_things.difference(old_things)' *do* work). The current workaround is for remote methods to coerce everything to a locally-preferred form before use. Better solutions to this are still being sought. The most promising approach is for Foolscap to unconditionally deserialize to the builtin types on python >= 2.4, but then an application which works fine on 2.3 (by using sets.Set) will fail when moved to 2.4 . ** Tub.stopService now indicates full connection shutdown, helping Trial tests Like all twisted.application.service.MultiService instances, the Tub.stopService() method returns a Deferred that indicates when shutdown has finished. Previously, this Deferred could fire a bit early, when network connections were still trying to deliver the last bits of data. This caused problems with the Trial unit test framework, which insist upon having a clean reactor between tests. Trial test writers who use Foolscap should include the following sequence in their twisted.trial.unittest.TestCase.tearDown() methods: def tearDown(self): from foolscap.eventual import flushEventualQueue d = tub.stopService() d.addCallback(flushEventualQueue) return d This will insure that all network activity is complete, and that all message deliveries thus triggered have been retired. This activity includes any outbound connections that were initiated (but not completed, or finished negotiating), as well as any listening sockets. The only remaining problem I've seen so far is with reactor.resolve(), which is used to translate DNS names into addresses, and has a window during which you can shut down the Tub and it will leave a cleanup timer lying around. The only solution I've found is to avoid using DNS names in URLs. Of course for real applications this does not matter: it only makes a difference in Trial unit tests which are making heavy use of short-lived Tubs and connections.
On Apr 4, 2007, at 1:36 PM, Brian Warner wrote:
I've just released Foolscap-0.1.1, the next-generation-of-PB RPC library, available in the usual place at:
http://twistedmatrix.com/trac/wiki/FoolsCap http://twistedmatrix.com/~warner/Foolscap/
Wow, I had a read through the docs you made based on oldpb (http:// twistedmatrix.com/~warner/Foolscap/copyable.html and http:// twistedmatrix.com/~warner/Foolscap/using-pb.html) and think I finally understand what Foolscap is about :) - I like how you've simplified Referenceable/Copyable/Cacheable to just the one type - Referenceable - while making the whole thing more powerful in the process. - Being able to pass RemoteReferences to a third party - very cool! - Serialisers are a great idea! I often want to send classes down the wire without having to subclass them. Also, many python types can be serialised now (e.g. datetime - I don't think these work with PB, so I have to convert them to strings and back again... I do this with Decimals too.... yuck!) - You can write a Slicer/Unslicer pair to get an object that is copied by value the first time it sent and then copied by reference all later times. I *really* would like to be able to do that. I'm writing application now where uses create objects with lots of references to other objects so this would be very useful as I'm often sending these objects to clients again as they are updated, or are returned from some remote procedure call etc. and it would be nice if the client saw these as the same object. Other parts of Foolscap I've yet to understand, but I just wanted to say I like where PB is heading. Robert
Robert Gravina <robert@gravina.com> writes:
Wow, I had a read through the docs you made based on oldpb (http:// twistedmatrix.com/~warner/Foolscap/copyable.html and http:// twistedmatrix.com/~warner/Foolscap/using-pb.html) and think I finally understand what Foolscap is about :)
Thanks!
- I like how you've simplified Referenceable/Copyable/Cacheable to just the one type - Referenceable - while making the whole thing more powerful in the process.
Well, to be honest, this is more a result of not having gotten around to implementing Cacheable (or really finishing work on Copyable) than an explicit design decision. But yes, the fundamental operation is the Slicer/Unslicer pair, and Referenceables are just the pass-by-reference form of this (and Copyables are just the pass-by-copy form). I suspect that Cacheable is best implemented with something fairly application-specific, and so I'm inclined to gather some use cases before providing explicit support within Foolscap (and possibly corralling people into a non-ideal solution).
- Being able to pass RemoteReferences to a third party - very cool!
This is a big new feature, and makes things much more transparent. The actual implementation is going to change at some point (to provide some better ordering guarantees, and reduce the amount of traffic on the wire), but the user-visible API will remain the same: just pass your object through a callRemote() somehow.
- Serialisers are a great idea! I often want to send classes down the wire without having to subclass them. Also, many python types can be serialised now (e.g. datetime - I don't think these work with PB, so I have to convert them to strings and back again... I do this with Decimals too.... yuck!)
Yeah, the ability to register an ISlicer adapter is what makes this powerful. You can register an adapter for some arbitrary 3rd-party class and it will get used to serialize the instances, no matter how buried they might be inside the object graph. I'm still looking for common patterns and useful refactorings here, though. If you have examples of ways that this is helpful, please feel free to post them. For example, I'm not sure if the helper classes for Copyable are actually all that helpful, because I haven't yet hit a personal need to do pass-by-copy with anything larger than a dict.
- You can write a Slicer/Unslicer pair to get an object that is copied by value the first time it sent and then copied by reference all later times. I *really* would like to be able to do that.
The ISlicer interface should be powerful enough to do that, although I should mention that the definition of "first time" is a bit hazy. Look at foolscap.referenceable.ReferenceableSlicer.sliceBody() for an example of how it detects "first time" versus later times. There is a network optimization that we could make if we didn't need to make this distinction (it would mean the 'decref' message wouldn't need a response), but since I want to send RemoteInterface names on that first time and not thereafter, I'm inclined to retain the distinction. The biggest thing I want to figure out how to add is an object that's serialized somewhere in-between Referenceable and Copyable. My use-case is a little music-player client/server app I wrote: the server has a list of Song objects which it can give to the client for display to (and selection by) the user, and the client sends back the Song that it wants to add to the playlist. Each Song has a set of strings describing the artist name, album name, song title, etc. For the purpose of referencing which song to play, I want Song to be Referenceable. For the purpose of carrying those strings along with the Song (so the client can display a list of song titles), I want it to be Copyable. Ideally, the first time the Song was sent over the wire, it would be serialized as a connection-local ID (aka 'clid') plus a set of immutable attributes. The client-side SongCopy should have attributes that can be read, and when it gets sent back to the server in a playlist, it should be serialized with just the clid. I figure this might be a useful enough pattern to warrant a base class or some kind of support code within Foolscap (maybe a CopyableAndReferenceable class?), but I haven't yet figured out how it ought to be implemented.
Other parts of Foolscap I've yet to understand, but I just wanted to say I like where PB is heading.
Thanks! There's a lot of code that I've implemented and started to use, but have not yet started to document (like eventual-send, or much of the schema/constraint mechanics). And there more that I've written but haven't figured out how to use yet (like promises), and even more that's still in the design stages (like remote promises, promise-pipelining, and automatic keepalives). It's been very educational thus far, and promises (no pun intended) to be even more interesting in the next couple months. cheers, -Brian
On Wednesday 04 April 2007 07:36, Brian Warner wrote:
I've just released Foolscap-0.1.1, the next-generation-of-PB RPC library,
Hi Brian, Thanks for the latest release; Foolscap is turning out to be very powerful and pleasant to use. I appreciate all your work. I have two questions. 1. I'm trying to create an UnauthenticatedTub that listens on a system-assigned port (that is, on "tcp:0"). (I do this because I pass the underlying Referenceable to a remote server, and the random port is just a convenience to assist in debugging). Combining the example in the documentation and the comments in foolscap/pb.py (particularly the Listener class) implies that something like this might be possible: class MyServer ( Referenceable ): def remote_Foo ( self, blah ): return blah myserver = MyServer() tub = UnauthenticatedTub() l = tub.listenOn("tcp:0") tub.setLocation("localhost:%d" % l.getPortnum() ) url = tub.registerReference(myserver, "my-service") print "the object is available at:", url tub.startService() reactor.run() However, when code like this runs, it falls foul of the "assert self.s.running" at pb.py:73 (in getPortnum()). This seems to imply that the reactor needs to be running before we can actually assign a port, so we can't call setLocation() or registerReference() until this is done. However, I can't see where I would hang my callback to perform the remained of the setup once the reactor had started. 2. I sent an email a while back about a possible bug I found; the archived version is here: http://twistedmatrix.com/pipermail/twisted-python/2007-March/014914.html The behaviour I describe is still present in SVN, so I run against a patched install which seems to cure the problem, although I'm not claiming that it's The Right Way to solve it. Cheers, Ricky
kgi <iacovou@gmail.com> writes:
Thanks for the latest release; Foolscap is turning out to be very powerful and pleasant to use. I appreciate all your work.
Thanks!
1. I'm trying to create an UnauthenticatedTub that listens on a system-assigned port (that is, on "tcp:0"). (I do this because I pass the underlying Referenceable to a remote server, and the random port is just a convenience to assist in debugging).
tub = UnauthenticatedTub() l = tub.listenOn("tcp:0") tub.setLocation("localhost:%d" % l.getPortnum() ) url = tub.registerReference(myserver, "my-service") print "the object is available at:", url
tub.startService() reactor.run()
However, when code like this runs, it falls foul of the "assert self.s.running" at pb.py:73 (in getPortnum()).
This seems to imply that the reactor needs to be running before we can actually assign a port, so we can't call setLocation() or registerReference() until this is done.
Close.. it requires that the Tub has been started, which is a slightly weaker requirement than the reactor being running. If you rearrange the order of operations to do: tub = UnauthenticatedTub() tub.startService() l = tub.listenOn("tcp:0") tub.setLocation("localhost:%d" % l.getPortnum() ) url = tub.registerReference(myserver, "my-service") print "the object is available at:", url reactor.run() Then you should find that it starts working ok. Port numbers are allocated as soon as the Tub service is started, and isn't "slow" (in the sense that it requires multiple turns of the reactor to complete). The Tub is not supposed to do any network IO until it is started, so it won't allocate the port until that point, but it doesn't really need to be post-reactor.run(). (incidentally, if you need to know when the reactor has started for other reasons, you can use reactor.callWhenRunning(cb)) The current version of Foolscap doesn't quite honor this "don't start until I tell you to" rule: if you do getReference() before startService(), it will cheerfully initiate outbound network connections anyways. This will be fixed at some point.
2. I sent an email a while back about a possible bug I found; the archived version is here:
http://twistedmatrix.com/pipermail/twisted-python/2007-March/014914.html
Yeah, sorry about not responding to that.. things got busy that month :). (to be honest there are probably a number of issues with Copyable, as I haven't personally used it nearly as much as the rest of Foolscap.) Reading over your note, I think your analysis is exactly right. Copyable.slice() is obligated to set self.streamable before any child objects might get seralized, which means before the first yield(). I'll fix this tonight. I wonder why the unit tests didn't catch it.. thanks! -Brian
participants (4)
-
Brian Warner -
Itamar Shtull-Trauring -
kgi -
Robert Gravina