I'm pleased to announce the release of Foolscap-0.1.5, the latest version of the next-generation Perspective Broker library. My intention with this library is to make many of the communication and cooperation features of the E language available to Python programmers, and to simplify secure communication between twisted-based python progams. (there have been several intermediate releases since foolscap-0.1.1 in April, which I neglected to announce. oops.) Foolscap has moved out of the Twisted sandbox and into its very own Mercurial repository. It also has its own Trac instance. For downloads, bugs, and all things Foolscapish (with the exception of a mailing list, for which we're still using twisted-python), please visit: http://foolscap.lothar.com/ The release can be downloaded directly from http://foolscap.lothar.com/releases/foolscap-0.1.5.tar.gz , or installed by typing 'easy_install foolscap' on a setuptools-enabled system. A large number of bugs and features have been addressed since 0.1.1, please see the NEWS file (attached) for full details. Many of them relate to the handling of third-party references ("Gifts"), connection keepalives, and Constaints. In addition, a number of usability improvements have been made based upon experience gained by deploying Foolscap in a controlled environment[1][2]. I am grateful to my employer, Allmydata.com, for providing me with the time and environment to use and develop Foolscap for a real-world application. Please download and hack away. File bugs and patches in the foolscap Trac, and discuss anything which doesn't fit into a ticket here on twisted-python. share and enjoy! -Brian [1]: http://allmydata.org/trac/tahoe/wiki (all tahoe connections use Foolscap) [2]: http://www.allmydata.com/wordpress/?p=24 (the 1.8 release uses Foolscap-0.1.4 internally) * Release 0.1.5 (07 Aug 2007) ** Compatibility This release is fully compatible with 0.1.4 and 0.1.3 . ** CopiedFailure improvements When a remote method call fails, the calling side gets back a CopiedFailure instance. These instances now behave slightly more like the (local) Failure objects that they are intended to mirror, in that .type now behaves much like the original class. This should allow trial tests which result in a CopiedFailure to be logged without exploding. In addition, chained failures (where A calls B, and B calls C, and C fails, so C's Failure is eventually returned back to A) should work correctly now. ** Gift improvements Gifts inside return values should properly stall the delivery of the response until the gift is resolved. Gifts in all sorts of containers should work properly now. Gifts which cannot be resolved successfully (either because the hosting Tub cannot be reached, or because the name cannot be found) will now cause a proper error rather than hanging forever. Unresolvable gifts in method arguments will cause the message to not be delivered and an error to be returned to the caller. Unresolvable gifts in method return values will cause the caller to receive an error. ** IRemoteReference() adapter The IRemoteReference() interface now has an adapter from Referenceable which creates a wrapper that enables the use of callRemote() and other IRemoteReference methods on a local object. The situation where this might be useful is when you have a central introducer and a bunch of clients, and the clients are introducing themselves to each other (to create a fully-connected mesh), and the introductions are using live references (i.e. Gifts), then when a specific client learns about itself from the introducer, that client will receive a local object instead of a RemoteReference. Each client will wind up with n-1 RemoteReferences and a single local object. This adapter allows the client to treat all these introductions as equal. A client that wishes to send a message to everyone it's been introduced to (including itself) can use: for i in introductions: IRemoteReference(i).callRemote("hello", args) In the future, if we implement coercing Guards (instead of compliance-asserting Constraints), then IRemoteReference will be useful as a guard on methods that want to insure that they can do callRemote (and notifyOnDisconnect, etc) on their argument. ** Tub.registerNameLookupHandler This method allows a one-argument name-lookup callable to be attached to the Tub. This augments the table maintained by Tub.registerReference, allowing Referenceables to be created on the fly, or persisted/retrieved on disk instead of requiring all of them to be generated and registered at startup. * Release 0.1.4 (14 May 2007) ** Compatibility This release is fully compatible with 0.1.3 . ** getReference/connectTo can be called before Tub.startService() The Tub.startService changes that were suggested in the 0.1.3 release notes have been implemented. Calling getReference() or connectTo() before the Tub has been started is now allowed, however no action will take place until the Tub is running. Don't forget to start the Tub, or you'll be left wondering why your Deferred or callback is never fired. (A log message is emitted when these calls are made before the Tub is started, in the hopes of helping developers find this mistake faster). ** constraint improvements The RIFoo -style constraint now accepts gifts (third-party references). This also means that using RIFoo on the outbound side will accept either a Referenceable that implements the given RemoteInterface or a RemoteReference that points to a Referenceable that implements the given RemoteInterface. There is a situation (sending a RemoteReference back to its owner) that will pass the outbound constraint but be rejected by the inbound constraint on the other end. It remains to be seen how this will be fixed. ** foolscap now deserializes into python2.4-native 'set' and 'frozenset' types Since Foolscap is dependent upon python2.4 or newer anyways, it now unconditionally creates built-in 'set' and 'frozenset' instances when deserializing 'set'/'immutable-set' banana sequences. The pre-python2.4 'sets' module has non-built-in set classes named sets.Set and sets.ImmutableSet, and these are serialized just like the built-in forms. Unfortunately this means that Set and ImmutableSet will not survive a round-trip: they'll be turned into set and frozenset, respectively. Worse yet, 'set' and 'sets.Set' are not entirely compatible. This may cause a problem for older applications that were written to be compatible with both python-2.3 and python-2.4 (by using sets.Set/sets.ImmutableSet), for which the compatibility code is still in place (i.e. they are not using set/frozenset). These applications may experience problems when set objects that traverse the wire via Foolscap are brought into close proximity with set objects that remained local. This is unfortunate, but it's the cleanest way to support modern applications that use the native types exclusively. ** bug fixes Gifts inside containers (lists, tuples, dicts, sets) were broken: the target method was frequently invoked before the gift had properly resolved into a RemoteReference. Constraints involving gifts inside containers were broken too. The constraints may be too loose right now, but I don't think they should cause false negatives. The unused SturdyRef.asLiveRef method was removed, since it didn't work anyways. ** terminology shift: FURL The preferred name for the sort of URL that you get back from registerReference (and hand to getReference or connectTo) has changed from "PB URL" to "FURL" (short for Foolscap URL). They still start with 'pb:', however. Documentation is slowly being changed to use this term. * Release 0.1.3 (02 May 2007) ** Incompatibility Warning The 'keepalive' feature described below adds a new pair of banana tokens, PING and PONG, which introduces a compatibility break between 0.1.2 and 0.1.3 . Older versions would throw an error upon receipt of a PING token, so the version-negotiation mechanism is used to prevent banana-v2 (0.1.2) peers from connecting to banana-v3 (0.1.3+) peers. Our negotiation mechanism would make it possible to detect the older (v2) peer and refrain from using PINGs, but that has not been done for this release. ** Tubs must be running before use Tubs are twisted.application.service.Service instances, and as such have a clear distinction between "running" and "not running" states. Tubs are started by calling startService(), or by attaching them to a running service, or by starting the service that they are already attached to. The design rule in operation here is that Tubs are not allowed to perform network IO until they are running. This rule was not enforced completely in 0.1.2, and calls to getReference()/connectTo() that occurred before the Tub was started would proceed normally (initiating a TCP connection, etc). Starting with 0.1.3, this rule *is* enforced. For now, that means that you must start the Tub before calling either of these methods, or you'll get an exception. In a future release, that may be changed to allow these early calls, and queue or otherwise defer the network IO until the Tub is eventually started. (the biggest issue is how to warn users who forget to start the Tub, since in the face of such a bug the getReference will simply never complete). ** Keepalives Tubs now keep track of how long a connection has been idle, and will send a few bytes (a PING of the other end) if no other traffic has been seen for roughly 4 to 8 minutes. This serves two purposes. The first is to convince an intervening NAT box that the connection is still in use, to prevent it from discarding the connection's table entry, since that would block any further traffic. The second is to accelerate the detection of such blocked connections, specifically to reduce the size of a window of buggy behavior in Foolscap's duplicate-connection detection/suppression code. This problem arises when client A (behind a low-end NAT box) connects to server B, perhaps using connectTo(). The first connection works fine, and is used for a while. Then, for whatever reason, A and B are silent for a long time (perhaps as short as 20 minutes, depending upon the NAT box). During this silence, A's NAT box thinks the connection is no longer in use and drops the address-translation table entry. Now suppose that A suddenly decides to talk to B. If the NAT box creates a new entry (with a new outbound port number), the packets that arrive on B will be rejected, since they do not match any existing TCP connections. A sees these rejected packets, breaks the TCP connection, and the Reconnector initiates a new connection. Meanwhile, B has no idea that anything has gone wrong. When the second connection reaches B, it thinks this is a duplicate connection from A, and that it already has a perfectly functional (albeit quiet) connection for that TubID, so it rejects the connection during the negotiation phase. A sees this rejection and schedules a new attempt, which ends in the same result. This has the potential to prevent hosts behind NAT boxes from ever reconnecting to the other end, at least until the the program at the far end is restarted, or it happens to try to send some traffic of its own. The same problem can occur if a laptop is abruptly shut down, or unplugged from the network, then moved to a different network. Similar problems have been seen with virtual machine instances that were suspended and moved to a different network. The longer-term fix for this is a deep change to the way duplicate connections (and cross-connect race conditions) are handled. The keepalives, however, mean that both sides are continually checking to see that the connection is still usable, enabling TCP to break the connection once the keepalives go unacknowledged for a certain amount of time. The default keepalive timer is 4 minutes, and due to the way it is implemented this means that no more than 8 minutes will pass without some traffic being sent. TCP tends to time out connections after perhaps 15 minutes of unacknowledged traffic, which means that the window of unconnectability is probably reduced from infinity down to about 25 minutes. The keepalive-sending timer defaults to 4 minutes, and can be changed by calling tub.setOption("keepaliveTimeout", seconds). In addition, an explicit disconnect timer can be enabled, which tells Foolscap to drop the connection unless traffic has been seen within some minimum span of time. This timer can be set by calling tub.setOption("disconnectTimeout", seconds). Obviously it should be set to a higher value than the keepaliveTimeout. This will close connections faster than TCP will. Both TCP disconnects and the ones triggered by this disconnectTimeout run the risk of false negatives, of course, in the face of unreliable networks. ** New constraints When a tuple appears in a method constraint specification, it now maps to an actual TupleOf constraint. Previously they mapped to a ChoiceOf constraint. In practice, TupleOf appears to be much more useful, and thus better deserving of the shortcut. For example, a method defined as follows: def get_employee(idnumber=int): return (str, int, int) # (name, room_number, age) can only return a three-element tuple, in which the first element is a string (specifically it conforms to a default StringConstraint), and the second two elements are ints (which conform to a default IntegerConstraint, which means it fits in a 32-bit signed twos-complement value). To specify a constraint that can accept alternatives, use ChoiceOf: def get_record(key=str): """Return the record (a string) if it is present, or None if it is not present.""" return ChoiceOf(str, None) UnicodeConstraint has been added, with minLength=, maxLength=, and regexp= arguments. The previous StringConstraint has been renamed to ByteStringConstraint (for accuracy), and it is defined to *only* accept string objects (not unicode objects). 'StringConstraint' itself remains equivalent to ByteStringConstraint for now, but in the future it may be redefined to be a constraint that accepts both bytestrings and unicode objects. To accomplish the bytestring-or-unicode constraint now, you might try schema.AnyStringConstraint, but it has not been fully tested, and might not work at all. ** Bugfixes Errors during negotiation were sometimes delivered in the wrong format, resulting in a "token prefix is limited to 64 bytes" error message. Several error messages (including that one) have been improved to give developers a better chance of determining where the actual problem lies. RemoteReference.notifyOnDisconnect was buggy when called on a reference that was already broken: it failed to fire the callback. Now it fires the callback soon (using an eventual-send). This should remove a race condition from connectTo+notifyOnDisconnect sequences and allow them to operate reliably. notifyOnDisconnect() is now tolerant of attempts to remove something twice, which should make it easier to use safely. Remote methods which raise string exceptions should no longer cause Foolscap to explode. These sorts of exceptions are deprecated, of course, and you shouldn't use them, but at least they won't break Foolscap. The Reconnector class (accessed by tub.connectTo) was not correctly reconnecting in certain cases (which appeared to be particularly common on windows). This should be fixed now. CopyableSlicer did not work inside containers when streaming was enabled. Thanks to iacovou-AT-gmail.com for spotting this one. ** Bugs not fixed Some bugs were identified and characterized but *not* fixed in this release *** RemoteInterfaces aren't defaulting to fully-qualified classnames When defining a RemoteInterface, you can specify its name with __remote_name__, or you can allow it to use the default name. Unfortunately, the default name is only the *local* name of the class, not the fully-qualified name, which means that if you have an RIFoo in two different .py files, they will wind up with the same name (which will cause an error on import, since all RemoteInterfaces known to a Foolscap-using program must have unique names). It turns out that it is rather difficult to determine the fully-qualified name of the RemoteInterface class early enough to be helpful. The workaround is to always add a __remote_name__ to your RemoteInterface classes. The recommendation is to use a globally-unique string, like a URI that includes your organization's DNS name. *** Constraints aren't constraining inbound tokens well enough Constraints (and the RemoteInterfaces they live inside) serve three purposes. The primary one is as documentation, describing how remotely-accessible objects behave. The second purpose is to enforce that documentation, by inspecting arguments (and return values) before invoking the method, as a form of precondition checking. The third is to mitigate denial-of-service attacks, in which an attacker sends so much data (or carefully crafted data) that the receiving program runs out of memory or stack space. It looks like several constraints are not correctly paying attention to the tokens as they arrive over the wire, such that the third purpose is not being achieved. Hopefully this will be fixed in a later release. Application code can be unaware of this change, since the constraints are still being applied to inbound arguments before they are passed to the method. Continue to use RemoteInterfaces as usual, just be aware that you are not yet protected against certain DoS attacks. ** Use os.urandom instead of falling back to pycrypto Once upon a time, when Foolscap was compatible with python2.3 (which lacks os.urandom), we would try to use PyCrypto's random-number-generation routines when creating unguessable object identifiers (aka "SwissNumbers"). Now that we require python2.4 or later, this fallback has been removed, eliminating the last reference to pycrypto within the Foolscap source tree. * Release 0.1.2 (04 Apr 2007) ** Bugfixes Yesterday's release had a bug in the new SetConstraint which rendered it completely unusable. This has been fixed, along with some new tests. ** More debian packaging Some control scripts were added to make it easier to create debian packages for the Ubuntu 'edgy' and 'feisty' distributions.