[Twisted-Python] [ANN] Foolscap-0.2.6 released
I've just finished releasing Foolscap-0.2.6, the next-generation remote object protocol, intended as the successor to Perspective Broker.
This release contains more logging improvements, including the long-awaited "incident-based logging" system (which dumps "black box" records of recent events when something bad happens), and several bugfixes. The full release notes are attached below.
Tarballs can be obtained from foolscap.lothar.com and from PyPI. All releases are signed by my GPG public key: 0x1514A7BD .
http://pypi.python.org/packages/source/f/foolscap/foolscap-0.2.6.tar.gz#md5=... http://pypi.python.org/packages/source/f/foolscap/foolscap-0.2.6.zip#md5=1b2...
For documentation, bugs, and patches, please visit http://foolscap.lothar.com/trac .
share and enjoy, -Brian
* Release 0.2.6 (06 May 2008)
** Compatibility
All releases between 0.1.3 and 0.2.6 (inclusive) are fully wire-compatible.
The saved logfiles produced (by e.g. 'flogtool tail --save-to' and the log-gatherer) in 0.2.6 and beyond are not readable by tools (e.g. 'flogtool dump' and 'flogtool filter') from 0.2.5 and earlier.
FURLs which contain "extensions" (described below) will not be tolerated by foolscap 0.2.5 or earlier. If, at some point in the future, we add such extensions to published FURLs, then such an application will require foolscap-0.2.6 or later to interpret those FURLs.
** Logging Changes
*** "Incident" Logging
This release finally implements the "strangeness-triggered logging" espoused in doc/logging.xhtml . By giving the foolscap logging code a directory to work with, the logging system will automatically save a compressed pickled logfile containing the messages that lead up to sufficiently-severe log event. The docs explain how to control what "sufficiently-severe" means. These events are retrievable through the logport, although no tools have been written yet to actually extract them. They are also retrievable by using 'flogtool dump' directly on the incident files.
*** 'flogtool as a subcommand
The implementation of the 'flogtool' executable has been rearranged to make it possible to add a 'flogtool' subcommand to some other executable.
*** 'flogtool filter' now has --above LEVEL and --from TUBID options *** 'flogtool dump' has --rx-time option, also shows failure tracebacks *** gatherer: don't add erroneous UTC-implying "Z" suffix to filename timestamps *** 'flogtool tail': don't add spurious "0" to timestamps
** constraints no longer reject ('reference',) sequences
The foolscap/banana serialization protocol responds to sending two separate copies of the same object in the same callRemote message by emitting one serialized object sequence and one 'reference' sequence: this is the standard way by which cycles are broken in the serialized data. Unfortunately, the wire-level constraint checkers in 0.2.5 and earlier would reject reference sequences with a Violation exception: if they were expecting a tuple, they would reject anything else, even a reference sequence that pointed at a tuple.
Worse yet, python's normal constant-object folding code can produce shared references where you might not expect. In the following example, the two tuples are identical objects (and result in a 'reference' sequence on the wire), despite the programmer having typed them separately:
rref.callRemote("haveTwoTuples", (0,1), (0,1) )
Foolscap-0.2.6 now allows reference sequence in all wire-level constraint checks, to avoid this false-negative Violation. The post-deserialization check will still enforce the constraint properly. It just does it late enough to be able to tell what the reference points to.
** Twisted/pyopenssl compatibility
*** compatible with Twisted-8.0.x
Some unit test failures under Twisted-8.0.x (the most recent release) were fixed: tests now pass against Twisted-8.0.x, and a buildbot is used to make sure compatibility is maintained in the future.
*** incompatible with pyOpenSSL-0.7
An incompatibility has been discovered with the most recent version of PyOpenSSL. pyopenssl 0.6 works correctly, but pyopenssl 0.7 causes modern versions of Twisted (both 2.5.x and 8.0.x) to follow a code path that breaks the Foolscap unit tests. This may or may not cause a problem in actual application use (the symptom is that the non-winning parallel connections are not disconnected properly, and several negotiation timers are left running). Until a fix is developed for either Twisted or PyOpenSSL, the recommended workaround is to downgrade to PyOpenSSL-0.6 . Twisted bug #3218 and Foolscap bug #62 exist to track this problem.
** setup.py is more setuptools-friendly
The foolscap version string is located with a regular expression rather than an import, allowing setuptools to install Foolscap as a build-dependency of some other project without having Twisted available first. If setuptools is available, we also declare a dependency upon Twisted (at least 2.4.0), to give more information to the setuptools dependency-auto-installer.
** Unauthenticated FURLs can now contain multiple connection hints
Previously they were limited to a single one
** FURLs can now contain extensions, providing forwards-compatibility
The parsing of FURLs has been refined to tolerate (and ignore) certain kinds of extensions. The "tubid" section will be able to have additional identifiers (perhaps stronger hashes for the public key, or an encryption-ready EC-DSA public key). In addition, the "connection hints" section will be able to contain alternate protocol specifiers (for TCP over IPv6, or a less connection-oriented UDP transport).
By ignoring such extensions, foolscap-0.2.6 will tolerate FURLs produced (with extensions) by some future version of foolscap. This marks the beginning of a "transition period": when such extensions are introduced, 0.2.6 will be the oldest version still capable of interoperating with the extension-using new version.
On May 6, 2008, at 3:24 PM, Brian Warner wrote:
I've just finished releasing Foolscap-0.2.6, the next-generation remote object protocol, intended as the successor to Perspective Broker.
With all respect (as I am very interested and excited by the work being done with Foolscap), I have to ask if it is still accurate to market Foolscap as the 'successor' or 'replacement' for PB.
Considering the confusion around a similar situation (e.g., web2 as a rewrite of web1), it seems like it would be best to avoid referring to something as the replacement for Twisted component until that's actually the case...
Just my 2/100ths...
-phil
With all respect (as I am very interested and excited by the work being done with Foolscap), I have to ask if it is still accurate to market Foolscap as the 'successor' or 'replacement' for PB.
Fair enough. In the early days (um, 5 years ago? wow), we called this project PB2. I moved it out of the Twisted SVN tree when the development policies began to diverge too far (UQDS vs experimental+fast-moving code). I renamed it to Foolscap at about the same time because of just the sort of uncertainty that you mentioned, and because sticking with twisted.pb2 was an installation headache (namespace packages, etc).
At PyCon, the sort of communal marketing position that we seemed to wind up in (i.e. the set of statements that, once uttered, weren't immediately corrected or contradicted by everyone else in the room) was that there's a spectrum, from AMP to PB to Foolscap, in roughly increasing order of featurefullness/size/complexity. While I see places where I'd use AMP instead of the others, personally at this point I don't see any reasons to use PB over Foolscap. But of course I feel more comfortable using Foolscap than anyone else in the world :-).
I stopped paying attention to PB tickets years ago, and fortunately more diligent people than me have stepped in and fixed them, so PB is in a better shape than it was before. It still can't do everything that Foolscap can, and is unlikely to, because, well, that's what Foolscap is for. But PB is not going anywhere soon, so if you're intersted in building something that doesn't require anything outside of the twisted tree, and you don't want capability-oriented encryption, third-party references, eventual-sends, adaptor-based third-party serialization, sealers/unsealers, or promises, then PB is for you.
I still suspect that Foolscap may find its way into twisted/foolscap/ some day, but I'm not pushing it, and I think it needs to be a lot more mature first (i.e. I need to be content with seeing the rate of change drop drastically, as befits anything under the UQDS discipline).
cheers, -Brian
On Tue, May 6, 2008 at 1:24 PM, Brian Warner warner@lothar.com wrote:
I've just finished releasing Foolscap-0.2.6, the next-generation remote object protocol, intended as the successor to Perspective Broker.
What's the current stability of foolscap? Is it deemed stable enough for production, or is it more beta, alpha, etc.?
~ Nathan
What's the current stability of foolscap? Is it deemed stable enough for production, or is it more beta, alpha, etc.?
That's an.. emotional question :-), because I've got two hats on.
The foolscap developer hat -wearing Brian wants the freedom to improve the wire protocol, which probably means breaking compatibility. There are some versioning hooks in place to make it possible to do this less-breakingly, but it's a hassle, and if this Brian had his druthers he'd be changing and breaking things left and right to make them better. This Brian would like to tell you that it's still alpha.
But the Tahoe/AllMyData developer hat -wearing Brian (you know, the one who gets a paycheck) is deploying a commercial system with thousands of users who all use Foolscap-0.2.5 to connect to our servers. This provides a strong incentive to maintain compatibility :). This Brian is forced to admit that it's at least beta, and the fact that his company is using it in a production system suggests that there's at least one group of developers who deems it stable enough for production.
So Foolscap is likely to maintain wire compatibility for the near future, and if I really need to add protocol features, I'm pretty sure that I'll go through the extra work of doing it in a non-breaking way. Take from that what you will :-).
hope that helps, -Brian
On Tue, May 6, 2008 at 8:35 PM, Brian Warner warner@lothar.com wrote:
What's the current stability of foolscap? Is it deemed stable enough for production, or is it more beta, alpha, etc.?
That's an.. emotional question :-), because I've got two hats on.
The foolscap developer hat -wearing Brian wants the freedom to improve the wire protocol, which probably means breaking compatibility. There are some versioning hooks in place to make it possible to do this less-breakingly, but it's a hassle, and if this Brian had his druthers he'd be changing and breaking things left and right to make them better. This Brian would like to tell you that it's still alpha.
But the Tahoe/AllMyData developer hat -wearing Brian (you know, the one who gets a paycheck) is deploying a commercial system with thousands of users who all use Foolscap-0.2.5 to connect to our servers. This provides a strong incentive to maintain compatibility :). This Brian is forced to admit that it's at least beta, and the fact that his company is using it in a production system suggests that there's at least one group of developers who deems it stable enough for production.
So Foolscap is likely to maintain wire compatibility for the near future, and if I really need to add protocol features, I'm pretty sure that I'll go through the extra work of doing it in a non-breaking way. Take from that what you will :-).
Great answer! For my purposes, I'll consider that as "stable but still under heavy development" ... and wait until it makes it into Twisted proper. I avoid extra external dependencies whenever I can.
~ Nathan
participants (3)
-
Brian Warner -
Nathan -
Phil Christensen