[Twisted-Python] Foolscap-0.2.5 released

I'm pleased to announce the release of Foolscap-0.2.5. Foolscap is the successor to Perspective Broker, offering cryptographically-secure remote method invocation, flexible object-graph serialization, a capability-based object model, and more. Foolscap, for all your remote-object needs. Source distributions are available from the Foolscap home page, PyPI, and wherever fine remote-object products are sold: http://foolscap.lothar.com/trac http://foolscap.lothar.com/releases/foolscap-0.2.5.tar.gz http://foolscap.lothar.com/releases/foolscap-0.2.5.zip As always, Foolscap releases are signed by my GPG key (1514A7BD). In case you're wondering what happened to 0.2.4, or 0.2.3, you're right: there have been several unannounced releases. A lot of the recent work has gone into improvements in the logging subsystem (know as the "flog" system). It is easy for Foolscap-using programs to publish a 'logport', to which the 'flogtool tail' command can connect and watch log events scroll by. Log events can be gathered and inspected with command-line tools or a web-based browser. Future work is planned to enable cross-application causality tracing using the Causeway viewer. In addition to logging changes, several other improvements have been made. Duplication connection handling should be working better now. A number of sample programs have been added to the distribution. Default size limits have been removed from all Constraints. A complete set of release notes for the last six months of releases is attached below. All versions since 0.1.3 (released in May 2007) are wire-compatible. Foolscap is currently being deployed in the allmydata.org "Tahoe" least-authority grid filesystem, which improves the chances that wire-compatibility will be retained in the future. I am grateful to allmydata.com (my employer) for supporting Foolscap development. Please direct any questions to this, the twisted-devel mailing list. Bugs, patches, and feature requests should be placed on the Foolscap Trac, at http://foolscap.lothar.com/ . cheers, -Brian * Release 0.2.5 (25 Mar 2008) ** Compatibility All releases between 0.1.3 and 0.2.5 (inclusive) are fully wire-compatible. The new 'flogtool tail --catch-up' command requires a log publisher running 0.2.5 or later. 'flogtool tail' without the --catch-up option will work with earlier publishers. ** Licensing clarification Foolscap is distributed under the (very liberal) terms of the MIT license, which is the same license that Twisted uses. It's been like this since the beginning, but this is the first release to make this obvious by including a LICENSE file. ** foolscap.logging Changes 'flogtool tail' now has a --catch-up option, which prompts the remote publisher to deliver stored historical events to the subscribe, in proper sequential order. This allows you to connect to a process that has just done something interesting and grab a copy of the log events relevant to that event. 'flogtool tail' also has a --save-to option, which specifies a filename to which all captured events should be saved. This file can be processed further with 'flogtool dump', 'flogtool filter', or 'flogtool web-viewer'. This behaves much like the unix 'tee' utility, except that the saved data is recorded in a lossless binary format (whereas the text emitted to stdout is not particularly machine-readable). 'flogtool tail' and 'flogtool dump' both emit human-readable log messages by default. The --verbose option will emit raw event dictionaries, which contain slightly more information but are harder to read. 'flogtool create-gatherer' will create a log gatherer .tac file in a new working directory. This .tac file can be launched with 'twistd', the standard Twisted daemon-launching program. This is significantly easier to work with than the previous 'flogtool gather' command (which has been removed). The new --rotate option will cause the log-gatherer to switch to a new output file every N seconds. The --bzip option will make it compress the logfiles after rotating them. For example, a log gatherer that rotates and compresses log files once per day could be created and launched with: flogtool create-gatherer --rotate 86400 --bzip ./workdir (cd workdir && twistd -y gatherer.tac) ** New sample programs doc/listings/command-server.py and command-client.py are a pair of programs that let you safely grant access to a specific command. The server is configured with a command line to run, and a directory to run it from. The client gets a FURL: when the client is executed, the server will run its preconfigured command. The client gets to see stdout and stderr (and the exit status), but does not get to influence the command being run in any way. This is much like setting up an ssh server with a restricted command, but somewhat easier to configure. doc/listings/xfer-server.py and xfer-client.py are similar, but provide file transfer services instead of command execution. ** New Features Tub.setLocationAutomatically() will try to determine an externally-visible IP address and feed it to Tub.setLocation(). It does this by preparing to send a packet to a well-known public IP address (one of the root DNS servers) and seeing which network interface would be used. This will tend to find the outbound default route, which of course is only externally-visible if the host is externally-visible. Applications should not depend upon this giving a useful value, and should give the user a way to configure a list of hostname+portnumbers so that manually-configured firewalls, port forwarders, and NAT boxes can be dealt with. * Release 0.2.4 (28 Jan 2008) ** Compatibility All releases between 0.1.3 and 0.2.4 (inclusive) are fully wire-compatible. ** foolscap.logging Changes *** 'flogtool filter' command added This mode is used to take one event-log file and produce another with a subset of the events. There are several options to control the filtering: "--strip-facility=foolscap" would remove all the foolscap-related messages, and "--after=start --before=end" will retain events that occur within the given period. The syntax is still in flux, expect these options to change in the next few releases. The main idea is to take a very large logfile and turn it into a smaller, more manageable one. *** error logging Applications should avoid recording application-specific instances in log events. Doing so will forces the log viewer to access the original source code. The current release of foolscap uses pickle, so such instances will be loadable if the viewer can import the same code, but future versions will probably switch to using Banana, at which point trying to log such instances will cause an error. In this release, foolscap stringifies the type of an exception/Failure passed in through the failure= kwarg, to avoid inducing this import dependency in serialized Failures. It also uses the CopiedFailure code to improve portability of Failure instances, and CopiedFailures have been made pickleable. The preferred way to log a Failure instance is to pass it like so: def _oops(f): log.msg("Oh no, it failed", failure=f, level=log.BAD) d.addErrback(_oops) Finally, a 'log.err()' method was added, which behaves just like Twisted's normal log.err(): it can be used in a Deferred errback, or inside an exception handler. *** 'flogtool web-viewer' Adding a "sort=time" query argument to the all-events viewing page URL will turn off the default nested view, and instead will sort all events strictly by time of generation (note that unsynchronized clocks may confuse the relative ordering of events on different machines). "sort=number" sorts all events by their event number, which is of dubious utility (since these numbers are only scoped to the Tub). "sort=nested" is the default mode. The web-viewer now provides "summary views", which show just the events that occurred at a given severity level. Each event is a hyperlink to the line in the all-events page (using anchor/fragment tags), which may make them more convenient to bookmark or reference externally. * Release 0.2.3 (24 Dec 2007) ** Compatibility All releases between 0.1.3 and 0.2.3 (inclusive) are fully wire-compatible. ** Bug Fixes RemoteReference.getSturdyRef() didn't work (due to bitrot). It has been fixed. ** foolscap.logging Changes This release is mostly about flogging improvements: some bugs and misfeatures were fixed: *** tolerate '%' in log message format strings Dictionary-style kwarg formatting is now done with a twisted-style style format= argument instead of happening implicitly. That means the acceptable ways to call foolscap.logging.log.msg are: log.msg("plain string") log.msg("no args means use 0% interpolation") log.msg("pre-interpolated: %d apples" % apples) log.msg("posargs: %d apples and %d oranges", apples, oranges) log.msg(format="kwargs: %(numapples)d apples", numapples=numapples) The benefit of the latter two forms are that the arguments are recorded separately in the event dictionary, so viewing tools can filter on the structured data, think of something like: [e for e in allEvents if e.get("numapples",0) > 4] *** log facility names are now dot-separated, to match stdlib logging *** log levels are derived from numerical stdlib logging levels *** $FLOGFILE to capture flog events during 'trial' runs One challenge of the flogging system is that, once an application was changed to write events to flogging instead of twisted's log, those events do not show up in the normal places where twisted writes its logfiles. For full applications this will be less of an issue, because application startup will tell flogging where events should go (flogging is intended to supplant twisted logging for these applications). But for events emitted during unit tests, such as those driven by Trial, these events would get lost. To address this problem, the 0.2.3 flogging code looks for the "FLOGFILE" environment variable at import time. This specifies a filename where flog events (a series of pickled event dictionaries) should be written. The file is opened at import time, events are written during the lifetime of the process, then the file is closed at shutdown using a Twisted "system event trigger" (which happens to be enough to work properly under Trial: other environments may not work so well). If the FLOGFILE filename ends in .bz2, the event pickles will be compressed, which is highly recommended because it can result in a 30x space savings (and e.g. the Tahoe unit test run results in 90MB of uncompressed events). All 'flogtool' modes know how to handle a .bz2 compressed flogfile as well as an uncompressed one. The "FLOGTWISTED" environment variable, if set, will cause this same code to bridge twisted log messages into the flogfile. This makes it easier to see the relative ordering of Twisted actions and foolscap/application events. (without this it becomes very hard to correlate the two sources of events). The "FLOGLEVEL" variable specifies a minimum severity level that will be put into the flogfile. This defaults to "1", which puts pretty much everything into the file. The idea is that, for tests, you might as well record everything, and use the filtering tools to control the display and isolate the important events. Real applications will use more sophisticated tradeoffs between disk space and interpretation effort. The recommended way to run Trial on a unit test suite for an application that uses Foolscap is: FLOGFILE=flog.out FLOGTWISTED=1 trial PACKAGENAME Note that the logfile cannot be placed in _trial_temp/, because trial deletes that directory after flogging creates the logfile, so the logfile would get deleted too. Also note that the file created by $FLOGFILE is truncated on each run of the program. * Release 0.2.2 (12 Dec 2007) ** Compatibility All releases between 0.1.3 and 0.2.2 (inclusive) are fully wire-compatible. New (optional) negotiation parameters were added in 0.2.1 (really in 0.2.0). ** Bug Fixes The new duplicate-connection handling code in 0.2.1 was broken. This release probably fixes it. There were other bugs in 0.2.1 which were triggered when a duplicate connection was shut down, causing remote calls to never be retired, which would also prevent the Reconnector from doing its job. These should be fixed now. ** Other Changes Foolscap's connection-negotiation mechanism has been modified to use foolscap logging ("flog") instead of twisted.log . Setting the FLOGFILE= environment variable will cause a Foolscap-using program to write pickled log events to a file of that name. This is particularly useful when you want to record log events during 'trial' unit test run. The normal API for setting this file will be added later. The FLOGTWISTED= environment variable will cause the creation of a twisted.log bridge, to copy all events from the twisted log into the foolscap log. The 'flogtool web-view' mode has been enhanced to color-code events according to their severity, and to format Failure tracebacks in a more-readable way. * Release 0.2.1 (10 Dec 2007) ** Compatibility All releases between 0.1.3 and 0.2.1 (inclusive) are fully wire-compatible. 0.2.1 introduces some new negotiation parameters (to handle duplicate connections better), but these are ignored by older versions, and their lack is tolerated by 0.2.1 . ** New Features *** new logging support Foolscap is slowly acquiring advanced diagnostic event-logging features. See doc/logging.xhtml for the philosophy and design of this logging system. 0.2.1 contains the first few pieces, including a tool named bin/flogtool that can be used to watch events on a running system, or gather events from multiple applications into a single place for later analysis. This support is still preliminary, and many of the controls and interfaces described in that document are not yet implemented. *** better handling of duplicate connections / NAT problems The connection-management code in 0.1.7 and earlier interacted badly with programs that run behind NAT boxes (especially those with aggressive connection timeouts) or on laptops which get unplugged from the network abruptly. Foolscap seeks to avoid duplicate connections, and various situtations could cause the two ends to disagree about the viability of any given connection. The net result (no pun intended) was that a client might have to wait up to 35 minutes (depending upon various timeout values) before being able to reestablish a connection, and the Reconnector's exponential backoff strategy could easily push this into 90 minutes of downtime. 0.2.1 uses a different approach to accomplish duplicate-suppression, and should provide much faster reconnection after netquakes. To benefit from this, both ends must be running foolscap-0.2.1 or newer, however there is an additional setting (not enabled by default) to improve the behavior of pre-0.2.1 clients: tub.setOption("handle-old-duplicate-connections", True). *** new Reconnector methods The Reconnector object (as returned by Tub.connectTo) now has three utility methods that may be useful during debugging. reset() drops the backoff timer down to one second, causing the Reconnector to reconnect quickly: you could use this to avoid an hour-long delay if you've just restarted the server or re-enabled a network connection that was the cause of the earlier connection failures. getDelayUntilNextAttempt() returns the number of seconds remaining until the next connection attempt. And getLastFailure() returns a Failure object explaining why the last connection attempt failed, which may be a useful diagnostic in trying to resolve the connection problems. ** Bug Fixes There were other minor changes: an OS-X unit test failure was resolved, CopiedFailures are serialized in a way that doesn't cause constraint violations, and the figleaf code-coverage tools (used by foolscap developers to measure how well the unit tests exercise the code base) have been improved (including an emacs code-used/unused annotation tool).
participants (1)
-
Brian Warner