Mailman 3 March 2008 - Twisted

[Twisted-Python] Advice sought on application evolution
by Don Dwiggins March 28, 2008

March 28, 2008

glyph(a)divmod.com wrote in the "Teach Me Twisted Redux" thread: > One of the things I did at PyCon was to reach out to the Django > community and ask them to start considering what it would take for > Twisted to be the preferred container and deployment platform for > Django. They're thinking about it. I've started thinking about something vaguely similar. Currently, I have a small xml-rpc server in twisted, which provides a few operations on a large, complex database for a remote client. (The bulk of the database operations are carried out by a desktop app.) This was my first serious exposure to Twisted, and I found it reasonably easy to get going, once I got the hang of the concepts. (It was probably eased by my having previous experience with asynchronous I/O, back in Paleolithic times when that was _the_ way it was done.) My intention is to evolve this server toward a true "middle-tier" layer containing the business logic that's now scattered helter-skelter among various desktop apps and database stored procedures. This layer, then, will be accessed both by remote clients and "local" (on the same LAN) desktop apps. I'm leaning toward evolving the current xml-rpc server in this direction, but I'm uncertain whether it's a good idea to implement a large amount of business logic directly in an asynchronous paradigm, and specifically directly in the Twisted framework. Twisted looks to be an excellent base for a node (loosely speaking) in a communication network, but what are the implications of using it for what I have in mind? A small example of something that bothers me is the necessity to break up otherwise "unitary" functions because some parts of them involve communications or heavy computation. More concretely, consider the following sketch of a method: def frobulate(...): step_1() step_2() step_3() step_4() step_5() return result5 Where the steps are logically coherent parts of the frobulation (in Beck's terms, the method "smells good"). Now assume that step_3 involves a database access. To avoid blocking, I might twist it like this: def frobulate(): step_1() step_2() deferred = step_3_async() deferred.addCallback(frobulate_4_and_5) return deferred def frobulate_4_and_5(...): step_4() step_5() return result5 (Where step_3 has been changed to start the access and return a deferred.) In this simple sketch, the "visual noise" introduced isn't too bad, but I'm afraid that as the functions multiply and become more complex, it'll make the application considerably harder to understand and manage, and possibly more prone to errors and/or unintended blocking, where twisting a synchronous operation would require serious re-architecting. Another twist, using generators, might be: @defer.deferredGenerator def frobulate(): step_1() step_2() step3InProgress = defer.waitForDeferred(step_3_async()) yield step3InProgress step3Rslt = step3InProgress.getResult() step_4() step_5() yield result5 return (I've introduced a bit more detail, in that the result of step 3 might be used in subsequent steps.) In Python 2.5, I can say this more cleanly: @defer.inlineCallbacks def frobulate(): step_1() step_2() step3Rslt = yield step_3_async() step_4() step_5() returnValue(result5) This is looking a lot better, but still the asynchrony of step_3 "shows through". (Again, I'm taking the viewpoint of someone trying to understand, maintain, and extend the functionality of the application, without having the asynchronous aspects "thrust in his face" more than absolutely necessary.) Of course, I can avoid all this by going to a threading or separate process solution, but I'd like to find out if I'm missing something before I make such a commitment. Thanks for any good words, -- Don Dwiggins Advanced Publishing Technology

9 13

[Twisted-Python] could plugin cache get currupted?
by Gabriel Rossetti March 27, 2008

March 27, 2008

Hello, I coded a program that uses Twisted's plugin system. I have noticed that if I interrupt the application using Ctrl+C (SIGINT) that when I run it next time I get this : File "/usr/lib/python2.5/site-packages/twisted/plugin.py", line 227, in getPlugins allDropins = getCache(package) --- <exception caught here> --- File "/usr/lib/python2.5/site-packages/twisted/plugin.py", line 178, in getCache provider = namedAny(module.__name__ + '.' + moduleName) File "/usr/lib/python2.5/site-packages/twisted/python/reflect.py", line 375, in namedAny obj = getattr(obj, n) exceptions.AttributeError: 'module' object has no attribute 'SendFile' Could this mean that the plugin cache is corrupt? If so, How can I keep it from getting corrupted (is there a safe way of closing it)? Thanks, Gabriel

2 3

[Twisted-Python] Re: Advice sought on application evolution
by Andrew Francis March 27, 2008

March 27, 2008

Hi Justin: > Too much serialization, even accidental >serialization, trades comprehensibility for >performance. I think that's a reasonable place >to start: make your system easy to understand, then >make it fast once you know what's going on. JW>How does the saying go? "Premature optimisation is JW>the root of all evil"? Butler Lampson in the paper "Hints for Computer System Design" saids "Get it right! Neither abstraction nor simplicity is a substitute for getting it right." I think if there is a tao of Twisted, its tenets would be performance and getting it right. A part of getting things right, is knowing what wrong looks like. Both Twisted and (Stackless - what I work with), don't dedicate much time to showing what is definitely wrong. >The easiest place to start learning is with code that >looks like all the other code you've written before. I am not sure about this approach although there is much to be said about having a programming style that makes you feel confident to tackle problems. I find a good way to learn is looking at good but simple examples that do something useful, then tinker with them. This is the way some of my favourite computer books work - the various Kernighan/Richie/Plauger books, Steven's "Unix Network Programming" In the case of Twisted, I found the Abe Fettig book good - I looked at webcat.py and requesthandler.py and that gave me a start. From time to time, I also posted on this mailing list to get little examples - very helpful. I also like the design patterns approach to learning: this is the name, the situation/context, the problem, the solution, references. I think Twisted would benefit from this approach when one is dealing with common situations like say "call database from request handler" or "separate business logic from network logic." Cheers, Andrew ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping

3 2

[Twisted-Python] Re: [ANN] Twisted 8.0
by Steve Holden March 27, 2008

March 27, 2008

Colin Alston wrote: > On 27/03/2008 10:09 Duncan McGreggor wrote: >> On Wed, Mar 26, 2008 at 8:26 PM, Christopher Armstrong >> <radix(a)twistedmatrix.com> wrote: >>> http://twistedmatrix.com/ >>> >>> MASSACHUSETTS (DP) -- Version 8.0 of the Twisted networking framework >>> has been released, Twisted Matrix Laboratories announced Wednesday. >>> >>> Enslaved by his new robotic overloads, Master of the Release >>> Christopher Armstrong presented the new package to the Internet on >>> March 26th. Armstrong was unable to comment, because of a device worn >>> around his neck preventing him from doing so, scientists say. >> >> A reader (who fears the robotic overlords, and thus wishes to remain >> anonymous) has suggested that we do a full press release with this >> picture: >> http://chrisbishop.com/fineart/art-happycity.html > > I'm pleased to see Twisted is still one of the few Open Source projects > out there which still has a genuine sense of humour - and delivers > staggeringly high quality code in spite of it. > > Well done all involved - and thank you :) > Not in spite of it, because of it. A sense of humor is essential in retaining perspective as one continually fails to communicate one's meaning to the stupid machine. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/

1 0

[Twisted-Python] [ANN] Twisted 8.0
by Christopher Armstrong March 27, 2008

March 27, 2008

http://twistedmatrix.com/ MASSACHUSETTS (DP) -- Version 8.0 of the Twisted networking framework has been released, Twisted Matrix Laboratories announced Wednesday. Enslaved by his new robotic overloads, Master of the Release Christopher Armstrong presented the new package to the Internet on March 26th. Armstrong was unable to comment, because of a device worn around his neck preventing him from doing so, scientists say. Secretary of Defense Jean-Paul Calderone was asked about concerns that French interference may have played a role in the delay of this release. "I find such speculation preposterous. Thomas Hervé is an upstanding member of the Labs and his loyalties lie with us. He is a fine addition to our team." Rumors in the community allege that Secretary Calderone is holding Hervé's cat ransom until the release is successfully distributed. Hervé was unavailable for comment. This release comes shortly after the announcement by Chief of Public Affairs Duncan McGreggor that Twisted had joined the Software Freedom Conservancy. "We're happy to join the SFC, and we are now accepting sponsorship. The fact that we are now ruled by a cabal of robots should not put off potential donors. Our robotic overlords are running us at peak efficiency, so we can most effectively distribute The Love." Asked about the version number jump in this release, Commander-in-Chief Glyph Lefkowitz had the following to say: "Our benefactors have found our previous dice-rolling version number scheme to be inadequate, and have deigned to propose to us a more... logical system of versioning." ===== Twisted is an event-based framework for Internet applications which works on Python 2.3, 2.4, and 2.5. It can be downloaded from http://twistedmatrix.com/ Twisted 8.0 is a major feature release, with several new features and a great number of bug fixes. Some of the highlights follow. - The IOCP reactor is now much improved and many bugs have been resolved. - Twisted is now easy_installable. - Many improvements were made to Trial, Twisted's unit testing system. - A new memcache client protocol implementation was added. - So much more[1]! To see the full list of changes in its fifteen kilobytes of glory, see the release notes[1]. We welcome you to download and enjoy, and please file any bugs you find[2] and send comments to the mailing list[3]. Why the large version number bump? We've decided to switch to a time-based versioning scheme. "8.0" means the first release in 2008. [1] http://twistedmatrix.com/trac/browser/tags/releases/twisted-8.0.1/NEWS?form… [2] Register: http://twistedmatrix.com/trac/register New ticket: http://twistedmatrix.com/trac/newticket [3] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python Thanks! -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

2 1

[Twisted-Python] announcing allmydata.org "Tahoe", the Least-Authority Filesystem, v1.0
by zooko March 26, 2008

March 26, 2008

Folks: This is it! We're committing to backwards compatibility and active support for the v1.0 branch of Tahoe for the forseeable future. Tomorrow, http://allmydata.com will open its doors to new customers, relying on Tahoe for its backend. Regards, Zooko ANNOUNCING Allmydata.org "Tahoe", the Least-Authority Filesystem, v1.0 We are pleased to announce the release of version 1.0 of the "Tahoe" Least Authority Filesystem. The "Tahoe" Least Authority Filesystem is a secure, decentralized, fault-tolerant filesystem. All of the source code is available under a Free Software, Open Source licence (or two). This filesystem is encrypted and distributed over multiple peers in such a way it continues to function even when some of the peers are unavailable, malfunctioning, or malicious. A one-page explanation of the security and fault-tolerance properties that it offers is visible at: http://allmydata.org/source/tahoe/trunk/docs/about.html We believe that this version of Tahoe is stable enough to rely on as a permanent store of valuable data. The version 1 branch of Tahoe will be actively supported and maintained for the forseeable future, and future versions of Tahoe will retain the ability to read files and directories produced by Tahoe v1.0 for the forseeable future. This release of Tahoe will form the basis of the new consumer backup product from Allmydata, Inc. -- http://allmydata.com . This is the successor to Allmydata.org "Tahoe" Least Authority Filesystem v0.9, which was released March 13, 2008 [1]. Since v0.9 we've made the following changes: * Use an added secret for convergent encryption to better protect the confidentiality of immutable files, and remove the publically readable hash of the plaintext (ticket #365). * Add a "mkdir-p" feature to the WAPI (ticket #357). * Many updates to the Windows installer and Windows filesystem integration. Tahoe v1.0 produces files which can't be read by older versions of Tahoe, although files produced by Tahoe >= 0.8 can be read by Tahoe 1.0. The reason that older versions of Tahoe can't read files produced by Tahoe 1.0 is that those older versions require the file to come with a publically-readable hash of the plaintext, but exposing such a hash is a confidentiality leak, so Tahoe 1.0 does not do it. WHAT IS IT GOOD FOR? With Tahoe, you can distribute your filesystem across a set of computers, such that if some of the computers fail or turn out to be malicious, the filesystem continues to work from the remaining computers. You can also share your files with other users, using a strongly encrypted, capability-based access control scheme. Because this software is the product of less than a year and a half of active development, we do not categorically recommend it for the storage of data which is extremely confidential or precious. However, we believe that the combination of erasure coding, strong encryption, and careful engineering makes the use of this software a much safer alternative than common alternatives, such as RAID, or traditional backup onto a remote server, removable drive, or tape. This software comes with extensive unit tests [2], and there are no known security flaws which would compromise confidentiality or data integrity. (For all currently known security issues please see the Security web page: [3].) This release of Tahoe is suitable for the "friendnet" use case [4] -- it is easy to create a filesystem spread over the computers of you and your friends so that you can share files and disk space with one another. LICENCE You may use this package under the GNU General Public License, version 2 or, at your option, any later version. See the file "COPYING.GPL" for the terms of the GNU General Public License, version 2. You may use this package under the Transitive Grace Period Public Licence, version 1.0. The Transitive Grace Period Public Licence says that you may distribute proprietary derived works of Tahoe without releasing the source code of that derived work for up to twelve months, after which time you are obligated to release the source code of the derived work under the Transitive Grace Period Public Licence. See the file "COPYING.TGPPL.html" for the terms of the Transitive Grace Period Public Licence, version 1.0. (You may choose to use this package under the terms of either licence, at your option.) INSTALLATION Tahoe works on Linux, Mac OS X, Windows, Cygwin, and Solaris. For installation instructions please see "docs/install.html" [5]. HACKING AND COMMUNITY Please join us on the mailing list [6] to discuss uses of Tahoe. Patches that extend and improve Tahoe are gratefully accepted -- the RoadMap page [7] shows the next improvements that we plan to make and CREDITS [8] lists the names of people who've contributed to the project. The wiki Dev page [9] contains resources for hackers. SPONSORSHIP Tahoe is sponsored by Allmydata, Inc. [10], a provider of consumer backup services. Allmydata, Inc. contributes hardware, software, ideas, bug reports, suggestions, demands, and money (employing several allmydata.org Tahoe hackers and instructing them to spend part of their work time on this free-software project). We are eternally grateful! Zooko O'Whielacronx on behalf of the allmydata.org team March 25, 2008 San Francisco, California, USA [1] http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=2315 [2] http://allmydata.org/trac/tahoe/wiki/Dev [3] http://allmydata.org/trac/tahoe/wiki/Security [4] http://allmydata.org/trac/tahoe/wiki/UseCases [5] http://allmydata.org/source/tahoe/trunk/docs/install.html [6] http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev [7] http://allmydata.org/trac/tahoe/roadmap [8] http://allmydata.org/trac/tahoe/browser/CREDITS?rev=2345 [9] http://allmydata.org/trac/tahoe/wiki/Dev [10] http://allmydata.com

1 0

[Twisted-Python] Re: Twisted-Python Digest, Vol 48, Issue 41
by Andrew Francis March 25, 2008

March 25, 2008

Hi Don: [Don quote] Thanks to all the good folks who replied to this; good stuff to chew on. I particularly liked glyph's advice on when and where to use the inline callback style, and when/where to avoid it. I think I'll continue along the path I described, at least for a while. I may be back with more on this topic. ~ I saw your post but didn't have time to respond earlier.... I second Glyph's advice concerning inlineCallbacks.... I am building a WS-BPEL processor. WS-BPEL is about business logic and exotic message exchange patterns. I gave a talk at PyCon 2008 called "Adventures in Stackless Python Twisted Integration." My strategy is to separate the business logic as much as possible from the underlying networking. At the lower levels, I use Stackless Python in conjunction with Twisted. A side-effect of this approach is that I can block waiting for a Twisted call to finish. Right now, I am simplifying one of my prototypes so I can post it as a full-blown example of all the techniques I use. Give me a few days. To add to what others have said, the main problem I have encountered boil down to identifying classic concurrency problems. Unfortunately APIs and transitivity obscure bugs. I know this sounds obvious but no matter which concurrency model you use: event, pre-emptive threads, cooperate, you really have to understand how problems like say, deadlock and race conditions manifest themselves and take precautions. In the narrow field of Stackless and Twisted, I am trying to map out what those precautions are. Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

3 3

[Twisted-Python] Foolscap-0.2.5 released
by Brian Warner March 25, 2008

March 25, 2008

I'm pleased to announce the release of Foolscap-0.2.5. Foolscap is the successor to Perspective Broker, offering cryptographically-secure remote method invocation, flexible object-graph serialization, a capability-based object model, and more. Foolscap, for all your remote-object needs. Source distributions are available from the Foolscap home page, PyPI, and wherever fine remote-object products are sold: http://foolscap.lothar.com/trac http://foolscap.lothar.com/releases/foolscap-0.2.5.tar.gz http://foolscap.lothar.com/releases/foolscap-0.2.5.zip As always, Foolscap releases are signed by my GPG key (1514A7BD). In case you're wondering what happened to 0.2.4, or 0.2.3, you're right: there have been several unannounced releases. A lot of the recent work has gone into improvements in the logging subsystem (know as the "flog" system). It is easy for Foolscap-using programs to publish a 'logport', to which the 'flogtool tail' command can connect and watch log events scroll by. Log events can be gathered and inspected with command-line tools or a web-based browser. Future work is planned to enable cross-application causality tracing using the Causeway viewer. In addition to logging changes, several other improvements have been made. Duplication connection handling should be working better now. A number of sample programs have been added to the distribution. Default size limits have been removed from all Constraints. A complete set of release notes for the last six months of releases is attached below. All versions since 0.1.3 (released in May 2007) are wire-compatible. Foolscap is currently being deployed in the allmydata.org "Tahoe" least-authority grid filesystem, which improves the chances that wire-compatibility will be retained in the future. I am grateful to allmydata.com (my employer) for supporting Foolscap development. Please direct any questions to this, the twisted-devel mailing list. Bugs, patches, and feature requests should be placed on the Foolscap Trac, at http://foolscap.lothar.com/ . cheers, -Brian * Release 0.2.5 (25 Mar 2008) ** Compatibility All releases between 0.1.3 and 0.2.5 (inclusive) are fully wire-compatible. The new 'flogtool tail --catch-up' command requires a log publisher running 0.2.5 or later. 'flogtool tail' without the --catch-up option will work with earlier publishers. ** Licensing clarification Foolscap is distributed under the (very liberal) terms of the MIT license, which is the same license that Twisted uses. It's been like this since the beginning, but this is the first release to make this obvious by including a LICENSE file. ** foolscap.logging Changes 'flogtool tail' now has a --catch-up option, which prompts the remote publisher to deliver stored historical events to the subscribe, in proper sequential order. This allows you to connect to a process that has just done something interesting and grab a copy of the log events relevant to that event. 'flogtool tail' also has a --save-to option, which specifies a filename to which all captured events should be saved. This file can be processed further with 'flogtool dump', 'flogtool filter', or 'flogtool web-viewer'. This behaves much like the unix 'tee' utility, except that the saved data is recorded in a lossless binary format (whereas the text emitted to stdout is not particularly machine-readable). 'flogtool tail' and 'flogtool dump' both emit human-readable log messages by default. The --verbose option will emit raw event dictionaries, which contain slightly more information but are harder to read. 'flogtool create-gatherer' will create a log gatherer .tac file in a new working directory. This .tac file can be launched with 'twistd', the standard Twisted daemon-launching program. This is significantly easier to work with than the previous 'flogtool gather' command (which has been removed). The new --rotate option will cause the log-gatherer to switch to a new output file every N seconds. The --bzip option will make it compress the logfiles after rotating them. For example, a log gatherer that rotates and compresses log files once per day could be created and launched with: flogtool create-gatherer --rotate 86400 --bzip ./workdir (cd workdir && twistd -y gatherer.tac) ** New sample programs doc/listings/command-server.py and command-client.py are a pair of programs that let you safely grant access to a specific command. The server is configured with a command line to run, and a directory to run it from. The client gets a FURL: when the client is executed, the server will run its preconfigured command. The client gets to see stdout and stderr (and the exit status), but does not get to influence the command being run in any way. This is much like setting up an ssh server with a restricted command, but somewhat easier to configure. doc/listings/xfer-server.py and xfer-client.py are similar, but provide file transfer services instead of command execution. ** New Features Tub.setLocationAutomatically() will try to determine an externally-visible IP address and feed it to Tub.setLocation(). It does this by preparing to send a packet to a well-known public IP address (one of the root DNS servers) and seeing which network interface would be used. This will tend to find the outbound default route, which of course is only externally-visible if the host is externally-visible. Applications should not depend upon this giving a useful value, and should give the user a way to configure a list of hostname+portnumbers so that manually-configured firewalls, port forwarders, and NAT boxes can be dealt with. * Release 0.2.4 (28 Jan 2008) ** Compatibility All releases between 0.1.3 and 0.2.4 (inclusive) are fully wire-compatible. ** foolscap.logging Changes *** 'flogtool filter' command added This mode is used to take one event-log file and produce another with a subset of the events. There are several options to control the filtering: "--strip-facility=foolscap" would remove all the foolscap-related messages, and "--after=start --before=end" will retain events that occur within the given period. The syntax is still in flux, expect these options to change in the next few releases. The main idea is to take a very large logfile and turn it into a smaller, more manageable one. *** error logging Applications should avoid recording application-specific instances in log events. Doing so will forces the log viewer to access the original source code. The current release of foolscap uses pickle, so such instances will be loadable if the viewer can import the same code, but future versions will probably switch to using Banana, at which point trying to log such instances will cause an error. In this release, foolscap stringifies the type of an exception/Failure passed in through the failure= kwarg, to avoid inducing this import dependency in serialized Failures. It also uses the CopiedFailure code to improve portability of Failure instances, and CopiedFailures have been made pickleable. The preferred way to log a Failure instance is to pass it like so: def _oops(f): log.msg("Oh no, it failed", failure=f, level=log.BAD) d.addErrback(_oops) Finally, a 'log.err()' method was added, which behaves just like Twisted's normal log.err(): it can be used in a Deferred errback, or inside an exception handler. *** 'flogtool web-viewer' Adding a "sort=time" query argument to the all-events viewing page URL will turn off the default nested view, and instead will sort all events strictly by time of generation (note that unsynchronized clocks may confuse the relative ordering of events on different machines). "sort=number" sorts all events by their event number, which is of dubious utility (since these numbers are only scoped to the Tub). "sort=nested" is the default mode. The web-viewer now provides "summary views", which show just the events that occurred at a given severity level. Each event is a hyperlink to the line in the all-events page (using anchor/fragment tags), which may make them more convenient to bookmark or reference externally. * Release 0.2.3 (24 Dec 2007) ** Compatibility All releases between 0.1.3 and 0.2.3 (inclusive) are fully wire-compatible. ** Bug Fixes RemoteReference.getSturdyRef() didn't work (due to bitrot). It has been fixed. ** foolscap.logging Changes This release is mostly about flogging improvements: some bugs and misfeatures were fixed: *** tolerate '%' in log message format strings Dictionary-style kwarg formatting is now done with a twisted-style style format= argument instead of happening implicitly. That means the acceptable ways to call foolscap.logging.log.msg are: log.msg("plain string") log.msg("no args means use 0% interpolation") log.msg("pre-interpolated: %d apples" % apples) log.msg("posargs: %d apples and %d oranges", apples, oranges) log.msg(format="kwargs: %(numapples)d apples", numapples=numapples) The benefit of the latter two forms are that the arguments are recorded separately in the event dictionary, so viewing tools can filter on the structured data, think of something like: [e for e in allEvents if e.get("numapples",0) > 4] *** log facility names are now dot-separated, to match stdlib logging *** log levels are derived from numerical stdlib logging levels *** $FLOGFILE to capture flog events during 'trial' runs One challenge of the flogging system is that, once an application was changed to write events to flogging instead of twisted's log, those events do not show up in the normal places where twisted writes its logfiles. For full applications this will be less of an issue, because application startup will tell flogging where events should go (flogging is intended to supplant twisted logging for these applications). But for events emitted during unit tests, such as those driven by Trial, these events would get lost. To address this problem, the 0.2.3 flogging code looks for the "FLOGFILE" environment variable at import time. This specifies a filename where flog events (a series of pickled event dictionaries) should be written. The file is opened at import time, events are written during the lifetime of the process, then the file is closed at shutdown using a Twisted "system event trigger" (which happens to be enough to work properly under Trial: other environments may not work so well). If the FLOGFILE filename ends in .bz2, the event pickles will be compressed, which is highly recommended because it can result in a 30x space savings (and e.g. the Tahoe unit test run results in 90MB of uncompressed events). All 'flogtool' modes know how to handle a .bz2 compressed flogfile as well as an uncompressed one. The "FLOGTWISTED" environment variable, if set, will cause this same code to bridge twisted log messages into the flogfile. This makes it easier to see the relative ordering of Twisted actions and foolscap/application events. (without this it becomes very hard to correlate the two sources of events). The "FLOGLEVEL" variable specifies a minimum severity level that will be put into the flogfile. This defaults to "1", which puts pretty much everything into the file. The idea is that, for tests, you might as well record everything, and use the filtering tools to control the display and isolate the important events. Real applications will use more sophisticated tradeoffs between disk space and interpretation effort. The recommended way to run Trial on a unit test suite for an application that uses Foolscap is: FLOGFILE=flog.out FLOGTWISTED=1 trial PACKAGENAME Note that the logfile cannot be placed in _trial_temp/, because trial deletes that directory after flogging creates the logfile, so the logfile would get deleted too. Also note that the file created by $FLOGFILE is truncated on each run of the program. * Release 0.2.2 (12 Dec 2007) ** Compatibility All releases between 0.1.3 and 0.2.2 (inclusive) are fully wire-compatible. New (optional) negotiation parameters were added in 0.2.1 (really in 0.2.0). ** Bug Fixes The new duplicate-connection handling code in 0.2.1 was broken. This release probably fixes it. There were other bugs in 0.2.1 which were triggered when a duplicate connection was shut down, causing remote calls to never be retired, which would also prevent the Reconnector from doing its job. These should be fixed now. ** Other Changes Foolscap's connection-negotiation mechanism has been modified to use foolscap logging ("flog") instead of twisted.log . Setting the FLOGFILE= environment variable will cause a Foolscap-using program to write pickled log events to a file of that name. This is particularly useful when you want to record log events during 'trial' unit test run. The normal API for setting this file will be added later. The FLOGTWISTED= environment variable will cause the creation of a twisted.log bridge, to copy all events from the twisted log into the foolscap log. The 'flogtool web-view' mode has been enhanced to color-code events according to their severity, and to format Failure tracebacks in a more-readable way. * Release 0.2.1 (10 Dec 2007) ** Compatibility All releases between 0.1.3 and 0.2.1 (inclusive) are fully wire-compatible. 0.2.1 introduces some new negotiation parameters (to handle duplicate connections better), but these are ignored by older versions, and their lack is tolerated by 0.2.1 . ** New Features *** new logging support Foolscap is slowly acquiring advanced diagnostic event-logging features. See doc/logging.xhtml for the philosophy and design of this logging system. 0.2.1 contains the first few pieces, including a tool named bin/flogtool that can be used to watch events on a running system, or gather events from multiple applications into a single place for later analysis. This support is still preliminary, and many of the controls and interfaces described in that document are not yet implemented. *** better handling of duplicate connections / NAT problems The connection-management code in 0.1.7 and earlier interacted badly with programs that run behind NAT boxes (especially those with aggressive connection timeouts) or on laptops which get unplugged from the network abruptly. Foolscap seeks to avoid duplicate connections, and various situtations could cause the two ends to disagree about the viability of any given connection. The net result (no pun intended) was that a client might have to wait up to 35 minutes (depending upon various timeout values) before being able to reestablish a connection, and the Reconnector's exponential backoff strategy could easily push this into 90 minutes of downtime. 0.2.1 uses a different approach to accomplish duplicate-suppression, and should provide much faster reconnection after netquakes. To benefit from this, both ends must be running foolscap-0.2.1 or newer, however there is an additional setting (not enabled by default) to improve the behavior of pre-0.2.1 clients: tub.setOption("handle-old-duplicate-connections", True). *** new Reconnector methods The Reconnector object (as returned by Tub.connectTo) now has three utility methods that may be useful during debugging. reset() drops the backoff timer down to one second, causing the Reconnector to reconnect quickly: you could use this to avoid an hour-long delay if you've just restarted the server or re-enabled a network connection that was the cause of the earlier connection failures. getDelayUntilNextAttempt() returns the number of seconds remaining until the next connection attempt. And getLastFailure() returns a Failure object explaining why the last connection attempt failed, which may be a useful diagnostic in trying to resolve the connection problems. ** Bug Fixes There were other minor changes: an OS-X unit test failure was resolved, CopiedFailures are serialized in a way that doesn't cause constraint violations, and the figleaf code-coverage tools (used by foolscap developers to measure how well the unit tests exercise the code base) have been improved (including an emacs code-used/unused annotation tool).

1 0

[Twisted-Python] Controlling twisted via stdin
by Brian Baquiran March 25, 2008

March 25, 2008

How do I use the reactor.addReader/Writer methods? I would like to control a twisted server using stdin. Does data on stdin get passed to a protocol's dataReceived? Thanks. Brian

2 1

[Twisted-Python] Re: Twisted-Python Digest, Vol 48, Issue 41
by Andrew Francis March 24, 2008

March 24, 2008

Hi Don: [Don quote] Thanks to all the good folks who replied to this; good stuff to chew on. I particularly liked glyph's advice on when and where to use the inline callback style, and when/where to avoid it. I think I'll continue along the path I described, at least for a while. I may be back with more on this topic. ~ I saw your post but didn't have time to respond earlier.... I second Glyph's advice concerning inlineCallbacks.... I am building a WS-BPEL processor. WS-BPEL is about business logic and exotic message exchange patterns. I gave a talk at PyCon 2008 called "Adventures in Stackless Python Twisted Integration." My strategy is to separate the business logic as much as possible from the underlying networking. At the lower levels, I use Stackless Python in conjunction with Twisted. A side-effect of this approach is that I can block waiting for a Twisted call to finish. Right now, I am simplifying one of my prototypes so I can post it as a full-blown example of all the techniques I use. Give me a few days. To add to what others have said, the main problem I have encountered boil down to identifying classic concurrency problems. Unfortunately APIs and transitivity obscure bugs. I know this sounds obvious but no matter which concurrency model you use: event, pre-emptive threads, cooperate, you really have to understand how problems like say, deadlock and race conditions manifest themselves and take precautions. In the narrow field of Stackless and Twisted, I am trying to map out what those precautions are. Cheers, Andrew ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

1 0