[Twisted-Python] flow module release canidate and future schedule
Hello. Over this last week, quite alot of work has been put into the flow module. As of today, I have other responsibilities which I must attend... but flow is being used in one of my internal projects at Axista, Inc. and is quite stable. I encourage people to try it out.. it's quite nice if I don't say so myself. What is flow? Flow is a module which leverages generators to provide for incremental result multi-stage operations (like http requests) which can cooperate with other operations without blocking. See sandbox/flow.html for more information Current Status: 0. The documentation (flow.html) is now much more clear and "up-to-date" with respect to newer changes. Please take it for a spin. All of the examples are full-fledged programs and should run. 1. Flow is quite faster now, about 2-3x from the previous versions. This is due to some significant profiling and an enhancement proposed by exarkun; in particular, each yield can support returning a row set rather than a single row. This is a boon for database applications. 2. Flow supports protocols with one callback, which should handle most protocol requirements. An echo 'client' and echo 'server' are included in sandbox/cce/echo.py 3. Several 'helpers' such as Merge, Zip, are now available and well tested (Merge isn't perfect yet though). 4. Flow supports not only deferreds, but also callbacks; this is flow.Callback, which is used, not suprizingly, by flow.Protocol 5. The "Instruction" mechanism has been revamped to support 'CallLater' instructions. This is an internal refactor which cleaned up alot of code. 6. The test suite has been greatly expanded to test these items. Future Directions: 1. As I start playing with more protocols, I'll probably add a few 'filters' which would take a raw flow.Protocol and return items by line or by a specified chunk size. 2. The 'Merge' code needs some reviewing as it isn't perfect and doesn't behave quite right yet. I don't have any serious use case for this right now, so it may not get fixed for another month or so. 3. Eventually (probably late this year) I'll consider writing flow as a "C" module for speed. This should be about 2-3x faster in most cases. So, if you are not using flow beacuse of speed concerns, keep this in mind and let me know. Best, Clark
On Wednesday, May 28, 2003, at 16:55 America/New_York, Clark C. Evans wrote:
Future Directions:
1. As I start playing with more protocols, I'll probably add a few 'filters' which would take a raw flow.Protocol and return items by line or by a specified chunk size.
Take a look at the example I posted to the mailing list last Friday for an example of how I did this (per-yield chunk size, not delimiter based). It greatly simplified my gzip/gunzip code. Also, the "byte pipe" I'm using can surely be accelerated by using iovec (in sandbox/exarkun), I basically rewrote it and extended it so it could accomodate pipe usage.
2. The 'Merge' code needs some reviewing as it isn't perfect and doesn't behave quite right yet. I don't have any serious use case for this right now, so it may not get fixed for another month or so.
3. Eventually (probably late this year) I'll consider writing flow as a "C" module for speed. This should be about 2-3x faster in most cases. So, if you are not using flow beacuse of speed concerns, keep this in mind and let me know.
If I end up finding any significant use for flow, which I almost definitely will, I'll help you out with that. -bob
participants (2)
-
Bob Ippolito
-
Clark C. Evans