On Sunday 28 May 2006 13:39, Phil Mayers wrote: ...
All that said, the BBC Kamaelia project uses a generator-based consumer/producer pipeline as its underlying primitive, and seems to get along fine with it. But a consumer/producer component is not the same as a lightweight thread of course - broadly a c/p will only ever interact with its input, local variables, output and library calls. A thread might be expected to interact with other threads and shared data, from whence all difficulty springs.
</delurk> [ Since someone else raised us, I'd like to say a couple of things, since I think one is related to the conversation in hand, and the other is relevant to a conversation started at Europython last year about our work feeding into Twisted in a good way (ie a way useful to people using Twisted). ] Hopefully both points are useful. Given we're now churning out systems at work with Kamaelia which are getting /close/ to what I'd be happy with as production systems, so it //seems// to be working out OK so far as you say. One thing that's less obvious about the term consumer producer (even though its correct) is people tend to think pipelines, and many systems don't tend to be (we've invented the term graphlines instead, for want of a better name). As part of getting to that stage - getting back to why I was replying - as well as pure generators, we also properly support threaded components now (albeit with a couple of caveats). The only difference between those and standard components is the baseclass you inherit from and the fact your main method doesn't have a yield statement in it. Threaded components are also useful for forcing true system quiescence. This also gives a fairly simple migration route for code for integration. This also means bridging sync and async worlds is pretty trivial - for example a crappy interactive web-tester application can look like this: (would be nicer to use sys.argv...) pipeline( ConsoleReader(), # This uses raw_input in a thread. TCPClient("www.google.com", 80), # Async and uses a selector component # running in another thread so the # system can be quiescent ConsoleEchoer(), # Normal Async ).run() (The reason TCPClient fires up a selector component is because it expects to be in a situation where lots of networking stuff is happening. If it's not the CPU/memory load addition is minimal) There's more useful examples than the above, but I quite like the above because of simplicity. (The system we've got for dumping DVB-T transport streams, and transcoding them is much more fun, but longer winded :) On the second point... I think it's worth mentioning that our goal of trying to make it possible to integrate our code with twisted by Europython this year was woefully optimistic (we've been spending time on optimisations instead, which will probably actually simplify integrating with twisted). (Integrating as in "work well with", not as in "viewed as a good idea by everyone involved with twisted :)" ) [[ Sorry Tommi, I know there were aspects you wanted to nick... :-( ]] <relurk> Best Regards, Michael. -- Michael Sparks, Senior Research Engineer, BBC Research, Technology Group michael.sparks@rd.bbc.co.uk, Kamaelia Project Lead, http://kamaelia.sf.net/