Hi, I've just come across this thread (unsubbed from the twisted list during EuroOSCON, just resubbed after a) getting back b) fixing my machine from a reiser corruption...). Much as Johann Borck in the thread said "I feel like i have to make a statement ...", I feel it's probably worth putting in a tuppence worth on the Kamaelia comments in the thread. (If only to try and avoid negative misunderstandings. I suck though when I try to avoid misunderstandings though, so here's fingers crossed I don't cause any here ... ) Also, sorry for the length of this if it's too long. I could reply to the comments in thread, but that might just cause a thread explosion. If this is off topic, just delete. Next paragraph probably is probably a sufficient summary :-) Some clarification. First of all, I don't see Kamaelia as in competition with Twisted. Indeed, Kamaelia should be guttable (people are welcome to do so), and we'd rather collaborate rather than compete. (I've never understood the desire to compete in some open source projects, except on a "for fun"/variety basis) At all talks longer than my ill-advised lightning talk at Europython 2004 I've made it patently clear that I think that Twisted is best of breed in python for what it does. And not only that, damn fine at it as well. The architecture in Kamaelia is however borne from a) working with architectures similar to, but not quite identical to twisted and wondering if b) *maybe* there's a better way. I'm not saying Kamaelia *is* that better way (I'm too close to it), just an alternate approach. That said, if you're trying an alternate approach, starting from scratch is one way of doing so, and that's the simple reason we're not using Twisted :-) After my talk at Europython 2004 I realised to my dismay that some people mistook my viewpoint of *maybe* there being a better way (badly expressed then as well :-( ) as being "Twisted sucks/is bad", which is far from the truth. Taking some specific points from the thread, which refer to the various projects: Nicola Larosa nico at tekNico.net wrote:

Why isn't Twisted seen, by these projects at least, as a viable tool? Is the event-driven Python world undergoing the same fragmentation as the web framework one?

1: Don't ask on the twisted list, ask the people who are running those projects :-) (I only discovered the thread because I was resubbing and wondered if I'd missed something) 2: We do view twisted as a viable tool. We're just trying something different. The "R" part of BBC R&D suggests that trying something different might occasionally be a good idea :-) (Then again it might not, experience is something you get after you needed it (maybe:) ) 3: Kamaelia is much more dataflow driven between active objects rather than viewing the system as event driven. Sure, data delivery/receipt are clearly events, but that's used for optimisation rather than the core idea. (It's also based on recognising that some ways of viewing network systems look very similar to asynchronous hardware design) Maybe they're just different mappings on the same problem space. If so, then maybe the different mapping might be a useful different way of looking at the problem. **Maybe**. It's an interesting comment from Freevo's changelog: "It is a huge monster and hard to understand" I think there's an element of truth in that to //any// sufficiently useful framework or library. It'd be nice if elements of Kamaelia were able to be gutted into twisted, *IF* they'd help with avoiding the above problem. (*IF*) (A core aim in Kamaelia is to allow novices to get up to learn the core quickly) Stefan Behnel wrote:

Kamaelia ... is not as 'feature-complete'. So it may get ugly once it implements more features. That's a general problem

Indeed, this is possible. However, one interesting thing has shown up recently - we appear to be converging towards something similar to deferred generators. Specific example - ask the scheduler to run a different generator in the place of the current generator, and return control when done: yield WaitComplete( self.requestDisplay(DISPLAYREQUEST=True, callback = (self,"control"), size = (width, height), position = self.position ) ) self.requestDisplay returns a generator, WaitComplete creates an Axon.Ipc.WaitComplete message, which is passed to the scheduler which runs it instead of the main component. This then isn't really a huge step to deferreds/defgens. (If I understand those right) I'm not totally happy with the syntax above, and it's likely to change, but it's an initial step. Itamar Shtull-Trauring itamar at itamarst.org

kamaelia: an experimental platform for what I hear are some interesting ideas we should look at;

Please do. Tell us what sucks, take the good stuff. We'll then try and make the sucky stuff not suck and then take that :-) Itamar Shtull-Trauring itamar at itamarst.org

probably would be a good candidate for running on top of Twisted since they seem to deal with higher levels of abstraction mostly.

Agreed. Not immediately though, we're aiming for around 9-12 months from now. We should either be running on top of twisted at that point, or side by side with, or the other way round, or Kamaelia subsumed by something else. Antoine Pitrou solipsis at pitrou.net

(OTOH, I very briefly looked at Kamaelia's introduction doc and I don't find it attractive - this kind of dataflow-programming model looks quite unflexible)

I don't suppose you could follow up on that on the Kamaelia list could you? Moe Aboulkheir moe at divmod.com

[ on PEP 342 potentially removing the reliance on frameworks for accomplishing clean asychronous IO - libraries whose clients accrue a dependency on a non-standard reactor/event loop/runtime/whatever in order to operate. kamaelia probably doesn't fit into this category, i wouldn't know. ]

I think PEP 342 has the potential to simplify everyone's systems. At one point Kamaelia components could just be run like any other generator. Since then the dependency on the scheduler has grown somewhat. Prior to seeing this post I'd already decided that we needed to regrow the earlier simplicity to allow easier cherry picking by other projects. Also, personally I'd hope that *if* a similar transform happening in Twisted, PEP342 may result in is a WSGI for events type situation. That would probably result in most networked apps being powered by Twisted on some level, maybe composed in a Kamaelia style way, maybe not. (I've noted the twisted community have a knack for taking an idea, turning it on it's head and doing something cool with it :-) ) glyph at divmod.com glyph at divmod.com:

Also, it gives the Twisted team some potential competition to focus on. Possibly a source of motivation, and besides, everybody loves a race :).

As long as it's a "Wacky Races" style of competition, I'm happy with Kamaelia being competition. (Seriously though, Kamaelia is from a "what if", and hopefully scavengeable - either now or in future)

Kamaelia seems more promising to me; I imagine that they might produce some serious competition eventually if that's what they aim to do.

Only in terms of approach. Any sufficiently useful open source project is easily consumable by another. (Being served up for dinner and eaten is the ultimate success IMO) I do aim it to be a useful alternative way of thinking about problems similar to those that twisted thinks about. I saw your post here http://www.livejournal.com/users/glyf/41709.html , and it struck me as interesting. Essentially part of the comments appear to reuse of generators, which strikes a chord (largely because that's how hardware would have to work, and Kamaelia's core is based on asynchronous hardware design[*] ). Again, if we do head down that route (depends on resources, etc), then I'd hope people feel happy to rip out code where needed. [*] Whilst Kamaelia has no chance at present of compiling to silicon, things like MyHDL give me hope that it could be possible someday, and in that scenario, you need a system similar to the one you described.

However, they claim that only the "Axon" part of their project is stable, and that doesn't yet include anything that does select().

We consider Axon to be API stable (ie stuff that's there is not likely to be removed). Axon is designed to be a small framework for handling concurrency, communications and composition between (mainly) generators, and threads have recently been introduced as well. By itself, Axon is intended to be very, very boring. The instant it's not being boring (such as doing a select, or poll, or ...) it's doing too much :-) As a result Axon's version number reflects what we think about it's stability. The reason for the jump from 1.0.X to 1.1.X was due to threaded components. (These were added earlier than expected due to a need for threaded components on Nokia mobiles, and have a suboptimal implementation and API right now.) The bulk of Kamaelia however is a collection of components, which is growing and we can do a large variety of Stuff. However, we still think (despite having used Kamaelia now for some time), that the collection of components we have at present is incomplete (around 30% of what would be needed generally speaking), and that the API for the components (specifically which inboxes/outboxes they have, names of components, file hierarchy etc), is still in flux. This collection of Stuff is being driven by application spikes, in application areas which are off the beaten track from the project's main goal (to round out the system), as well as in the main project goal area. So far this is proving to be an interesting approach. As a result we do have things like TCP server/client components, a selector component (which advertises itself as a service internally) which manages a select subsystem (which we're thinking of farming off into a thread to allow the entire system to sleep waiting for it), pygame Stuff, Tk Stuff, and so on. The latest application spike was a simple paint program for example, which was useful for testing a variety of subsystems. Similarly for the presentation tool spike. However, the low version number (currently) 0.3.0 is really to act as a warning "we still expect quite a lot to change before we hit 1.0.0". I could wax lyrical on that, but will only do so if people are interested :) However the reason it's 0.3.0, is because at minimum: * We want to have sensible integration with twisted (ie Kamaelia usable in twisted or vice versa) when we hit 1.0.0 * There's a whole bunch of core ideas we want to extend before then, including the graphical editting toolset. * We know we're just skimming the surface of possible components and systems (despite the amount of stuff we /can/ do). The component set needs to be fleshed out more before we know if the approach has worth. (It feels like it to me at present, but then I'm biassed) That said, generally, things are more stable than unstable, though currently I prefer to play up the instability to allow more freedom to experiment with APIs and ideas in the short/medium term. Anyway, I hope that this mail assures everyone on the twisted list (who was worried - struck me as a minority) that we're not really in /competition/, but more looking at providing /options/ for everyone, including Twisted :-) So far we're finding our approach useful and fun, which at the end of the day is all that matters. (For work the former is important, if it was just my personal time, then it could be either :-) Best Regards, Michael. -- Michael.Sparks@rd.bbc.co.uk, http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated.