Mailman 3 October 2005 - Twisted

[Twisted-Python] Weekly Bug Summary
by James Y Knight 26 Oct '05

26 Oct '05

Bug summary ______________________ Since 2005-10-19 06:00:02.200000: Bugs opened: 7 Bugs closed: 3 Total open bugs: 482 (+4) Urgent: 3 (+0) Bug: 235 (+2) Feature: 166 (+2) Wish: 78 (+0) New / Reopened Bugs ______________________ ==== Urgent ==== [DOCUMENTATION, TRIAL] Trial documentation (testing.xhtml) is wildly inaccurate (2 days ago) CLOSED http://twistedmatrix.com/bugs/issue1281 opened by jml ==== Bug ==== [BACKPORT, WORDS] UnboundLocalError in twisted.words.tap (yesterday) http://twistedmatrix.com/bugs/issue1283 opened by mike [TRIAL] twisted.trial.test.test_util.TestFindObject.test_importModule fails intermittently (yesterday) http://twistedmatrix.com/bugs/issue1285 opened by jml [BACKPORT, CONCH, DOCUMENTATION] conch examples index is all messed up (yesterday) CLOSED http://twistedmatrix.com/bugs/issue1284 opened by exarkun iocpreactor/udp.py disconnects udp sockets on writeErr (6 days ago) http://twistedmatrix.com/bugs/issue1279 opened by ghazel ==== Feature ==== [DOCUMENTATION] Draft "deeper concepts" document for interfaces (2 days ago) http://twistedmatrix.com/bugs/issue1282 opened by jml Win32Gtk2Reactor (2 days ago) http://twistedmatrix.com/bugs/issue1280 opened by dialtone Bugs Closed ______________________ ==== Urgent ==== [DOCUMENTATION, TRIAL] Trial documentation (testing.xhtml) is wildly inaccurate (2 days ago) http://twistedmatrix.com/bugs/issue1281 opened by jml, closed by jml ==== Bug ==== [BACKPORT, CONCH, DOCUMENTATION] conch examples index is all messed up (yesterday) http://twistedmatrix.com/bugs/issue1284 opened by exarkun, closed by exarkun win32eventreactor goes crazy at 64 connections (2 weeks ago) http://twistedmatrix.com/bugs/issue1260 opened by ghazel, closed by justinj

1 0

[Twisted-Python] Twisted - SQLite and SQLObject
by Michael Schneider 25 Oct '05

25 Oct '05

Can I used twisted with embedded databases such as SQLite? I would like to use SQLObject and SQLite in my twisted application. Thanks Mike

5 5

[Twisted-Python] Life cycle of ApplicationServices
by Eric Basley 25 Oct '05

25 Oct '05

I'm writting a FIX (Financial Information Exchange) application with Twisted with the target to compete with quickfix. I'm very close, everything is perfect except the way to control life cycle of services. Let's take an example : The FIX main application is an engine able to manage many FIX sessions and is controled by web services. A session may act as a TCP server or client. Sessions have starting time and stopping time. For a client session I'm using a protocol.ReconnectingClientFactory. With Twisted-2.0.1 there is no way to stop properly a service and there is no way to destroy (factory, protocol, connector, ...) associated to the service. Does someone have solutions about clean life cycle management ? regards. eric basley

2 1

[Twisted-Python] question about threading
by Phil Christensen 25 Oct '05

25 Oct '05

hey folks, i've got a quick question for anyone that can help me out. i'm also in the process of looking this up in the twisted docs/code, but this is some last minute coding for a big demo tomorrow, so i'm hedging my bets by posting here. i've got a relatively simple async twisted server that has a custom protocol. most everything the server needs to do on a receipt of a message is short running, but i have one task that isn't. i need to spawn a thread that executes a couple of binaries (html2ps and ps2pdf) which are generally long-running. after executing those binaries, i need to send a message back to the client that sent the original command to generate the pdf. i tried just creating a basic thread that holds onto a reference to the client object, but it doesn't seem to send the command back to the client until i send another message to the server (any kind of message). i know there's a bunch of reactor methods that deal with threads, but i've never used them before, and i'm not sure which one will fix this issue. here's my thread subclass: class PDFGenerator(threading.Thread): def __init__(self, client): threading.Thread.__init__(self) self.client = client def run(self): pres = event.getActivePresentation() t = datetime.datetime.now() base_name = 'sample-file-name' out_html = file('/tmp/' + base_name + '.html', 'w') # [snip snip snip] # write some html to the file out_html.close() os.system('html2ps /tmp/' + base_name + '.html > /tmp/' + base_name + '.ps') # > /tmp/' + base_name + '_ps.log') os.system('mkdir data/transcripts/' + str(pres.id)) os.system('ps2pdf /tmp/' + base_name + '.ps data/ transcripts/' + str(pres.id) + '/' + base_name + '.pdf > /tmp/' + base_name + '_pdf.log') lock = threading.Lock() lock.acquire() self.client.sendCommand('presentTranscript', ['/ transcripts/' + str(pres.id) + '/' + base_name + '.pdf']) lock.release() the client object holds a reference to the protocol object, and sendCommand basically just executes: print "Sending " + message + " to " + str(self.protocol.source) self.protocol.transport.write(message + "\0") i made sure to set the lock, but i think the problem is in how i'm using threads. the clue i'm seeing is that in the log, i see the following: 2005/10/25 10:40 EDT [SSProtocol,25,127.0.0.1] Got generateTranscript ('') from IPv4Address(TCP, '127.0.0.1', 49179) 2005/10/25 10:40 EDT [-] Sending presentTranscript,/transcripts/ phil_bubblehouse.org_2005-10-25-10-40-13.pdf to IPv4Address(TCP, '127.0.0.1', 49179) you can see that the second log entry doesn't print the protocol object that's generating the message. i know there's something simple i'm not doing; any help would be appreciated. -phil

4 4

[Twisted-Python] ssh tunnel
by Yves Glodt 25 Oct '05

25 Oct '05

Hi, I'm new to conch and I wonder if somebody could point me to an example of how to create an ssh tunnel with conch to forward a connection (e.g. database or vnc) through that tunnel (if that's possible at all...) Thanks in advance and best regards, Yves

1 0

[Twisted-Python] Re: Freevo 2.0, Kamaelia, pyevent, Eventnet/LGT: what's going on?
by Michael 24 Oct '05

24 Oct '05

Hi, I've just come across this thread (unsubbed from the twisted list during EuroOSCON, just resubbed after a) getting back b) fixing my machine from a reiser corruption...). Much as Johann Borck in the thread said "I feel like i have to make a statement ...", I feel it's probably worth putting in a tuppence worth on the Kamaelia comments in the thread. (If only to try and avoid negative misunderstandings. I suck though when I try to avoid misunderstandings though, so here's fingers crossed I don't cause any here ... ) Also, sorry for the length of this if it's too long. I could reply to the comments in thread, but that might just cause a thread explosion. If this is off topic, just delete. Next paragraph probably is probably a sufficient summary :-) Some clarification. First of all, I don't see Kamaelia as in competition with Twisted. Indeed, Kamaelia should be guttable (people are welcome to do so), and we'd rather collaborate rather than compete. (I've never understood the desire to compete in some open source projects, except on a "for fun"/variety basis) At all talks longer than my ill-advised lightning talk at Europython 2004 I've made it patently clear that I think that Twisted is best of breed in python for what it does. And not only that, damn fine at it as well. The architecture in Kamaelia is however borne from a) working with architectures similar to, but not quite identical to twisted and wondering if b) *maybe* there's a better way. I'm not saying Kamaelia *is* that better way (I'm too close to it), just an alternate approach. That said, if you're trying an alternate approach, starting from scratch is one way of doing so, and that's the simple reason we're not using Twisted :-) After my talk at Europython 2004 I realised to my dismay that some people mistook my viewpoint of *maybe* there being a better way (badly expressed then as well :-( ) as being "Twisted sucks/is bad", which is far from the truth. Taking some specific points from the thread, which refer to the various projects: Nicola Larosa nico at tekNico.net wrote: > Why isn't Twisted seen, by these projects at least, as a viable tool? Is > the event-driven Python world undergoing the same fragmentation as the web > framework one? 1: Don't ask on the twisted list, ask the people who are running those projects :-) (I only discovered the thread because I was resubbing and wondered if I'd missed something) 2: We do view twisted as a viable tool. We're just trying something different. The "R" part of BBC R&D suggests that trying something different might occasionally be a good idea :-) (Then again it might not, experience is something you get after you needed it (maybe:) ) 3: Kamaelia is much more dataflow driven between active objects rather than viewing the system as event driven. Sure, data delivery/receipt are clearly events, but that's used for optimisation rather than the core idea. (It's also based on recognising that some ways of viewing network systems look very similar to asynchronous hardware design) Maybe they're just different mappings on the same problem space. If so, then maybe the different mapping might be a useful different way of looking at the problem. **Maybe**. It's an interesting comment from Freevo's changelog: "It is a huge monster and hard to understand" I think there's an element of truth in that to //any// sufficiently useful framework or library. It'd be nice if elements of Kamaelia were able to be gutted into twisted, *IF* they'd help with avoiding the above problem. (*IF*) (A core aim in Kamaelia is to allow novices to get up to learn the core quickly) Stefan Behnel wrote: > Kamaelia ... is not as 'feature-complete'. So it may get ugly once it > implements more features. That's a general problem Indeed, this is possible. However, one interesting thing has shown up recently - we appear to be converging towards something similar to deferred generators. Specific example - ask the scheduler to run a different generator in the place of the current generator, and return control when done: yield WaitComplete( self.requestDisplay(DISPLAYREQUEST=True, callback = (self,"control"), size = (width, height), position = self.position ) ) self.requestDisplay returns a generator, WaitComplete creates an Axon.Ipc.WaitComplete message, which is passed to the scheduler which runs it instead of the main component. This then isn't really a huge step to deferreds/defgens. (If I understand those right) I'm not totally happy with the syntax above, and it's likely to change, but it's an initial step. Itamar Shtull-Trauring itamar at itamarst.org > kamaelia: an experimental platform for what I hear are some interesting > ideas we should look at; Please do. Tell us what sucks, take the good stuff. We'll then try and make the sucky stuff not suck and then take that :-) Itamar Shtull-Trauring itamar at itamarst.org > probably would be a good candidate for running > on top of Twisted since they seem to deal with higher levels of > abstraction mostly. Agreed. Not immediately though, we're aiming for around 9-12 months from now. We should either be running on top of twisted at that point, or side by side with, or the other way round, or Kamaelia subsumed by something else. Antoine Pitrou solipsis at pitrou.net > (OTOH, I very briefly looked at Kamaelia's introduction doc and I don't > find it attractive - this kind of dataflow-programming model looks quite > unflexible) I don't suppose you could follow up on that on the Kamaelia list could you? Moe Aboulkheir moe at divmod.com >[ on PEP 342 potentially removing the reliance on frameworks for > accomplishing clean asychronous IO - libraries whose clients accrue > a dependency on a non-standard reactor/event loop/runtime/whatever > in order to operate. kamaelia probably doesn't fit into this category, i > wouldn't know. ] I think PEP 342 has the potential to simplify everyone's systems. At one point Kamaelia components could just be run like any other generator. Since then the dependency on the scheduler has grown somewhat. Prior to seeing this post I'd already decided that we needed to regrow the earlier simplicity to allow easier cherry picking by other projects. Also, personally I'd hope that *if* a similar transform happening in Twisted, PEP342 may result in is a WSGI for events type situation. That would probably result in most networked apps being powered by Twisted on some level, maybe composed in a Kamaelia style way, maybe not. (I've noted the twisted community have a knack for taking an idea, turning it on it's head and doing something cool with it :-) ) glyph at divmod.com glyph at divmod.com: > Also, it gives the Twisted team some potential competition to focus on. > Possibly a source of motivation, and besides, everybody loves a race :). As long as it's a "Wacky Races" style of competition, I'm happy with Kamaelia being competition. (Seriously though, Kamaelia is from a "what if", and hopefully scavengeable - either now or in future) > Kamaelia seems more promising to me; I imagine that they might produce some > serious competition eventually if that's what they aim to do. Only in terms of approach. Any sufficiently useful open source project is easily consumable by another. (Being served up for dinner and eaten is the ultimate success IMO) I do aim it to be a useful alternative way of thinking about problems similar to those that twisted thinks about. I saw your post here http://www.livejournal.com/users/glyf/41709.html , and it struck me as interesting. Essentially part of the comments appear to reuse of generators, which strikes a chord (largely because that's how hardware would have to work, and Kamaelia's core is based on asynchronous hardware design[*] ). Again, if we do head down that route (depends on resources, etc), then I'd hope people feel happy to rip out code where needed. [*] Whilst Kamaelia has no chance at present of compiling to silicon, things like MyHDL give me hope that it could be possible someday, and in that scenario, you need a system similar to the one you described. > However, they claim that only the "Axon" part of their project is stable, > and that doesn't yet include anything that does select(). We consider Axon to be API stable (ie stuff that's there is not likely to be removed). Axon is designed to be a small framework for handling concurrency, communications and composition between (mainly) generators, and threads have recently been introduced as well. By itself, Axon is intended to be very, very boring. The instant it's not being boring (such as doing a select, or poll, or ...) it's doing too much :-) As a result Axon's version number reflects what we think about it's stability. The reason for the jump from 1.0.X to 1.1.X was due to threaded components. (These were added earlier than expected due to a need for threaded components on Nokia mobiles, and have a suboptimal implementation and API right now.) The bulk of Kamaelia however is a collection of components, which is growing and we can do a large variety of Stuff. However, we still think (despite having used Kamaelia now for some time), that the collection of components we have at present is incomplete (around 30% of what would be needed generally speaking), and that the API for the components (specifically which inboxes/outboxes they have, names of components, file hierarchy etc), is still in flux. This collection of Stuff is being driven by application spikes, in application areas which are off the beaten track from the project's main goal (to round out the system), as well as in the main project goal area. So far this is proving to be an interesting approach. As a result we do have things like TCP server/client components, a selector component (which advertises itself as a service internally) which manages a select subsystem (which we're thinking of farming off into a thread to allow the entire system to sleep waiting for it), pygame Stuff, Tk Stuff, and so on. The latest application spike was a simple paint program for example, which was useful for testing a variety of subsystems. Similarly for the presentation tool spike. However, the low version number (currently) 0.3.0 is really to act as a warning "we still expect quite a lot to change before we hit 1.0.0". I could wax lyrical on that, but will only do so if people are interested :) However the reason it's 0.3.0, is because at minimum: * We want to have sensible integration with twisted (ie Kamaelia usable in twisted or vice versa) when we hit 1.0.0 * There's a whole bunch of core ideas we want to extend before then, including the graphical editting toolset. * We know we're just skimming the surface of possible components and systems (despite the amount of stuff we /can/ do). The component set needs to be fleshed out more before we know if the approach has worth. (It feels like it to me at present, but then I'm biassed) That said, generally, things are more stable than unstable, though currently I prefer to play up the instability to allow more freedom to experiment with APIs and ideas in the short/medium term. Anyway, I hope that this mail assures everyone on the twisted list (who was worried - struck me as a minority) that we're not really in /competition/, but more looking at providing /options/ for everyone, including Twisted :-) So far we're finding our approach useful and fun, which at the end of the day is all that matters. (For work the former is important, if it was just my personal time, then it could be either :-) Best Regards, Michael. -- Michael.Sparks(a)rd.bbc.co.uk, http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated.

1 0

[Twisted-Python] twisted.python.log vs. standard logging module
by Antony Kummel 23 Oct '05

23 Oct '05

Hi, I'm looking for a discussion on this subject I once saw where glyph lengthly elaborated on this matter. I'm just trying to find some useful idioms for logging my application, and I wonder why twisted doesn't have logging levels etc. I remember that discussion made some interesting phylosophical points, so if anyone knows where it is... Thanks, Antony Kummel __________________________________ Yahoo! FareChase: Search multiple travel sites in one click. http://farechase.yahoo.com

2 1

[Twisted-Python] testing my application
by Stefano Canepa 22 Oct '05

22 Oct '05

Hi all, I am looking for a simple way to write test cases for my project. What is the right way to be as standard as possible? I know the application works becouse I developed a client and the server speaking the same protocol and used lots of print to have output and a sniffer to see network trafic but I understand this is not the right way. I read unittest docs, I found twisted.trial and pocked around the test cases of some twisted protocol, what I was unable to understand is how to write my taste cases. Could somebody show me the way or point me to some docs? TIA Stefano -- Stefano Canepa aka sc: sc(a)linux.it http://www.stefanocanepa.it Three great virtues of a programmer: laziness, impatience and hubris. Le tre grandi virtù di un programmatore: pigrizia, impazienza e arroganza. (Larry Wall)

4 12

RE: [Twisted-Python] Win32 installers of Twisted 2.1?
by Grant McDonald 21 Oct '05

21 Oct '05

> Win32 requires a "sumo" package to be built. Can someone adequately explain what a sumo package is in relation to the normal twisted package? I've seen them for download but never paired with an explanation :) Thanks, Grant

3 2

[Twisted-Python] Freevo 2.0, Kamaelia, pyevent, Eventnet/LGT: what's going on?
by Nicola Larosa 21 Oct '05

21 Oct '05

One year ago the Freevo project, for its 2.0 rewrite, dropped Twisted, and then went and reimplemented something similar. Then there are Kamaelia, pyevent, and EventNet inside LGT. Why isn't Twisted seen, by these projects at least, as a viable tool? Is the event-driven Python world undergoing the same fragmentation as the web framework one? "Freevo is an open-source home theatre PC platform based on Linux and a number of open-source audio/video tools." http://freevo.sourceforge.net/about.html "CurrentStatus For Freevo 2.0 we changed many parts of the code... Changes * Changed the main loop to use PyNotifier. No threads are allowed anymore and every part must take care that the notifier loop is kept alive. Not all parts respect that right now. * Removed Twisted. It is a huge monster and hard to understand. For inter-process communication a python implementation (PyMbus) of the Mbus transport protocol is used. As a side effect, modules using twisted need to be rewritten." http://freevo.sourceforge.net/cgi-bin/freevo-2.0/CurrentStatus The reasons for the switch are outlined in this mailing list thread: switching to pynotifier and pymbus http://sourceforge.net/mailarchive/forum.php?thread_id=5523105&forum_id=9200 Kamaelia http://kamaelia.sourceforge.net/ pyevent http://monkey.org/~dugsong/pyevent/ EventNet - LGT http://lgt.berlios.de/#eventnet -- Nicola Larosa - nico(a)tekNico.net When people with lots of weapons and training in violence feel cornered, it tends to not be a pretty picture. -- Kirby Urner, August 2005

12 17