Mailman 3 January 2002 - Twisted

[Twisted-Python] web widgets
by Sean Riley 14 Jan '02

14 Jan '02

some observations of web widgets.. the "page" variable of the Gadget class was confusing to me.. I initially assumed that it was an instance of a Page object - rather than a class object. Maybe renaming it to something like "pageClass" would be more explicit? "Gadget" is also pretty ambiguous before it is placed in context. I would maybe prefer GadgetResource to make the context of it explicit - but i know how glyph dislikes compound class names. This would impress the standard twisted.web terminology on the naming within web.widgets... GadgetResource would be symetrical to "WidgetResource" which does exist - although i am not sure if it is used? "Page" is derived from Resource also which suggests to me it could be called "PageResource". User classes inheriting from both Gadget and Widget seems to be a very common pattern - such as all the classes in both twisted.forum.gadgets and twisted.metrics.gadgets. This would suggest another class in the framework that fills this niche. This could also refactor all the "if isinstance(self, widget)" code which is polluting Gadget into this new class. This would become a convenient place for users to simply derive classes from.. I don't know what to call it - GadgetWidget or WidgetGadget are just too meaningless. It is really a renderable collection of widgets.. any ideas? WidgetResource? WidgetGadgetResource? GadgetWidgetResource? Wadget? Gidget? Watchamacallit? attached is a class diagram for web widgets done in dia :) ---- "If it's not running programs or fusing atoms, it's just bending space.", Ken Macleod "That's it, I'm outta here.", Homer Simpson's Brain Sean Riley sean(a)ninjaneering.com

2 1

[Twisted-Python] Per-connection and per-protocol stats
by Itamar Shtull-Trauring 13 Jan '02

13 Jan '02

As I mentioned before in the metrics message, I want to have for each protocol events stats, per server and per server connection. This is the code I currently use (comments on the hits per second algorithm and in general are welcome.) I also have a web Resource for this so I can view stats with a nice interface. ====================================== """Event monitoring support""" import operator from twisted.python import threadable, delay class ProtocolMonitor: """Monitor a server that serves connections. Accepts a list of event names which the protocol can then log with log_event. """ def __init__(self, name, eventNames): self.name = name self.eventNames = eventNames self.connections = {} self.eventcounters = EventCounters(eventNames) def install(self): """Make sure reset() is run every 0.1 seconds.""" from twisted.internet import main self.looper = delay.Delayed() self.looper.ticktime = 0.1 self.looper.loop(self.reset, 1) main.addDelayed(self.looper) def uninstall(self): """Uninstall Delayed.""" from twisted.internet import main main.removeDelayed(self.looper) del self.looper def reset(self): """Runs once a second, resets all EventCounters.""" for e in self.connections.values(): e.reset() self.eventcounters.reset() def log_connectionOpened(self, conn): self.connections[conn] = EventCounters(self.eventNames) def log_connectionClosed(self, conn): del self.connections[conn] def log_event(self, conn, name): """First parameter should be a Protocol, second an event""" assert name in self.eventNames self.eventcounters.increment(name) if self.connections.has_key(conn): self.connections[conn].increment(name) synchronized = ["log_connectionOpened", "log_connectionClosed", "log_event", "reset"] class EventCounters: """Stores a number of event counters, and how many a second. Algorithm for determining how many events a second: Divide each second into 10 (N) equal parts. Make an 11 (N+1) items long list L containing how many events in each part of the second, plus the last part of the previous second. L = [0] * 11 When a new event is added, increment last item in list: L[-1] += 1 Every 0.1 (1.0/N) seconds, remove first item in L and append 0: del L[0]; L.append(0) To calculate how many events in the past second, sum up last 10 (N) parts of L and 0.5 of the first item in L. This gives us how many hits in the past 0.95 to 1.05 seconds (1-1.0/2N to 1+1.0/2N seconds), depending on when we run this function: (0.5 * L[0]) + reduce(operator.add, L[1:]) """ def __init__(self, eventNames, ): self.eventNames = eventNames self.counters = {} self.secondCounters = {} for name in eventNames: self.counters[name] = 0 self.secondCounters[name] = [0] * 10 def increment(self, name): self.counters[name] += 1 self.secondCounters[name][-1] += 1 def reset(self): """Should be called every 0.1 seconds.""" for name in self.eventNames: del self.secondCounters[name][0] self.secondCounters[name].append(0) def getInfo(self): """Return a dictionary. The key is an event name and the value a tuple: (counters, counters per second).""" result = {} for name in self.eventNames: l = self.secondCounters[name] perSecond = (0.5 * l[0]) + reduce(operator.add, l[1:]) result[name] = (self.counters[name], perSecond) return result synchronized = ["increment", "reset", "getInfo"] threadable.synchronize(ProtocolMonitor) threadable.synchronize(EventCounters)

1 0

[Twisted-Python] Use asyncore dispatchers with Twisted
by Itamar Shtull-Trauring 09 Jan '02

09 Jan '02

Yes, it is possible - you can use servers or clients written for asyncore together with twisted, using the following module. Should I check it in? If so, should it be in examples/ or twisted.internet, and what's a good module name? =================================================================== """Use asyncore dispatchers with Twisted. If you have an asyncore dispatcher, you can still use it with twisted. Just import this module, and don't run asyncore.loop() -- instead start the twisted event loop, either by calling twisted.internet.main.run(), or the usual way by calling an Application instance's run() method. """ import asyncore from twisted.internet import main class AsyncoreLooper: """Run the asyncore event loop for asyncore dispatchers.""" def timeout(self): if asyncore.socket_map: return 0.0 else: return None def runUntilCurrent(self): asyncore.poll(0.001) asyncoreLooper = AsyncoreLooper() main.addDelayed(asyncoreLooper) if __name__ == '__main__': # example, run debugging SMTP proxy (included with 2.1) # telnet to port 8025 to try it out. import smtpd d = smtpd.DebuggingServer(("localhost", 8025), ("gateway1.att.net", 25)) main.run()

2 2

[Twisted-Python] Re: [Twisted-commits] CVS: Twisted/twisted/internet main.py,1.51,1.52 poll.py,1.4,1.5
by Tommi Virtanen 06 Jan '02

06 Jan '02

Itamar Shtull-Trauring <itamarst(a)twistedmatrix.com> writes: > Update of /cvs/Twisted/twisted/internet > In directory zaibach:/tmp/cvs-serv23038 > > Modified Files: > main.py poll.py > Log Message: > made removing selectables at shutdown more generic so that poll-based event loop shutdown will work > > diff -b -u -d -r1.51 -r1.52 > --- main.py 25 Dec 2001 13:22:13 -0000 1.51 > +++ main.py 6 Jan 2002 12:22:10 -0000 1.52 > @@ -242,11 +242,8 @@ > log.msg('Select loop terminated.') > > finally: > - for reader in reads.keys(): > - if reads.has_key(reader): > - del reads[reader] > - if writes.has_key(reader): > - del writes[reader] > + selectables = removeAll() > + for reader in selectables: > log.logOwner.own(reader) > try: > reader.connectionLost() > @@ -297,6 +294,17 @@ > if writes.has_key(writer): > del writes[writer] > > +def removeAll(): > + """Remove all readers and writers, and return list of Selectables.""" > + readers = reads.keys() > + for reader in readers: > + if reads.has_key(reader): > + del reads[reader] > + if writes.has_key(reader): > + del writes[reader] > + return readers > + > + > class _Win32Waker(styles.Ephemeral): > """I am a workaround for the lack of pipes on win32. What if you have fd 42 that is only a writer, not a reader? That removeAll() won't process it. readers = uniq(reads.keys(), writes.keys()) ? This does not affect the poll as readers and writers are "united" there. -- tv(a){{hq.yok.utu,havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com} double a,b=4,c;main(){for(;++a<2e6;c-=(b=-b)/a++);printf("%f\n",c);}

1 0

[Twisted-Python] PB for Java status report
by Itamar Shtull-Trauring 06 Jan '02

06 Jan '02

It's now possible to create Referenceables in Java - meanwhile simple remote_xxx style ones only, no support for Perspectives or Viewables yet. In other words, you can now create PB services in Java as well - I think it's pluggable enough that you could integrate this with any Java server architecture that uses streams (which is probably all of them :).

3 4

[Twisted-Python] IRC protocol rewrite?
by Ivo van der Wijk 03 Jan '02

03 Jan '02

Hi All, I'm currently investigating the IRC Protocol as implemented by twisted, to see if I perhaps can use it in my irc client, or if it perhaps can use code from my irc protocol parsing. However, I noticed some bugs and (imho) design failures in the current implementation, which makes the code not very usefull for any other use than what it's currently used for (twisted irc client + server?) For example, channels are hardcoded as starting with '#', i.e. IRCClient.join prepends a '#' to the channelname. Hoewever, most servers support alernative style channels such as &channel (local channels), +channel (modeless channels), and perhaps more. Furthermore, IMHO, the IRCClient class implements too many protocols in one class. IRC, CTCP and DCC are distinct protocols, and each belong in their own class. Lastly, the current parsemsg routine can't handle ipv6 addresses in certain replies, i.e. a 311 (whois) reply like: :irc.xs4all.nl 311 VladDrac [p] -patrick 3ffe:2500:900:2000:1337:1337:1337:1337 *:foo will be parsed incorrectly. I may provide a patch for this later. Is replacing the current IRC protocol with a (non-api-compatible) new protocol discussable? If so, I'll make some suggestions on how I would implement the protocol. Cheers, Ivo -- Drs. I.R. van der Wijk -=- Brouwersgracht 132 Amaze Internet Services V.O.F. 1013 HA Amsterdam, NL -=- Tel: +31-20-4688336 Linux/Web/Zope/SQL/MMBase Fax: +31-20-4688337 Network Solutions Web: http://www.amaze.nl/ Consultancy Email: ivo(a)amaze.nl -=-

3 5

[Twisted-Python] Bug in t.w.w.WidgetPage
by Sune Kirkeby 03 Jan '02

03 Jan '02

Hi all, I think there might be a bug lurking in t.w.w.WidgetPage's handling of Deferreds. Take a look at t.w.w.WidgetPage.callback: def callback(self, result, position, decNeedsHeaders): if result != FORGET_IT: self.needsHeaders = self.needsHeaders - decNeedsHeaders else: result = [FORGET_IT] for i in xrange(len(result)): if isinstance(result[i], defer.Deferred): self._addDeferred(result[i], position+i) # print 'CALLBACK:',self.lst, position, result if not isinstance(result, types.ListType): result = [result] self.lst[position:position+1] = result assert self.position <= position self.keepRendering() for r in result: if isinstance(r, defer.Deferred): r.arm() Now consider this sequence of events: (1) We initialize a t.w.w.WidgetPage with two Deferreds (head and foot), so in our WidgetPage (let us call it foo) we have foo.lst = [head, foot]. (2) mouth is now almost done, so it defers the rest of it's computation, and callback's with a string 'Hello, ' and a Deferred (let us call it mouth). (3) After foo.callback is done we now have that foo.lst = ['Hello, ', mouth, foot]. (4) Now foot is done and callback's with a string 'World!'. (5) After foo.callback is done this time we have that foo.lst = ['Hello, ', 'World!', foot] Whoa! We just put our foot in our mouth! So to speak :-) -- Sune Kirkeby | In general, they do what you want, unless you want | consistency. --Larry Wall in the perl man page

2 3

[Twisted-Python] Return of the "write immediately" patch
by Itamar Shtull-Trauring 01 Jan '02

01 Jan '02

Last time I tried this, my benchmarks were being screwed up by a bug in twisted's threading. This time I'm getting consistent results - an improvement in write performance (large writes over loopback) of 22%! Index: twisted/internet/abstract.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/abstract.py,v retrieving revision 1.11 diff -c -r1.11 abstract.py *** twisted/internet/abstract.py 30 Oct 2001 23:55:39 -0000 1.11 --- twisted/internet/abstract.py 1 Jan 2002 17:40:27 -0000 *************** *** 106,112 **** if not self.connected: return if data: ! self.unsent = self.unsent + data if self.producer is not None: if len(self.unsent) > self.bufferSize: self.producerPaused = 1 --- 106,118 ---- if not self.connected: return if data: ! if not self.unsent: ! l = self.writeSomeData(data) ! if l < 0 or l == len(data): ! return ! self.unsent = data[l:] ! else: ! self.unsent = self.unsent + data if self.producer is not None: if len(self.unsent) > self.bufferSize: self.producerPaused = 1

1 0

[Twisted-Python] newbie question: error handling
by Ivo van der Wijk 01 Jan '02

01 Jan '02

Hi All, I'm currently migrating my irc client to twisted, and things look promising. However, I can't figure out the following things: My protocol is derived from protocols.basic.LineReceiver (which in turn derives from protocol.Protocol.protocol). How can I determine what specific error condition caused connectionLost() or connectionFailed()? I.e. connection refused, unknown hostname, no route to host, etc (the standard errno stuff). Also, how can I determine if Linereceived.sendLine() (or Protocol.Transport.write()) failed? It doesn't seem to return an error value or raise Exceptions, and depending on the invocation of connectionLost makes things perhaps too asynchronous.. Cheers, Ivo -- Drs. I.R. van der Wijk -=- Brouwersgracht 132 Amaze Internet Services V.O.F. 1013 HA Amsterdam, NL -=- Tel: +31-20-4688336 Linux/Web/Zope/SQL/MMBase Fax: +31-20-4688337 Network Solutions Web: http://www.amaze.nl/ Consultancy Email: ivo(a)amaze.nl -=-

2 3