Mailman 3 April 2007 - Twisted

[Twisted-Python] Implementing an event manager in twisted
by Justin Warren 13 Apr '07

13 Apr '07

Hi folks, I have had an idea to implement an event manager to glue a few different system components together, and I am wondering which pieces of twisted would be best suited to the task, or even if I should attempt it at all. I have programs that poll for data. I have a database that stores polled information. I have programs that listen for data to be provided to them from external entities. I have a desire to provide scheduler functionality (at time x, take action y). It seems to me that I could join all of these things via an event manager: A program polls for data, succeeds and notifies the event manager of the data that was polled. The event manager notifies a 'poll storage' object that stores the data in the database. It could also notify other things. Data is received from an external entity by a listener. The listener notifies the event manager of the data. The event manager notifies an object that stores it in the database, or runs a script, or uploads something to flickr, or whatever. Time passes. The event manager notifies interested objects that this is so. They take whatever action they deem necessary. None of this is new, so I wonder what others have done in this area? Am I heading down a dangerous path? What are the traps for new players? Has someone already written something that does all of this that I can use, saving myself time and headaches? -- Justin Warren <daedalus(a)eigenmagic.com>

5 5

[Twisted-Python] Foolscap-0.1.1 released
by Brian Warner 13 Apr '07

13 Apr '07

I've just released Foolscap-0.1.1, the next-generation-of-PB RPC library, available in the usual place at: http://twistedmatrix.com/trac/wiki/FoolsCap http://twistedmatrix.com/~warner/Foolscap/ This release enhances many of the "constraints" (a form of explicit typechecking that can be applied to messages sent over the wire), fixes a long-standing bug in Tub.stopService() which makes it easier to write trial unit tests for code which uses Foolscap (and which was hopefully responsible for many of the spurious test failures we've seen in the past), and adds a one-way no-response 'callRemoteOnly' method. Full details are in the release notes, attached below. Due to the implementation of callRemoteOnly, this release is *not* wire-protocol compatible with the previous 0.1.0 release. Fortunately, it knows this, and the version-negotiation code will refuse to connect to an incompatible peer. Many thanks to my employer, AllMyData.com, for supporting development of this release. We're starting to use it for an internal project, and we're discovering all sorts of usability improvements that need to be made. The next batch will probably be centered around the needs of long-running server programs: specifically persistent mapping from externally-visible (but generally unguessable) names to internal handler objects, such that each time the process gets restarted, the same name maps to the next incarnation of the handler object. We already have this facility for the private key (and thus the TubID), but now it would be nice to have it for individual objects. have a well-connected day, -Brian * Release 0.1.1 (03 Apr 2007) ** Incompatibility Warning Because of the technique used to implement callRemoteOnly() (specifically the commandeering of reqID=0), this release is not compatible with the previous release. The protocol negotiation version numbers have been bumped to avoid confusion, meaning that 0.1.0 Tubs will refuse to connect to 0.1.1 Tubs, and vice versa. Be aware that the errors reported when this occurs may not be ideal, in particular I think the "reconnector" (tub.connectTo) might not log this sort of connection failure in a very useful way. ** changes to Constraints Method specifications inside RemoteInterfaces can now accept or return 'Referenceable' to indicate that they will accept a Referenceable of any sort. Likewise, they can use something like 'RIFoo' to indicate that they want a Referenceable or RemoteReference that implements RIFoo. Note that this restriction does not quite nail down the directionality: in particular there is not yet a way to specify that the method will only accept a Referenceable and not a RemoteReference. I'm waiting to see if such a thing is actually useful before implementing it. As an example: class RIUser(RemoteInterface): def get_age(): return int class RIUserListing(RemoteInterface): def get_user(name=str): """Get the User object for a given name.""" return RIUser In addition, several constraints have been enhanced. StringConstraint and ListConstraint now accept a minLength= argument, and StringConstraint also takes a regular expression to apply to the string it inspects (the regexp can either be passed as a string or as the output of re.compile()). There is a new SetConstraint object, with 'SetOf' as a short alias. Some examples: HexIdConstraint = StringConstraint(minLength=20, maxLength=20, regexp=r'[\dA-Fa-f]+') class RITable(RemoteInterface): def get_users_by_id(id=HexIdConstraint): """Get a set of User objects; all will have the same ID number.""" return SetOf(RIUser, maxLength=200) These constraints should be imported from foolscap.schema . Once the constraint interface is stabilized and documented, these classes will probably be moved into foolscap/__init__.py so that you can just do 'from foolscap import SetOf', etc. *** UnconstrainedMethod To disable schema checking for a specific method, use UnconstrainedMethod in the RemoteInterface definition: from foolscap.remoteinterface import UnconstrainedMethod class RIUse(RemoteInterface): def set_phone_number(area_code=int, number=int): return bool set_arbitrary_data = UnconstrainedMethod The schema-checking code will allow any sorts of arguments through to this remote method, and allow any return value. This is like schema.Any(), but for entire methods instead of just specific values. Obviously, using this defeats te whole purpose of schema checking, but in some circumstances it might be preferable to allow one or two unconstrained methods rather than resorting to leaving the entire class left unconstrained (by not declaring a RemoteInterface at all). *** internal schema implementation changes Constraints underwent a massive internal refactoring in this release, to avoid a number of messy circular imports. The new way to convert a "shorthand" description (like 'str') into an actual constraint object (like StringConstraint) is to adapt it to IConstraint. In addition, all constraints were moved closer to their associated slicer/unslicer definitions. For example, SetConstraint is defined in foolscap.slicers.set, right next to SetSlicer and SetUnslicer. The constraints for basic tokens (like lists and ints) live in foolscap.constraint . ** callRemoteOnly A new "fire and forget" API was added to tell Foolscap that you want to send a message to the remote end, but do not care when or even whether it arrives. These messages are guaranteed to not fire an errback if the connection is already lost (DeadReferenceError) or if the connection is lost before the message is delivered or the response comes back (ConnectionLost). At present, this no-error philosophy is so strong that even schema Violation exceptions are suppressed, and the callRemoteOnly() method always returns None instead of a Deferred. This last part might change in the future. This is most useful for messages that are tightly coupled to the connection itself, such that if the connection is lost, then it won't matter whether the message was received or not. If the only state that the message modifies is both scoped to the connection (i.e. not used anywhere else in the receiving application) and only affects *inbound* data, then callRemoteOnly might be useful. It may involve less error-checking code on the senders side, and it may involve fewer round trips (since no response will be generated when the message is delivered). As a contrived example, a message which informs the far end that all subsequent messages on this connection will sent entirely in uppercase (such that the recipient should apply some sort of filter to them) would be suitable for callRemoteOnly. The sender does not need to know exactly when the message has been received, since Foolscap guarantees that all subsequently sent messages will be delivered *after* the 'SetUpperCase' message. And, the sender does not need to know whether the connection was lost before or after the receipt of the message, since the establishment of a new connection will reset this 'uppercase' flag back to some known initial-contact state. rref.callRemoteOnly("set_uppercase", True) # returns None! This method is intended to parallel the 'deliverOnly' method used in E's CapTP protocol. It is also used (or will be used) in some internal Foolscap messages to reduce unnecessary network traffic. ** new Slicers: builtin set/frozenset Code has been added to allow Foolscap to handle the built-in 'set' and 'frozenset' types that were introduced in python-2.4 . The wire protocol does not distinguish between 'set' and 'sets.Set', nor between 'frozenset' and 'sets.ImmutableSet'. For the sake of compatibility, everything that comes out of the deserializer uses the pre-2.4 'sets' module. Unfortunately that means that a 'set' sent into a Foolscap connection will come back out as a 'sets.Set'. 'set' and 'sets.Set' are not entirely interoperable, and concise things like 'added = new_things - old_things' will not work if the objects are of different types (but note that things like 'added = new_things.difference(old_things)' *do* work). The current workaround is for remote methods to coerce everything to a locally-preferred form before use. Better solutions to this are still being sought. The most promising approach is for Foolscap to unconditionally deserialize to the builtin types on python >= 2.4, but then an application which works fine on 2.3 (by using sets.Set) will fail when moved to 2.4 . ** Tub.stopService now indicates full connection shutdown, helping Trial tests Like all twisted.application.service.MultiService instances, the Tub.stopService() method returns a Deferred that indicates when shutdown has finished. Previously, this Deferred could fire a bit early, when network connections were still trying to deliver the last bits of data. This caused problems with the Trial unit test framework, which insist upon having a clean reactor between tests. Trial test writers who use Foolscap should include the following sequence in their twisted.trial.unittest.TestCase.tearDown() methods: def tearDown(self): from foolscap.eventual import flushEventualQueue d = tub.stopService() d.addCallback(flushEventualQueue) return d This will insure that all network activity is complete, and that all message deliveries thus triggered have been retired. This activity includes any outbound connections that were initiated (but not completed, or finished negotiating), as well as any listening sockets. The only remaining problem I've seen so far is with reactor.resolve(), which is used to translate DNS names into addresses, and has a window during which you can shut down the Tub and it will leave a cleanup timer lying around. The only solution I've found is to avoid using DNS names in URLs. Of course for real applications this does not matter: it only makes a difference in Trial unit tests which are making heavy use of short-lived Tubs and connections.

4 5

[Twisted-Python] Synchronization techniques
by Daniel Miller 12 Apr '07

12 Apr '07

Hello fellow twisters, I am developing a twisted application on Mac OS X (using Perspective Broker) in which the server component runs shell scripts to post orders to an accounting system. The accounting system is developed and maintained by a third party vendor, and is therefore beyond my control. For a given command the scripts must be executed in order, and all commands that affect the accounting system must be synchronized. For example: class AccountingService(pb.Root): def remote_post_order(self, data): self.lock.acquire() try: # put data in 'trans.dat' file # ... process = TwistedProcess() process.check_call(["load_dat.sh", "trans.dat"]) process.check_call(["post_trx.sh"]) finally: self.lock.release() The process.check_call() method has (nearly) the same semantics as the check_call() method in Python's built-in subprocess module, and the lock has (nearly) the same semantics as Python's built-in threading.Lock class. Unfortunately, I cannot use those built-in methods/classes because they do not play nice with Twisted--I get 'Interrupted system call' errors from subprocess.check_call(), and (I think) my server could deadlock if I used threading.Lock. So I developed a few helper classes to hack around these Twisted mis- features (questions follow the code): from Queue import Queue, Empty from twisted.internet.error import ProcessTerminated from twisted.internet.protocol import ProcessProtocol class ProcessError(Exception): pass class ProcessErrorHandler(ProcessProtocol): def __init__(self): self.errbuf = [] self.errors = [] self.waiting = True def errReceived(self, text): self.errbuf.append(text) def processEnded(self, status): if self.errbuf: self.errors.append("".join(self.errbuf)) if isinstance(status.value, ProcessTerminated): self.errors.append(status.getErrorMessage()) self.waiting = False def hasError(self): return bool(self.errors) def raiseError(self): raise ProcessError("\n".join(self.errors)) class TwistedProcess(object): def __init__(self, reactor=None): if reactor is None: from twisted.internet import reactor self.reactor = reactor def check_call(self, cmd): status = ProcessErrorHandler() self.reactor.spawnProcess(status, cmd[0], cmd, env=os.environ) while status.waiting: self.reactor.iterate(0.05) if status.hasError(): status.raiseError() class TwistedLock(object): """A lock object for Twisted The lock is instantiated in a released state. """ def __init__(self, reactor=None): if reactor is None: from twisted.internet import reactor self.reactor = reactor self.queue = Queue(1) self.release() def acquire(self): """Acquire the lock This method returns immediately when the lock is acquired. It "blocks" (i.e. iterates the reactor) indefinitely while it waits for the lock to be released. """ while True: try: self.queue.get_nowait() except Empty: # pseudo block (iterate the reactor) while waiting for the lock to be released self.reactor.iterate(0.05) else: break # the lock has been acquired def release(self): """Release the lock Raises Queue.Full if the lock was not acquired exactly once since it was last released. """ self.queue.put_nowait("TOKEN") Questions: 1. Is reactor.iterate() meant to be used this way (see TwistedLock.acquire() and TwistedProcess.check_call())? If not, what is the right way to do those things? I've read everything I can find on reactor.iterate() and while it seems to be the only way to do what I need to do, its use is often strongly discouraged. Why is it so difficult for twisted to handle non-async code? 2. Is my suspicion that the server could deadlock if I used threading.Lock correct? Note: the server is not multi-threaded (unless twisted is doing something with threads that I am not aware of). Example scenario: req 1: acquire lock req 1: spawn sub-process req 1: iterate reactor while waiting for process to complete req 2: acquire lock (block and wait) SERVER IS DEAD-LOCKED ? 3. Is there a better way to do what I'm trying to do here? (getting a new accounting system is not an option :) ~ Daniel

12 30

[Twisted-Python] Threadpool analysis?
by Jonathan Vanasco 11 Apr '07

11 Apr '07

My threadpool on a webspider keeps locking up. I've been using manhole and sys._current_frames to try and figure out whats going wrong -- and i've fixed a few issues. I'm wondering though if anyone has developed some sort of threadpool analysis plugin -- something that monitors when threads check in/ out - maybe store stuff using rrd. Also I'm wondering if there's a good /idiomatic way to mark/monitor threads -- I want to monitor certain stuff that I call defer to thread on, I don't really care about other stuff in the threadpool. Thanks. // Jonathan Vanasco | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | SyndiClick.com | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | FindMeOn.com - The cure for Multiple Web Personality Disorder | Web Identity Management and 3D Social Networking | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | RoadSound.com - Tools For Bands, Stuff For Fans | Collaborative Online Management And Syndication Tools | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

3 5

Re: [Twisted-Python] Trial docs?
by Terry Jones 11 Apr '07

11 Apr '07

| What's your login name for trac? I think someone will need to give you | edit permission, if they haven't already. Hi Glyph. I just went to add a few things to the top-level Trial page and see that I do need edit perms. I was mistaken - I've only changed stuff over at divmod.org. It's terrycojones. Terry

2 1

[Twisted-Python] Wrapping functions that return deferreds
by Ross Burton 11 Apr '07

11 Apr '07

Hi, I have a library that has a call() method, which returns a Deferred. My application used to just use .addCallback() but obviously that doesn't have any error handling. I then switched to .addCallbacks() so that I could call a common error handler (to pop up a dialog) but that got tedious to add to every invocation so I thought I'd be cunning and wrap the call() method. def wrap_deferred (func): def wrapped(*args, **kwargs): d = func (*args, **kwargs) d.addErrback (handle_twisted_error) return d return wrapped # flickr is the object, call is the method I'm overriding flickr.call = wrap_deferred (flickr.call) # Then later in my app flickr.call(foo).addCallback(handleFoo) However, this doesn't work: I get the standard unhandled error in deferred handling. Can anyone spot the mistake I'm making? Thanks, Ross -- Ross Burton mail: ross(a)burtonini.com jabber: ross(a)burtonini.com www: http://www.burtonini.com./ PGP Fingerprint: 1A21 F5B0 D8D0 CFE3 81D4 E25A 2D09 E447 D0B4 33DF

2 2

Re: [Twisted-Python] Question about Writing a Stackless Friendly Reactor
by Andrew Francis 11 Apr '07

11 Apr '07

Hello Glyph: >I disagree. Separate reactor implementations are for >different OS multiplexing mechanisms. >stackless.schedule() should *not* be called >"whenever possible", but rather, whenever stackless >tasks have something to do. To the best of my understanding, if one is using Stackless cooperative scheduling, a tasklet should yield, that is, call schedule() when it is finished doing work. Otherwise, other tasklets that could run, will not get an opportunity. In a Stackless application, the reactor is running in a tasklet. Leaving aside the issue of channels, if the reactor does not call schedule(), other tasklets will not run. For most applications, the Twisted Reactor blocking the application is not an issue since the tasklets have nothing to do if they do not have input In the case of my application, the blocking is an issue... That said, we can write simple programmes to illustrate this (or fix my first example). For instance, I illustrate the aforementioned on the Stackless Google code page. http://code.google.com/p/stacklessexamples/wiki/StacklessTwisted In the first example, the tick tasklet will print only after the request handler has received input. In the second threaded example, the tick tasklet will start printing immediately. I am looking for ways of getting the behaviour of the second example without using threads (or Twisted tasks that make schedule() calls) >Don't implement a reactor :). I think subclassing SelectorReactor may give me much of what I want. I suspect this is an unTwisted way of doing things. I am still experimenting with approaches. >I recently replied to a very similar question in a >different thread: >http://article.gmane.org/gmane.comp.python.twisted/13946 >What about my answer there was not clear? I don't agree with a part of the premise. That aside, I tried using reactor.callLater(). It didn't help. Perhaps I am using it wrong. Cheers, Andrew ____________________________________________________________________________________ Need Mail bonding? Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users. http://answers.yahoo.com/dir/?link=list&sid=396546091

1 0

[Twisted-Python] Re: epoll reactor problems
by Alec Matusis 11 Apr '07

11 Apr '07

PS. One more piece of data: we just switched that server back to poll, still on Twisted 2.5. The CPU is at 30-45% stable, with exactly the same usage. So this is either caused by the epoll() itself, or by the epoll reactor in twisted. From: Alec Matusis [mailto:matusis@matusis.com] Sent: Wednesday, April 11, 2007 12:52 AM To: 'Twisted general discussion' Subject: epoll reactor problems We just switched 2 types of production servers to epoll reactor( Twisted 2.5) from poll reactor (Twisted 2.2). The CPU%% utilization of the first type of server that does not do much except occasionally pushing messages to about 5000 clients dropped from about 40% to 8%, which is very good. The second type of server is more complicated. The CPU utilization of that server (as measured by top) went down from 40% to about 15% after switching to epoll. Here is the problem: after about 10min of running that server with CPU%% staying at about 15%, the CPU suddenly jumps to 99.9% for that process and just stays there. We reproduced this several times. The server remains responsive, even when top shows 99.9% CPU. (Which is very different from 99.9% CPU from real load when we used poll- we are intimately familiar with the performance in that regime unfortunately.) The kernel is 2.6.11.4-21.12-smp Can anybody help with this 99.9% CPU epoll problem?

1 0

[Twisted-Python] Memcached
by Matthew Glubb 11 Apr '07

11 Apr '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi All, I was wondering whether anyone has written a twisted protocol class / module for memcached? I am currently developing an application that makes use of memcached. I really don't like the fact that the memcache operations are blocking within my nice deferred environment. In lieu of a deferred memcache operation, would it be wise to place this functionality in a twisted thread? Would anyone be interested in a twisted module for memcached or would the additional overhead render it useless (too slow)? Regards, Matt m a t t h e w g l u b b ________________________________________________________________________ Z Group PLC Tel: +44 (0) 8700 111 173 Fax: +44 (0) 8707 051 393 Txt: +44 (0) 7800 140 877 Web: <http://www.zgroupplc.com/> This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. The opinions expressed in this mail are those of the author and do not necessarily represent the views of the company. If you have received this email in error please notify <service(a)zgroupplc.com> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFGG3zByI6MkdKPngkRAvRjAKCyC/96jtFHLTu+ouw+KK9wE4OIeQCeIDsJ wH1IgSPsRSFrX1/HXKmnWPE= =akr3 -----END PGP SIGNATURE-----

3 4

[Twisted-Python] Question about Writing a Stackless Friendly Reactor
by Andrew Francis 10 Apr '07

10 Apr '07

Hello : I have been using Twisted with Stackless. I am interested in writing a more Stackless "friendly" reactor. I believe a stackless friendly reactor would call stackless.schedule() whenever possible. I notice there are different types of reactor. I have been looking at SelectReactor and its base case, Posixbase. It seems easy enough to modify MainLoop to put in a stackless.schedule(). However this approach does not seem to be the most maintainable or effective, since I would be overriding MainLoop only to put in one line. What is the proper Twisted approach to doing this? Cheers, Andrew ____________________________________________________________________________________ Get your own web address. Have a HUGE year through Yahoo! Small Business. http://smallbusiness.yahoo.com/domains/?p=BESTDEAL

3 2