Mailman 3 August 2007 - Twisted

[Twisted-Python] non-blocking interface for local file system operations
by Eduardo Schettino 10 Aug '07

10 Aug '07

Hi, Is there an asynchronous wrapper for local file system operation (i.e. write a file to disk)? I was looking for something like what twisted.enterprise.adbapi do to database access. And I guess using deferToThread is not the right approach for I/O operations. Regards, Eduardo

3 3

[Twisted-Python] Running twisted.trial unittests using nose
by Chris Miles 08 Aug '07

08 Aug '07

I'm testing one of my modules that returns a Twisted deferred, using twisted.trial.unittest.TestCase. The test case methods return a deferred, after first adding a callback to a function that actually performs the assertion. All fairly standard for Twisted unit testing. If I use "trial" to kick off the tests they all pass fine (or fail as expected, if I force them to). If I run the same tests using "nosetests" they also pass (or fail correctly - which confirms the callbacks are being called properly). I'm simply curious as to whether it is safe for me to rely on a test runner other than "trial" to run the tests? I assume any magic (knowledge of deferreds) is within the twisted.trial.unittest.TestCase class and the "trial" command is simply another tool to discover and pretty print the results. From what I've seen, both "nosetests" and "trial" will produce the same report (although "trial" looks nicer :-). Or perhaps this wouldn't be the case if I needed to interact with the reactor? Cheers, Chris Miles

4 7

[Twisted-Python] struggling with twisted telnet
by Andrew Roden 08 Aug '07

08 Aug '07

Hi I'm pretty new to python which probably isn't helping, but I'm kind of stuck with twisted telnet. I'm trying to build a couple of 'emulators' for some simple router/switches just so that you can log on and run some 'standard' commands and get a 'standard' output for use elsewhere. I've managed to get an SSH version working with the massively helpful section in the O'Reilly book but I cannot get the telnet version working. the script 'runs' with no errors, but it doesn't stay running and bind to port 23 giving me my telnet server. I enclose a zip with the ssh and the telnet versions, if anyone can give me any pointers it would be massively appreciated (I haven't even attempted to add in the 'login' which is in the SSH version... so any help on that would be great too :) ) Thanks Andrew

2 4

[Twisted-Python] stop receiving from socket until processing finish
by Rodrigo Strauss 08 Aug '07

08 Aug '07

Hi! Consider the following scenario (each component is a twisted app): producer --> processor --> destination I need to stop receiving producer data until I'm able to process it and send to destination. Using sync sockets, the producer send() with block if processor isn't recv'ing as fast as it can send. How can I do this using twisted? Something like telling reactor to stop receiving from a specific socket until I enable it back. When I send something to destination, the reactor will come back to scene and recv data from producer. A simple solution is to make processor ask for data to process every time, but when processor is fast enough it will hurt the performance to much. Rodrigo

2 1

[Twisted-Python] thread in reactor problem
by Yan Zhu 07 Aug '07

07 Aug '07

Hi all, I run a reactor in my thread, so I will get many connection by twisted, and I run another thread for write back data to any connection. But I've found the data is in buffer, not sent back immediately. I traced the twisted, found it blocked in class SelectReactor.doSelect(): while 1: try: r, w, ignored = _select(reads.keys(), writes.keys(), [], timeout) break ... ... #do sth when the reactor is bloce by _select(), then my another thread modified the *writes* dictionary, but _select could not know if writes.keys() is changed, so when the thread want to write something, reactor won't do response. so, the problem is above. Is there any idea for this? thanks~ -- eSX

2 2

[Twisted-Python] Foolscap-0.1.5 released
by Brian Warner 07 Aug '07

07 Aug '07

I'm pleased to announce the release of Foolscap-0.1.5, the latest version of the next-generation Perspective Broker library. My intention with this library is to make many of the communication and cooperation features of the E language available to Python programmers, and to simplify secure communication between twisted-based python progams. (there have been several intermediate releases since foolscap-0.1.1 in April, which I neglected to announce. oops.) Foolscap has moved out of the Twisted sandbox and into its very own Mercurial repository. It also has its own Trac instance. For downloads, bugs, and all things Foolscapish (with the exception of a mailing list, for which we're still using twisted-python), please visit: http://foolscap.lothar.com/ The release can be downloaded directly from http://foolscap.lothar.com/releases/foolscap-0.1.5.tar.gz , or installed by typing 'easy_install foolscap' on a setuptools-enabled system. A large number of bugs and features have been addressed since 0.1.1, please see the NEWS file (attached) for full details. Many of them relate to the handling of third-party references ("Gifts"), connection keepalives, and Constaints. In addition, a number of usability improvements have been made based upon experience gained by deploying Foolscap in a controlled environment[1][2]. I am grateful to my employer, Allmydata.com, for providing me with the time and environment to use and develop Foolscap for a real-world application. Please download and hack away. File bugs and patches in the foolscap Trac, and discuss anything which doesn't fit into a ticket here on twisted-python. share and enjoy! -Brian [1]: http://allmydata.org/trac/tahoe/wiki (all tahoe connections use Foolscap) [2]: http://www.allmydata.com/wordpress/?p=24 (the 1.8 release uses Foolscap-0.1.4 internally) * Release 0.1.5 (07 Aug 2007) ** Compatibility This release is fully compatible with 0.1.4 and 0.1.3 . ** CopiedFailure improvements When a remote method call fails, the calling side gets back a CopiedFailure instance. These instances now behave slightly more like the (local) Failure objects that they are intended to mirror, in that .type now behaves much like the original class. This should allow trial tests which result in a CopiedFailure to be logged without exploding. In addition, chained failures (where A calls B, and B calls C, and C fails, so C's Failure is eventually returned back to A) should work correctly now. ** Gift improvements Gifts inside return values should properly stall the delivery of the response until the gift is resolved. Gifts in all sorts of containers should work properly now. Gifts which cannot be resolved successfully (either because the hosting Tub cannot be reached, or because the name cannot be found) will now cause a proper error rather than hanging forever. Unresolvable gifts in method arguments will cause the message to not be delivered and an error to be returned to the caller. Unresolvable gifts in method return values will cause the caller to receive an error. ** IRemoteReference() adapter The IRemoteReference() interface now has an adapter from Referenceable which creates a wrapper that enables the use of callRemote() and other IRemoteReference methods on a local object. The situation where this might be useful is when you have a central introducer and a bunch of clients, and the clients are introducing themselves to each other (to create a fully-connected mesh), and the introductions are using live references (i.e. Gifts), then when a specific client learns about itself from the introducer, that client will receive a local object instead of a RemoteReference. Each client will wind up with n-1 RemoteReferences and a single local object. This adapter allows the client to treat all these introductions as equal. A client that wishes to send a message to everyone it's been introduced to (including itself) can use: for i in introductions: IRemoteReference(i).callRemote("hello", args) In the future, if we implement coercing Guards (instead of compliance-asserting Constraints), then IRemoteReference will be useful as a guard on methods that want to insure that they can do callRemote (and notifyOnDisconnect, etc) on their argument. ** Tub.registerNameLookupHandler This method allows a one-argument name-lookup callable to be attached to the Tub. This augments the table maintained by Tub.registerReference, allowing Referenceables to be created on the fly, or persisted/retrieved on disk instead of requiring all of them to be generated and registered at startup. * Release 0.1.4 (14 May 2007) ** Compatibility This release is fully compatible with 0.1.3 . ** getReference/connectTo can be called before Tub.startService() The Tub.startService changes that were suggested in the 0.1.3 release notes have been implemented. Calling getReference() or connectTo() before the Tub has been started is now allowed, however no action will take place until the Tub is running. Don't forget to start the Tub, or you'll be left wondering why your Deferred or callback is never fired. (A log message is emitted when these calls are made before the Tub is started, in the hopes of helping developers find this mistake faster). ** constraint improvements The RIFoo -style constraint now accepts gifts (third-party references). This also means that using RIFoo on the outbound side will accept either a Referenceable that implements the given RemoteInterface or a RemoteReference that points to a Referenceable that implements the given RemoteInterface. There is a situation (sending a RemoteReference back to its owner) that will pass the outbound constraint but be rejected by the inbound constraint on the other end. It remains to be seen how this will be fixed. ** foolscap now deserializes into python2.4-native 'set' and 'frozenset' types Since Foolscap is dependent upon python2.4 or newer anyways, it now unconditionally creates built-in 'set' and 'frozenset' instances when deserializing 'set'/'immutable-set' banana sequences. The pre-python2.4 'sets' module has non-built-in set classes named sets.Set and sets.ImmutableSet, and these are serialized just like the built-in forms. Unfortunately this means that Set and ImmutableSet will not survive a round-trip: they'll be turned into set and frozenset, respectively. Worse yet, 'set' and 'sets.Set' are not entirely compatible. This may cause a problem for older applications that were written to be compatible with both python-2.3 and python-2.4 (by using sets.Set/sets.ImmutableSet), for which the compatibility code is still in place (i.e. they are not using set/frozenset). These applications may experience problems when set objects that traverse the wire via Foolscap are brought into close proximity with set objects that remained local. This is unfortunate, but it's the cleanest way to support modern applications that use the native types exclusively. ** bug fixes Gifts inside containers (lists, tuples, dicts, sets) were broken: the target method was frequently invoked before the gift had properly resolved into a RemoteReference. Constraints involving gifts inside containers were broken too. The constraints may be too loose right now, but I don't think they should cause false negatives. The unused SturdyRef.asLiveRef method was removed, since it didn't work anyways. ** terminology shift: FURL The preferred name for the sort of URL that you get back from registerReference (and hand to getReference or connectTo) has changed from "PB URL" to "FURL" (short for Foolscap URL). They still start with 'pb:', however. Documentation is slowly being changed to use this term. * Release 0.1.3 (02 May 2007) ** Incompatibility Warning The 'keepalive' feature described below adds a new pair of banana tokens, PING and PONG, which introduces a compatibility break between 0.1.2 and 0.1.3 . Older versions would throw an error upon receipt of a PING token, so the version-negotiation mechanism is used to prevent banana-v2 (0.1.2) peers from connecting to banana-v3 (0.1.3+) peers. Our negotiation mechanism would make it possible to detect the older (v2) peer and refrain from using PINGs, but that has not been done for this release. ** Tubs must be running before use Tubs are twisted.application.service.Service instances, and as such have a clear distinction between "running" and "not running" states. Tubs are started by calling startService(), or by attaching them to a running service, or by starting the service that they are already attached to. The design rule in operation here is that Tubs are not allowed to perform network IO until they are running. This rule was not enforced completely in 0.1.2, and calls to getReference()/connectTo() that occurred before the Tub was started would proceed normally (initiating a TCP connection, etc). Starting with 0.1.3, this rule *is* enforced. For now, that means that you must start the Tub before calling either of these methods, or you'll get an exception. In a future release, that may be changed to allow these early calls, and queue or otherwise defer the network IO until the Tub is eventually started. (the biggest issue is how to warn users who forget to start the Tub, since in the face of such a bug the getReference will simply never complete). ** Keepalives Tubs now keep track of how long a connection has been idle, and will send a few bytes (a PING of the other end) if no other traffic has been seen for roughly 4 to 8 minutes. This serves two purposes. The first is to convince an intervening NAT box that the connection is still in use, to prevent it from discarding the connection's table entry, since that would block any further traffic. The second is to accelerate the detection of such blocked connections, specifically to reduce the size of a window of buggy behavior in Foolscap's duplicate-connection detection/suppression code. This problem arises when client A (behind a low-end NAT box) connects to server B, perhaps using connectTo(). The first connection works fine, and is used for a while. Then, for whatever reason, A and B are silent for a long time (perhaps as short as 20 minutes, depending upon the NAT box). During this silence, A's NAT box thinks the connection is no longer in use and drops the address-translation table entry. Now suppose that A suddenly decides to talk to B. If the NAT box creates a new entry (with a new outbound port number), the packets that arrive on B will be rejected, since they do not match any existing TCP connections. A sees these rejected packets, breaks the TCP connection, and the Reconnector initiates a new connection. Meanwhile, B has no idea that anything has gone wrong. When the second connection reaches B, it thinks this is a duplicate connection from A, and that it already has a perfectly functional (albeit quiet) connection for that TubID, so it rejects the connection during the negotiation phase. A sees this rejection and schedules a new attempt, which ends in the same result. This has the potential to prevent hosts behind NAT boxes from ever reconnecting to the other end, at least until the the program at the far end is restarted, or it happens to try to send some traffic of its own. The same problem can occur if a laptop is abruptly shut down, or unplugged from the network, then moved to a different network. Similar problems have been seen with virtual machine instances that were suspended and moved to a different network. The longer-term fix for this is a deep change to the way duplicate connections (and cross-connect race conditions) are handled. The keepalives, however, mean that both sides are continually checking to see that the connection is still usable, enabling TCP to break the connection once the keepalives go unacknowledged for a certain amount of time. The default keepalive timer is 4 minutes, and due to the way it is implemented this means that no more than 8 minutes will pass without some traffic being sent. TCP tends to time out connections after perhaps 15 minutes of unacknowledged traffic, which means that the window of unconnectability is probably reduced from infinity down to about 25 minutes. The keepalive-sending timer defaults to 4 minutes, and can be changed by calling tub.setOption("keepaliveTimeout", seconds). In addition, an explicit disconnect timer can be enabled, which tells Foolscap to drop the connection unless traffic has been seen within some minimum span of time. This timer can be set by calling tub.setOption("disconnectTimeout", seconds). Obviously it should be set to a higher value than the keepaliveTimeout. This will close connections faster than TCP will. Both TCP disconnects and the ones triggered by this disconnectTimeout run the risk of false negatives, of course, in the face of unreliable networks. ** New constraints When a tuple appears in a method constraint specification, it now maps to an actual TupleOf constraint. Previously they mapped to a ChoiceOf constraint. In practice, TupleOf appears to be much more useful, and thus better deserving of the shortcut. For example, a method defined as follows: def get_employee(idnumber=int): return (str, int, int) # (name, room_number, age) can only return a three-element tuple, in which the first element is a string (specifically it conforms to a default StringConstraint), and the second two elements are ints (which conform to a default IntegerConstraint, which means it fits in a 32-bit signed twos-complement value). To specify a constraint that can accept alternatives, use ChoiceOf: def get_record(key=str): """Return the record (a string) if it is present, or None if it is not present.""" return ChoiceOf(str, None) UnicodeConstraint has been added, with minLength=, maxLength=, and regexp= arguments. The previous StringConstraint has been renamed to ByteStringConstraint (for accuracy), and it is defined to *only* accept string objects (not unicode objects). 'StringConstraint' itself remains equivalent to ByteStringConstraint for now, but in the future it may be redefined to be a constraint that accepts both bytestrings and unicode objects. To accomplish the bytestring-or-unicode constraint now, you might try schema.AnyStringConstraint, but it has not been fully tested, and might not work at all. ** Bugfixes Errors during negotiation were sometimes delivered in the wrong format, resulting in a "token prefix is limited to 64 bytes" error message. Several error messages (including that one) have been improved to give developers a better chance of determining where the actual problem lies. RemoteReference.notifyOnDisconnect was buggy when called on a reference that was already broken: it failed to fire the callback. Now it fires the callback soon (using an eventual-send). This should remove a race condition from connectTo+notifyOnDisconnect sequences and allow them to operate reliably. notifyOnDisconnect() is now tolerant of attempts to remove something twice, which should make it easier to use safely. Remote methods which raise string exceptions should no longer cause Foolscap to explode. These sorts of exceptions are deprecated, of course, and you shouldn't use them, but at least they won't break Foolscap. The Reconnector class (accessed by tub.connectTo) was not correctly reconnecting in certain cases (which appeared to be particularly common on windows). This should be fixed now. CopyableSlicer did not work inside containers when streaming was enabled. Thanks to iacovou-AT-gmail.com for spotting this one. ** Bugs not fixed Some bugs were identified and characterized but *not* fixed in this release *** RemoteInterfaces aren't defaulting to fully-qualified classnames When defining a RemoteInterface, you can specify its name with __remote_name__, or you can allow it to use the default name. Unfortunately, the default name is only the *local* name of the class, not the fully-qualified name, which means that if you have an RIFoo in two different .py files, they will wind up with the same name (which will cause an error on import, since all RemoteInterfaces known to a Foolscap-using program must have unique names). It turns out that it is rather difficult to determine the fully-qualified name of the RemoteInterface class early enough to be helpful. The workaround is to always add a __remote_name__ to your RemoteInterface classes. The recommendation is to use a globally-unique string, like a URI that includes your organization's DNS name. *** Constraints aren't constraining inbound tokens well enough Constraints (and the RemoteInterfaces they live inside) serve three purposes. The primary one is as documentation, describing how remotely-accessible objects behave. The second purpose is to enforce that documentation, by inspecting arguments (and return values) before invoking the method, as a form of precondition checking. The third is to mitigate denial-of-service attacks, in which an attacker sends so much data (or carefully crafted data) that the receiving program runs out of memory or stack space. It looks like several constraints are not correctly paying attention to the tokens as they arrive over the wire, such that the third purpose is not being achieved. Hopefully this will be fixed in a later release. Application code can be unaware of this change, since the constraints are still being applied to inbound arguments before they are passed to the method. Continue to use RemoteInterfaces as usual, just be aware that you are not yet protected against certain DoS attacks. ** Use os.urandom instead of falling back to pycrypto Once upon a time, when Foolscap was compatible with python2.3 (which lacks os.urandom), we would try to use PyCrypto's random-number-generation routines when creating unguessable object identifiers (aka "SwissNumbers"). Now that we require python2.4 or later, this fallback has been removed, eliminating the last reference to pycrypto within the Foolscap source tree. * Release 0.1.2 (04 Apr 2007) ** Bugfixes Yesterday's release had a bug in the new SetConstraint which rendered it completely unusable. This has been fixed, along with some new tests. ** More debian packaging Some control scripts were added to make it easier to create debian packages for the Ubuntu 'edgy' and 'feisty' distributions.

1 0

[Twisted-Python] custom work needed
by twisted＠dwalliance.com 07 Aug '07

07 Aug '07

Hi, We just posted this on guru.com but thought maybe one of you would be interested in helping us with this project. ***************** We need a special smtp server created using twisted-python for ubuntu linux(feisty fawn server)that will accept an email then send that email to multiple anti-spam servers. Each email needs to be tagged with an ID and when relayed needs to appear as if it was received from the original sender(ip spoofing). This SMTP server could be visualized as an ethernet bridge that puts an ID on each received email then tees the connection and sends that email to multiple servers simultaneously. The connections to the anti-spam servers need to be monitored for how long it takes to accept the email, reject the email, or the connection errors out. All this data needs to be inserted in real time to a mysql database. We expect that the smtp server will not act exactly like an ethernet bridge(pass through) since the sending will have significant overhead as compared to the receiving and that queuing will be a natural way for the system to manage a heavy workload. It is very important to us that the anti-spam servers see our computer as the previous sending server since many anti-spam systems rely on black-listed IP's. The receiving of emails will be from the internet the anti-spam servers will have limited access to the internet. The software should be able to send 10 messages or more per second. Receiving email should not be an issue since only simple queuing is required. This is an experimental project and well written and commented code is a requirement since we expect many updates to happen over a short amount of time. ************* Our guess is the stock twisted smtp server with twisted pair and some other foo like mysql db connection will be all that is needed. Hopefully no more than a few days work. We have a budget of $500 to $1000 if you are interested you should only reply to this email address and not the list. Thanks in advance! Jimmy Brake

1 0

[Twisted-Python] TDDing an ssh client - mocking the server
by Dave Kirby 06 Aug '07

06 Aug '07

Firstly a belated thanks to everyone who replied to my previous question - I had to put my project on hold for a while, but now I am back with more questions. My objective is to use twisted to do automated logging in to multiple ssh servers to carry out various tasks. Currently this is done by spawning a separate ssh executable for each session and controlling it through pexpect, and I would like to replace this with a twisted/conch system. So far I can create conch "exec" sessions to run a single command/script and return the result, but I would like to be able to create shell sessions and do more interactive work. Does anyone have any examples of doing this? The biggest issue I have at the moment is how to do Test Driven Development with conch. I would like to be able to mock out the server that the ssh client talks to, so that I can keep the unit tests self-contained and keep tight control of both sides of the interaction, e.g. for testing failure scenarios. I can think of three ways of doing this: 1) write a separate ssh server with conch that provides the desired behaviour, and have that spawned as a separate process by the unit tests. 2) as above, but run the ssh server in the same process as the tests, so they are both firing from the same reactor event loop. IMHO this is better than (1) since it is more self-contained. 3) stub out the low-level classes of the client so that they are not really talking to a server at all, but simulating the correct behaviour. The problem with both 1 and 2 is that they require writing test code that is as complex as the code I am testing. The problem with 3 is that it assumes that I know what the correct low-level behaviour is so that I can mock it out. At the moment I do not grok twisted in sufficient detail to do that. How have other people tackled this? Are there any other options that I am missing, or code I can reuse, or other suggestions or advice? TIA -- Dave Kirby Senior Software Engineer T: +44 20 7368 7330 | F: +44 20 7352 4922 E: d.kirby(a)tideway.com | W: www.tideway.com This email and any attachment may contain confidential, privileged information for the sole use of the intended recipient. If you are not the intended recipient, do not disclose, reproduce, disseminate or otherwise use this communication. If you received this communication in error, please immediately notify the sender via email and delete the communication from your system. Company information : Tideway Systems Ltd, Registered Office: Anchor House, 15-19 Britten Street, London, SW3 3TY. Registered in England & Wales Reg. Reg. No: 4598072 VAT No: 805 5153 50

3 2

FileFollower Was: [Twisted-Python] Where to start: log reader/analysis
by Yoann Aubineau 06 Aug '07

06 Aug '07

2007/8/6, Jean-Paul Calderone <exarkun(a)divmod.com>: > In order to make this class more usable within a Twisted application, I'd > make a few suggestions: Oh, thanks for this reply JP. I always love to get insightful feedbacks from "masters" :) This class was initially not intended to run in a twisted-aware environment, hence the use of time.sleep and no basic.LineReceiver. If you think it may be of any use amongs others Twisted protocols, I'm definitly willing to work on it : apply the changes you suggested, add unit tests, docs etc. Let me know. Yoann

1 0

[Twisted-Python] Design pattern for multi-stage web searches
by Luciano Ramalho 06 Aug '07

06 Aug '07

Hello, I am a Google Summer of Code student working with the Zope 3/Grok community this year. I wonder if there is an established design pattern or examples that use Twisted to fetch data via HTTP when the process can take a variable number of requests to be completed. For instance, if you search for book records online, some sites allow you to do an ISBN search which gives back the full book details, while others will present an intermediate results page from which links to the book details page can be obtained (many sites use a internal id for their products, including books). Still others can return an intermediate page containing a link to a different edition (!= ISBN), when the given ISBN refers to an out-of-print edition. So for each different metadata source I'd like to provide a chain of callbacks to be processed. Anyone knows of references that may help me? I browsed the GoF patterns but none of them fits this use case. BTW, the resulting code will be open source under the ZPL and hosted at http://svn.zope.org. Thanks for a great piece of software! Regards, Luciano PS. I've already implemented, using Twisted, a prototype of the collector for Amazon.com and it's affiliates in Europe and Japan. But I am Brazilian and I know their catalog is not very complete outside of the markets where they operate. To be really international, a book metadata collector must be pluggable to allow localized searches depending on the ISBN prefix.

2 2

Twisted August 2007