Mailman 3 December 2005 - Twisted

Re: [Twisted-Python] Re: plus mode was Re: how winnt fileops work and what to do about it
by Brian Warner 31 Dec '05

31 Dec '05

Ah, sorry, there are two separate bugs, and this thread is dealing with the other one. Nevermind. -Brian (plus, this thread makes me realize that renaming the directory won't work anyway)

1 0

Re: [Twisted-Python] Re: r15451 - Fix test failures under windows by changing the eventual-send operation to
by Brian Warner 31 Dec '05

31 Dec '05

> I think that your eventual-send queue should probably be its own > event-queuing API. If you have: > > callLater(0, foo); callLater(0, bar) > > and you expect ordering between them, something like this is better: > > callLater(0, lambda : foo(); bar()) > or perhaps > myRunQueue.put(foo); myRunQueue.put(bar); callLater(0, myRunQueue.activate) > > You get the idea. The issue here is with multiple eventual-sends being called from other code. Specifically, the context was an ITransport-providing Loopback class, used in newpb unit tests, which needs to decouple transport.write() from the resulting proto.dataReceived(). The write() calls are being made from all over the place, with no good location to coordinate or batch them together. Technically I could rig it so that transport.write() queues the data and then oblige the writing Protocol to invoke transport.flush() (newpb has a place for this to go, to support more datagram-oriented transports someday), but then who's to say that whatever code invokes *that* won't want to run multiple times in a brief period of time? I heartily agree that callLater() is not the appropriate tool for this purpose. A week ago I thought it might be, but at that point I didn't realize that I need the ordering guarantee described above. The implementation of callFromThread() happens to match both my needs and my expectations about efficiency, but of course it isn't documented to provide either, plus this eventual-send functionality should be completely unrelated to threads. -Brian

1 0

[Twisted-Python] Re: Twisted-Python Digest, Vol 21, Issue 39
by Michael Schneider 31 Dec '05

31 Dec '05

On Windows, you might use time.clock() it is more precise (keep with time.time on UNIX Hope this helps, Mike > > >Message: 2 >Date: Sat, 31 Dec 2005 12:42:07 -0500 >From: James Y Knight <foom(a)fuhm.net> >Subject: Re: [Twisted-Python] Re: [Twisted-commits] r15451 - Fix test > failures under windows by changing the eventual-send operation to >To: Twisted general discussion <twisted-python(a)twistedmatrix.com> >Message-ID: <5F60A50B-53F9-4AC2-AECA-681882929FB3(a)fuhm.net> >Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed > > >On Dec 31, 2005, at 11:59 AM, James Y Knight wrote: > > > >>On Dec 31, 2005, at 1:39 AM, Brian Warner wrote: >> >> >> >>>callLater(0) does not guarantee >>>relative ordering of sequentially-scheduled calls, >>> >>> >>> >>Yes it does: order of method call. >> >> >> >> >>>and the windows >>>reactors in fact execute them in random order. >>> >>> >>> >>That is broken. >> >> > >I'm afraid this is my fault. Heapq does not guarantee stable ordering >amongst elements with the same key, and on windows, two successive >calls to time.time() are very likely to return the same number, >because of its poor resolution. The same error could probably occur >on unix/linux systems as well, although currently unlikely to the >point of impossibility. It should be fixable by storing the last time >of an element added to the queue, and adding epsilon if current time >is the same. > >Just as a point of interest, on my system, this loop never finishes: >import time >def f(): > while 1: > if time.time()==time.time(): > break > >But this one does: >from time import time >def f(): > while 1: > if time()==time(): > break > > >James > > > >------------------------------ > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python(a)twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > >End of Twisted-Python Digest, Vol 21, Issue 39 >********************************************** > > > -- The greatest performance improvement occurs on the transition of from the non-working state to the working state.

3 2

[Twisted-Python] Re: [Twisted-commits] r15451 - Fix test failures under windows by changing the eventual-send operation to
by James Y Knight 31 Dec '05

31 Dec '05

On Dec 31, 2005, at 1:39 AM, Brian Warner wrote: > callLater(0) does not guarantee > relative ordering of sequentially-scheduled calls, Yes it does: order of method call. > and the windows > reactors in fact execute them in random order. That is broken. James

1 1

[Twisted-Python] Qtreactor problem on twisted 2.1
by Umit Oztosun 31 Dec '05

31 Dec '05

Hello, Our application uses twisted as the core network framework. It basically consists of a server and multiple clients. All communication is carried on a twisted.spread.pb channel. We have been using twisted 1.3 since the beginning, we decided to switch to twisted 2.1, the switch was easy but an interesting problem showed up. Our client is based on PyQt, so uses qtreactor. If something goes wrong on the server side, the error is transferred to the client (automagically, thanks to twisted & deferreds), and handled there (with an errback). We warn our users about the failure with a dialog. The interesting part is, with twisted 2.1, at the dialog's exec_loop() call, occasionally the connection between the server and client goes away with the following exception: ------------------------------------------- 2005/12/30 15:51 GTB Standard Time [-] Error in <<class 'twisted.internet.tcp.TLSConnection'> to (u'localhost', 4243) at 1312e50>.doRead() 2005/12/30 15:51 GTB Standard Time [-] Traceback (most recent call last): File "C:\Python24\Lib\site-packages\twisted\internet\defer.py", line 182, in addCallbacks self._runCallbacks() File "C:\Python24\Lib\site-packages\twisted\internet\defer.py", line 307, in _runCallbacks self.result = callback(self.result, *args, **kw) File ".\lq\lqtrxutils.py", line 126, in eb_localFailed File ".\lq\lqdialog.py", line 201, in showDialog --- <exception caught here> --- File "C:\Python24\Lib\site-packages\twisted\internet\qtreactor.py", line 62, in read why = w.doRead() File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line 111, in doRead return Connection.doRead(self) File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line 347, in doRead return self.protocol.dataReceived(data) File "C:\Python24\Lib\site-packages\twisted\spread\banana.py", line 173, in dataReceived gotItem(self.incomingVocabulary[num]) File "C:\Python24\Lib\site-packages\twisted\spread\banana.py", line 114, in gotItem self.callExpressionReceived(item) File "C:\Python24\Lib\site-packages\twisted\spread\banana.py", line 81, in callExpressionReceived self.expressionReceived(obj) File "C:\Python24\Lib\site-packages\twisted\spread\pb.py", line 567, in expressionReceived raise ProtocolError("Non-list expression received.") twisted.spread.pb.ProtocolError: Non-list expression received. 2005/12/30 15:51 GTB Standard Time [-] <twisted.internet.ssl.Connector instance at 0x0134BE90> will retry in 2 seconds 2005/12/30 15:51 GTB Standard Time [-] Stopping factory <lqclientfactory.lqClientFactory instance at 0x0134BDF0> ------------------------------------------- Line 201 at lqdialog.py reads as result = self.exec_loop() We are using a reconnecting client factory, the application reconnects in a matter of seconds but this behaviour is annoying. As I wrote above, the we had no problem when using 1.3. Any ideas? Regards, and happy new year! Umit Oztosun P.S.: Tested on both Windows XP and Ubuntu Linux (5.10 and 5.04). Disconnection occurs regardless of using or not using SSL.

2 2

[Twisted-Python] SSL client authentication
by dkarbott＠comcast.net 29 Dec '05

29 Dec '05

Is it possible to do client authentication with reactor.listenSSL() and reactor.connectSSL()? If so, are there any examples? Any info would be much appreciated. Regards, -- Darrell

2 1

[Twisted-Python] Weekly Bug Summary
by James Y Knight 28 Dec '05

28 Dec '05

Bug summary ______________________ Since 2005-12-21 07:00:01.630000: Bugs opened: 10 Bugs closed: 5 Total open bugs: 506 (+5) Urgent: 1 (+0) Bug: 241 (+1) Feature: 180 (+3) Wish: 84 (+1) New / Reopened Bugs ______________________ ==== Bug ==== twisted.web.client.HTTPClientFactory sends Host header (and others) more than once (yesterday) http://twistedmatrix.com/bugs/issue1382 opened by trickos [TESTS, TRIAL] t.trial.test.test_output tests fail on win32 select reactor (yesterday) http://twistedmatrix.com/bugs/issue1380 opened by jml [TRIAL] Implementing setUpClass should not require ClassSuite (2 days ago) http://twistedmatrix.com/bugs/issue1379 opened by jml [TRIAL] Fail test suite on import error (4 days ago) CLOSED http://twistedmatrix.com/bugs/issue1377 opened by jml iocpreactor WindowsError: (0, 'De bewerking is voltooid') (3 days ago) http://twistedmatrix.com/bugs/issue1378 opened by ghazel [TRIAL] Organise Trial's unit tests (5 days ago) CLOSED http://twistedmatrix.com/bugs/issue1374 opened by jml ==== Feature ==== [CONCH] hashed host entries in known_hosts for conch (SSH) (4 days ago) http://twistedmatrix.com/bugs/issue1376 opened by count0 hashed host entries in known_hosts for conch (SSH) (4 days ago) http://twistedmatrix.com/bugs/issue1375 opened by count0 [CORE] Support twisted.internet.stdio on Windows (6 days ago) http://twistedmatrix.com/bugs/issue1373 opened by itamarst ==== Wish ==== [CORE] Remove string import (yesterday) http://twistedmatrix.com/bugs/issue1381 opened by therve Bugs Closed ______________________ ==== Bug ==== [TESTS, WEB] twisted.web2.test.test_http calls reactor.iterate() (3 months ago) http://twistedmatrix.com/bugs/issue1186 opened by exarkun, closed by jknight [TRIAL] Fail test suite on import error (4 days ago) http://twistedmatrix.com/bugs/issue1377 opened by jml, closed by jml [TRIAL] Organise Trial's unit tests (5 days ago) http://twistedmatrix.com/bugs/issue1374 opened by jml, closed by jml [TRIAL] Clean up Trial's Runner and Loader concepts (2 weeks ago) http://twistedmatrix.com/bugs/issue1355 opened by jml, closed by jml [CORE, TRIAL] Raising SkipTest in setUp is treated as an error (7 days ago) http://twistedmatrix.com/bugs/issue1372 opened by exarkun, closed by jml

1 0

Re: [Twisted-Python] Allowing a type to be transmitted by jelly?
by Brian Warner 27 Dec '05

27 Dec '05

>>I am pretty sure newpb, which Brian Warner was threatening to merge any >>day now, will support this much better than PB does. Yup. In oldpb, object graphs are serialized through recursive calls to Jelly, which has a big switch statement (in twisted.spread.jelly._Jellier.jelly) that uses a different one-object serializer for each type. To teach oldpb how to handle a new type, you have to modify this function, and the corresponding unjellier, and then arrange for your Broker to use the new _Jellier classes instead of the normal ones. Subclassing pb.Copyable does away with this, but as you've noticed it is both slightly inefficent (particularly if you're using __slots__ anyway) and requires extensive surgery or pre-marshalling to use with third-party code that doesn't happen to inherit from pb.Copyable. In newpb, object graphs are serialized through iterative calls to a collection of Slicers. Each time a new object needs to be serialized, the enclosing Slicers help decide (based upon both the type of the object and any adapters that have been registered for it) which new Slicer to use. To teach newpb how to serialize a new type, you register an adapter for it (e.g. registerAdapter(BooleanSlicer, bool, ISlicer)). To teach newpb how to unserialize the resulting stream, you define an Unslicer class (there is some metaclass magic which auto-registers the Unslicer for you.. I'm still going back and forth on the best way to do this). The advantage of the adapter/registration scheme is that you don't have to modify the inheritance set of the classes you want to serialize, so third-party code is easy to accomodate without surgery or pre-marshalling. Pre-marshalling is a serious nuisance, because to do it correctly means looking inside every possible container for further instances of the type that needs to be converted into something jelly can handle, which is just as much work as the final serialization itself. Take a look at branches/newpb-again, in the doc/pb directory, for some examples and descriptions of how to use newpb. The code itself is in the twisted/pb directory (unless we come up with a different name for it). It's still in development: it was in the trunk for a few hours last week, but hopefully I'll get the remaining test failures resolved and it will go back into the trunk some time this week. cheers, -Brian

1 0

Re: [Twisted-Python] Receiving a root from a pb client
by Brian Warner 27 Dec '05

27 Dec '05

william.lewis(a)uk.bnpparibas.com writes: > I am using pb to communicate in a client-server configuration. My client > inherits from PBClientFactory. My server has a remote_take_root() method > by which the client can connect to the server and pass in its root, thus > allowing two-way communication. Is there anyway for the server to > re-establish the connection (should it go down) in this setup, or is > this the sole responsibility of the client? The other posters in this thread have it exactly right. The RemoteReference that is received by your remote_take_root() method does not carry enough information to allow anyone else (including your server) to initiate a new connection to the Broker which hosts it. In this configuration, the "client" side (specifically the side which initiates the PB connection with a PBClientFactory) must be responsible for any sort of reconnection. Incidentally, in newpb this won't necessarily be the case. Both ends can listen on a TCP socket, and the RemoteReference that you receive may contain enough information to connect to its source. It is still a good idea to make exactly one side be responsible for things like reconnection, though, because otherwise you get a bunch of ugly race conditions as both sides attempt to connect at the same time. (newpb handles this for you, so you'll only get a single connection between the two Tubs, but it's still ugly). Also, the application level should be aware of the reconnection events, otherwise it becomes more difficult to be sure about which methods have been successfully invoked and which have not. cheers, -Brian

1 0

[Twisted-Python] Python 2.2 Support
by Jonathan Lange 27 Dec '05

27 Dec '05

Should we drop Python 2.2 support in Subversion head for Twisted 2.2? - Debian stable now comes with Python 2.3 - Twisted 2.1 has been released with Python 2.2 support Dropping Python 2.2 support has some good consequences: - We get to delete code - An entire class of bugs gets eliminated - Disc space can be freed on the muon buildslave Some bad consequences: - People wanting to run Twisted 2.2 (or svn) on ancient versions of Python won't be able to. cheers, jml

13 30