Mailman 3 August 2006 - Twisted

Re: [Twisted-Python] newbie question: is twisted right for my app?
by radix＠twistedmatrix.com Aug. 29, 2006

Aug. 29, 2006

On 07:12 pm, rspringob(a)vmware.com wrote: >I am designing a scheduling system and am wondering if twister is the right >tool for the job. The typical model is to have a server answering many >clients. I am looking at the inverse of having a solitary dispatcher (client) >contact many different hosts (servers) in order to pass them a job. Looking >at the examples in the documentation, it looks like the normal procedure is >to have one persistent connection per reactor. Am I trying to do a bad thing >with twister? No, Twiste*d* lets you use multiple connections with the same reactor; in fact, you can only have one reactor instance in your program. The reactor is an event loop; it handles many event sources at once.

1 0

[Twisted-Python] newbie question: is twisted right for my app?
by Richard Springob Aug. 29, 2006

Aug. 29, 2006

I am designing a scheduling system and am wondering if twister is the right tool for the job. The typical model is to have a server answering many clients. I am looking at the inverse of having a solitary dispatcher (client) contact many different hosts (servers) in order to pass them a job. Looking at the examples in the documentation, it looks like the normal procedure is to have one persistent connection per reactor. Am I trying to do a bad thing with twister? I didn't want to ramble, so let me know if you need more info in order to understand what I am trying to do here. Thanks, Rick

2 1

Re: [Twisted-Python] Direct access to main server class
by Terry Jones Aug. 29, 2006

Aug. 29, 2006

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> If you think you can hide Python code by compiling it to bytecodes, glyph> have a look at http://www.crazy-compilers.com/decompyle/ -- that glyph> should quickly disabuse you of that notion. No, I have no intention of doing that - rather I assume the opposite (that code and tests will all be available), that's what I thought I was saying. glyph> Moreover, any security scheme that relies, in any way, upon the fact glyph> that your code is hidden is broken by design. From wikipedia: glyph> "computer science professors consider security through obscurity so glyph> obviously wrong that the matter doesn't need proof or study". Yes, I agree that's it's almost always an error (and those that choose it reflexively are also particularly prone to the attendant false sense of security. One would hope that some form of Darwinian selection might kick in, but it never does). glyph> However, even disregarding that, _some_ code in your system is glyph> instantiating authenticated users and calling methods on them. glyph> Assuming that your code is packaged and distributed (tests included, glyph> so as to make testing of deployments easier), anyone who can read glyph> the tests can read the rest of the code, and discover the same glyph> entrypoints. Right, that's what I was saying. glyph> So, I disagree in the strongest possible terms: it *isn't* an issue, I don't understand how the above sentence follows from the one before (and no, I'm not trolling either!). glyph> and nobody should ever consider it as such when writing tests. The glyph> correct way to write unit tests is one unit at a time, and it should glyph> not be polluted by misguided security concerns. glyph> Apologies if this sounds too harsh. I'm not just trolling, I glyph> promise :). We have a lot of tests in the Twisted suite which bend glyph> over backwards to instantiate things through obtuse glyph> higher-than-unit-level interfaces (which often aren't really glyph> themselves public), rather than testing functionality directly. glyph> These tests are all overcomplex, error prone, and generally contain glyph> race conditions and fail intermittently for unrelated reasons. OK, I think we have the same aim: I'm trying to NOT write obtuse and complex tests. That's why I was originally asking if there's an easy way to make tests go through the front door (via twisted.cred) to get something they can start making calls on. I wanted to do that precisely to _avoid_ bending over backwards etc. I have a couple of hundred unit tests (a very small number, I know), of which only a few are more than about 4 lines. I'm trying to keep it that way. I have the feeling that my original mail should probably have been written more clearly. And it's time for a coffee... Regards, Terry

2 1

Re: [Twisted-Python] Direct access to main server class
by Terry Jones Aug. 28, 2006

Aug. 28, 2006

Hi Andrew >>>>> "Andrew" == Andrew Bennetts <andrew-twisted(a)puzzling.org> writes: Andrew> I'm guessing a significant proportion of your code you want to Andrew> test, probably the majority of it, has nothing directly to do with Andrew> authentication or permissions. Also, *unit* tests (as opposed to Andrew> functional tests or integration tests or end-to-end tests or Andrew> whatever other terms you prefer to use) test one thing at a time, Andrew> so tests for this code shouldn't also be testing authentication. Andrew> If *every* test has to authenticate, then you're just wasting Andrew> effort: the tests run slower, there's more junk to step over in a Andrew> debugger, and you gain no extra testing value by testing the exact Andrew> same thing in 50 (or 500...) different tests. That's why I'd use a setUp method: to log in to the server and store the result for use in the unit tests. Andrew> No, it's quite common, and sensible, for tests to go directly to Andrew> the units (i.e. objects and methods) they want to test, so that Andrew> they test precisely what they should test. Right, I can do that (and do do that in the to-be-converted code). I suppose I'll just make myself an authenticated user object and then call into internal methods. It does make me wonder though, still. What if you're writing a Twisted app that you intend to license for commercial companies to run on their intranets: You install the Python source and presumably the unit test code gets run at some point, and maybe remains on the server. Then you're showing the world how to access the innards of your service without going through the approved authentication. Same comments apply to simply putting the code online and letting people see how to get in to any instance they can get local (user) access to. What can be done to prevent this? The most obvious thing seems to be to verify the uid of the process trying to create an authenticated user, and have both your unit tests and twistd run under this uid. If you don't do something about this, a normal user on the system can simply import your module, create an instance, and start making method calls (possibly unsuccessfully, due to file system permissions on databases, etc). Maybe they wont get far, but it's an issue: especially if the system has been built to allow this, and the unit tests show precisely how to pull it off. Thanks again, Terry

2 1

Re: [Twisted-Python] Direct access to main server class
by Terry Jones Aug. 28, 2006

Aug. 28, 2006

>>>>> "JP" == Jean-Paul Calderone <exarkun(a)divmod.com> writes: JP> Aside from the rest of your question and my response, this sounds JP> inside out. The general idea is that the user object itself has JP> methods which implement the actions which the protocol allows to be JP> taken. If a user lacks permission to perform a particular action, or JP> has restrictions on the particulars with which they may invoke that JP> action, or whatever other logic may be necessary on a per-user basis, JP> that is represented by a different implementation of the login JP> interface. OK, thanks for this explanation. As for the inside-out part, I do want things to be that way. E.g., there are methods that anyone can call and what you get back depends on the set of permissions on the things you request and also on your own preferences. So it's not known a priori what the user can and can't do - at least not at that fine grained level. At higher levels, sure, I do understand about returning user objects that simply do not have access to prohibited methods. After sending my mail I realized I should have gone to look through the top-level docs before mailing. There I found mention of Trial. I was going to follow up, but decided to see what people would suggest. You don't mention Trial however, and nor has anyone else yet. (Go on, tell me it's deprecated :-)) JP> It may not necessarily be the case that factoring your code as I've JP> just described would be better than what you are doing now, but if you JP> haven't considered it before, it may be worth doing so now. I will certainly do it to some extent. JP> Your server is started up from a .tac file. Yes. JP> The .tac file creates some objects and gives them particular JP> relationships. Just one instance of my top-level class. That class knows how to provide password checkers. JP> The relationships created in the .tac file are necessary for correct JP> operation of the server. No. The main thing that's needed after my last round of changes is that API methods are called in the context of an authenticated user. JP> The unit tests have no way to run the code in the .tac file in order to JP> duplicate these relationships. Right, except if I hack an end run around twisted.cred. But I don't really _want_ to do that. And if I've started a server via myApp.tac, then it feels (though it may not be the case) like I'm asking for trouble by simply instantiating and using another instance of my top-level class. Although the point is to allow highly parallel access, I was intending to do that through (e.g., a thread pool) created by a single twistd application. I'm not sure I want to have possibly many of them running at once (though perhaps I should, since this is part of the point). Does that make sense? I could configure the unit tests to just use a different setup (different DB etc) so it has nothing to do with the server version. That's in fact what I used to do. I was mainly asking: now that I've built a nice new front door using twisted.cred, how do I convert my existing unit tests to all go through the door, changing as little as possible. >> 1) Fake it. Call directly into my code to make myself an authenticated >> user object, and then start passing it in to methods on my own >> class. This is ugly though, plus if the service is already running it's >> a bad solution because it goes behind the back of twisted and asks for >> trouble. JP> This actually sounds a little like what I described above. What makes JP> you say it is ugly? To me, it sounds like organizing your code so that JP> you can test it a unit at a time. The tests which do this are testing JP> the behavior of your code when authentication has already succeeded. JP> You should also have tests for your authentication code, of course - JP> but that's a separate unit. It feels ugly in the way that Trial (if I understand it) feels right. There's an approved way to access the service, and robust tests should go through it. I don't fully believe this - I guess some combination, as you suggest, is probably going to be best. It feels ugly because, as I said, having built a new front door, I immediately hack my way around it. >> 2) Make all the methods of my class available via RPC and have all the >> unit test code use them. This seems like a major pain, though I suppose >> it would work. JP> In the past I have done this, and I have regretted it. This kind of JP> test is unnecessarily expensive to run and unnecessarily difficult to JP> debug. Thanks. >> I'd much rather just have the unit test code connect as a special local >> user and get back the instance of MyClass created by my .tac file and >> call its methods. Is that easy to do? JP> If you move the code responsible for instantiating MyClass out of the JP> .tac file into a real module, then it is easy :) Yes, modulo having multiple instances. I can deal with that though. >> OK, sorry for another typically long posting.... JP> No problem. I hope this was of some help. Yes, it's great, thanks for answering. I feel like such a babe in the woods with Twisted! Part of it, I guess, is only really getting to work on this stuff when I'm not doing what I'm actually supposed to be doing. Here's one more question, about cred. Given your initial comment and my emerging understanding of what goes on, what you call a user object (perhaps only because I called it that) is what's elsewhere called an avatar. If so, it seems an avatar can be anything at all and that generally an avatar is an instance of some class, and that that class has methods which the connecting user is allowed to call. I almost want to say "don't bother answering, I know I must be right", but I won't delete this paragraph just in case there are any other fundamental comments anyone might make. Regards, Terry

3 2

[Twisted-Python] Newb question?
by Benjamin Henry Aug. 28, 2006

Aug. 28, 2006

My question(s): I'm working on a custom network protocol (application layer in TCP/IP model) for a custom network. Please don't try to talk me out of this. I'm limiting many of the other protocols, because this is in a highly secure network, and latency/bandwidth is key. The packet sizes are minimized by the protocol design, which is ideal. Anyway. I'm having some problems with figuring out what to include in the Protocol class, and what to handle in the server class. Basically, I'm using Telnet, SMTP, and FTP protocol python files as a reference, and each have different strutures and methods of doing similar things. Basically, the protocol I'm developing is very similar to Telnet, so I used this code as a base. It is necessary to have a state machine, but there is dynamic information to send as a response to some received messages. For example, the server may need to get information from a DB, and send it in a packet as a reponse to a request. I'll give you a sample implementation, and the remainder of the specific questions below: server = a client = b 1a = NEED, 1 - NEED is a request to send data from client to server, 1 is the option, or type of data 2b = HAVE, 1 - HAVE is a positive response 3a = NEED, 1 - re-iterated for the purpose of being a server. The line "1a" is initiated only by a UI command. The line "3a" is initiated by as a response to the request "2b". 4b = PASV, 1 - same as FTP. It's basically, "Well, I have data...what port can I send it on?" 4a = PORT, 1, 51000 - not the same as FTP. It's the command, option, and port number 5b = ACK - acknowledge ok...now they both disconnect, or stop communication on this port, and server dynamically opens port 51000 on port 51000: 6b - <<data stream>> 7a - <<md5 or something other to verify correct and complete>> Questions I have: This is a state machine. The server has a class derived from the Protocol class whose methods are called prior to the Protocol's. For example, since the Server is using the derived Protocol class (let's say it's PProto), whenever a "dataReceived" is fired, PProto picks it up, and has to call Protocol's dataReceived handler explicitly. Where should all of this message handling take place? Should I split up the duties, like anything that's a singular response to one line should be in the Protocol class, and anything dynamic can be in the PProto class? What about dynamic ports? Where should this be implemented? The alternative is to have many connections on the one server port. Would this cause any issues, so that I could eliminate the PASV command altogether? I need to read the data being sent by the client. How can I correctly intercept this data without effecting the operation of the Protocol, and also so I can port the Protocol class to other applications without rewriting the server? I'm a newbie :). Thanks for your help.

1 0

[Twisted-Python] Direct access to main server class
by Terry Jones Aug. 28, 2006

Aug. 28, 2006

Here's a hopefully not-too-dumb question... I have a class that provides a service that I plan to make accessible via Twisted. I wrote the class and various unit tests. So I could formerly do simple things like: import unittest class myTest: def setUp(): self.c = MyClass() def tearDown(): self.c.close() def testX(): self.assert_(self.c.X() == 4) I recently added users, passwords, and permissions to the system. So now many method calls need a requesting user object passed to them, and check that the user has permission to carry out the action. I added simple twisted.cred class to provide realms, avatars, portal with checkers, etc. In the myApp.tac file, I create an instance of my top-level class (MyClass above). In the case of providing access to my service via Nevow, I instantiate a NevowSite and pass it the portal. That is all, I think, as it should be. But now I'm left wondering what happens to the old unit testing code. It obviously cannot just create a MyClass instance. The unit testing code should be using the service by authenticating itself just like any other client. But I'm not sure how best to do that. The unit test code wants to perform all manner of tests, and preferably this would be done by just making available the instance of MyClass created in my app's .tac file, allowing the test code to call its methods. But somehow that no longer feels quite right - the connection is in fact local, but it's coming in somehow through twisted.cred. I hope I'm being more or less clear about my dilemma, and I suppose this must be a situation people have to deal with all the time. Two solutions come to mind: 1) Fake it. Call directly into my code to make myself an authenticated user object, and then start passing it in to methods on my own class. This is ugly though, plus if the service is already running it's a bad solution because it goes behind the back of twisted and asks for trouble. 2) Make all the methods of my class available via RPC and have all the unit test code use them. This seems like a major pain, though I suppose it would work. I'd much rather just have the unit test code connect as a special local user and get back the instance of MyClass created by my .tac file and call its methods. Is that easy to do? OK, sorry for another typically long posting.... Thanks for any help. Regards, Terry

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Aug. 27, 2006

Aug. 27, 2006

Bug summary ______________________ Summary for 2006-08-20 through 2006-08-27 Bugs opened: 16 Bugs closed: 19 Total open bugs: 690 (-3) |== Type Changes |== Priority Changes |== Component Changes |Defect: -2 |Highest: -2 |Conch: -1 |Enhancement: +0 |High: -2 |Core: +1 |Task: +2 |Normal: +3 |Mail: -3 |Low: +1 |Trial: +1 |Vfs: -1 |Web: +2 |Web2: +1 |Website: -1 |Words: +1 New / Reopened Bugs ______________________ ===== Highest ===== [#2009] iq response tracker doesn't clean up (opened by ralphm) (CLOSED, fixed) defect words http://twistedmatrix.com/trac/ticket/2009 [#2019] t.w.p.msn does not properly execute errbacks when an error packet is received (opened by habnabit) defect words http://twistedmatrix.com/trac/ticket/2019 [#1922] make twistd pluggable, nerf mktap (opened by tjs) enhancement core http://twistedmatrix.com/trac/ticket/1922 ===== High ===== [#2014] twisted.internet.ssl.CertificateOptions docstring suggests server-only usage; CO is agnostic (opened by exarkun) defect core http://twistedmatrix.com/trac/ticket/2014 ===== Normal ===== [#2007] test ticket announcer bot changes (opened by exarkun) (CLOSED, fixed) task core http://twistedmatrix.com/trac/ticket/2007 [#2010] zipstream.py (unzipIter) cannot unzip directory correctly on Windows (opened by jacobg23) defect core http://twistedmatrix.com/trac/ticket/2010 [#2011] xmlrpc should return failures for any exception in method (opened by ghazel) (CLOSED, invalid) enhancement web http://twistedmatrix.com/trac/ticket/2011 [#2012] HTTP HEAD Request fails if Content-length given (opened by garrison) defect web http://twistedmatrix.com/trac/ticket/2012 [#2013] Deprecate mktap and document usage of Twisted Application Plugins with twistd (opened by radix) task core http://twistedmatrix.com/trac/ticket/2013 [#2015] Generic SASL library (opened by exarkun) enhancement core http://twistedmatrix.com/trac/ticket/2015 [#2016] twisted.web2.dav.test.util.TestCase.setUp is slow and can create inconsistent directory structures (opened by dreid) defect web2 http://twistedmatrix.com/trac/ticket/2016 [#2017] t.w.client.getPage returning ConnectionDone: Connection was closed cleanly error (opened by dimator) defect web http://twistedmatrix.com/trac/ticket/2017 [#2018] document installation of third-party plugins with distutils (opened by radix) task core http://twistedmatrix.com/trac/ticket/2018 [#2006] trial can misreport leftover DelayedCalls (opened by exarkun) defect trial http://twistedmatrix.com/trac/ticket/2006 [#2013] Deprecate mktap and document usage of Twisted Application Plugins with twistd (opened by radix) task core http://twistedmatrix.com/trac/ticket/2013 ===== Low ===== [#2008] selectreactor masks the 'dict' builtin, and commits various other offences. (opened by jerub) task core http://twistedmatrix.com/trac/ticket/2008 Closed Bugs ______________________ ===== Highest ===== [#1936] loopback.loopback blocks, should return Deferred (opened by jml, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/1936 [#630 ] t.p.log.FileLogObserver has problems with timezone names (mostly on windows). (opened by jknight, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/630 [#1761] give wiki edit permissions to everyone who should have them (opened by glyph, closed by radix, invalid) enhancement core http://twistedmatrix.com/trac/ticket/1761 [#2009] iq response tracker doesn't clean up (opened by ralphm, closed by ralphm, fixed) defect words http://twistedmatrix.com/trac/ticket/2009 [#1992] IMAPServer.__cbFetch can raise uncaught AttributeError (opened by tvachon, closed by exarkun, fixed) defect mail http://twistedmatrix.com/trac/ticket/1992 [#1922] make twistd pluggable, nerf mktap (opened by tjs, closed by radix, ) enhancement core http://twistedmatrix.com/trac/ticket/1922 ===== High ===== [#1978] fetch [message number] body[1] processed incorrectly for non-multipart messages (opened by tvachon, closed by exarkun, fixed) defect mail http://twistedmatrix.com/trac/ticket/1978 [#1183] test_conch calls reactor.iterate() in two places (opened by exarkun, closed by exarkun, fixed) defect conch http://twistedmatrix.com/trac/ticket/1183 [#1139] [NEEP HELP] Update mktap.xhtml to reflect current best practices (opened by hypatia, closed by radix, duplicate) defect website http://twistedmatrix.com/trac/ticket/1139 ===== Normal ===== [#2007] test ticket announcer bot changes (opened by exarkun, closed by exarkun, fixed) task core http://twistedmatrix.com/trac/ticket/2007 [#1846] Test run python2.4 Twisted 2.4 tarball, Suse 9, 64 bit (opened by scmikes, closed by exarkun, fixed) task core http://twistedmatrix.com/trac/ticket/1846 [#1845] twisted 2.4 test run on Suse 9, 64 bit, python 2.5b1 (opened by scmikes, closed by exarkun, fixed) task core http://twistedmatrix.com/trac/ticket/1845 [#1202] twisted.mail.test.test_mail calls reactor.iterate() (opened by exarkun, closed by exarkun, fixed) defect mail http://twistedmatrix.com/trac/ticket/1202 [#1113] intermittent test failure in testCallLater (opened by warner, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/1113 [#2006] trial can misreport leftover DelayedCalls (opened by exarkun, closed by ralphm, ) defect trial http://twistedmatrix.com/trac/ticket/2006 [#1805] SFTP adapter doesn't translate deferred errors (opened by spiv, closed by spiv, fixed) defect vfs http://twistedmatrix.com/trac/ticket/1805 [#2011] xmlrpc should return failures for any exception in method (opened by ghazel, closed by ghazel, invalid) enhancement web http://twistedmatrix.com/trac/ticket/2011 [#261 ] Document multicast (opened by Jerub, closed by jerub, fixed) enhancement core http://twistedmatrix.com/trac/ticket/261 [#2013] Deprecate mktap and document usage of Twisted Application Plugins with twistd (opened by radix, closed by radix, ) task core http://twistedmatrix.com/trac/ticket/2013

1 0

Re: [Twisted-Python] Multicast XMLRPC
by Chaz. Aug. 26, 2006

Aug. 26, 2006

Chaz. wrote: > radix(a)twistedmatrix.com wrote: >> On 03:49 pm, eprparadocs(a)gmail.com wrote: >> >I have a XMLPRC client that works well under TCP and I am now trying >> to get >> >it to work under Multicast conditions. >> >> It is unlikely that you will be able to get the XMLRPC protocol to >> work over multicast, given that multicast is an unreliable transport, >> like UDP. HTTP and XMLRPC don't know how to deal with that. >> >> >Here is the "TCP" code I used: >> > >> >class StorageService(internet.TCPServer): >> > def __init__(self,tcpPort,configInfo): >> > r = StoragePeer(configInfo) >> > xmlrpc.addIntrospection(r) >> > internet.TCPServer.__init__(self,tcpPort,server.Site(r)) >> >> >> Subclassing the thinks in twisted.application.internet is not really >> how those classes are meant to be used. Why did you do this instead of >> just instantiating a TCPServer with the appropriate port and factory? >> >> >I changed the call to TCPServer to: >> > >> > >> >class StorageService(internet.TCPServer): >> > def __init__(self,tcpPort,configInfo): >> > r = StoragePeer(configInfo) >> > xmlrpc.addIntrospection(r) >> > internet.MulticastServer.__init__(self,tcpPort,server.Site(r)) >> > >> >I thought this would work since, but it doesn't. What I get returned >> is the >> >following error message: >> > >> > >> >Failed to load application: unbound method __init__() must be called >> with >> >MulticastServer instance as first argument (got StorageService instance >> >instead) >> >> This is a trivial error in your Python. You can't call methods of >> classes directly unless the first argument is an instance of that class. >> > > I know how much work I will need to do to get any UDP-like protocol to > work with EXACTLY-ONCE semantics. > It seems my one problem was that in the definition of my class... class StorageService(internet.TCPServer) I should have used class StorageService(internet.MulticastServer) That solved my immediate problem, though I did find out that XMLRPC does in fact assume that you have a connection oriented protocol underneath it. Now I will just have to fix that problem. Also for those of you that said you can't do: internet.TCPServer.__init__(self,...) I would suggest you are wrong. In fact that is exactly how subclassing works in Python. But that is for another time. Once again thanks! Chaz.

8 25

Re: [Twisted-Python] Multicast XMLRPC
by Chaz. Aug. 25, 2006

Aug. 25, 2006

radix(a)twistedmatrix.com wrote: > On 06:43 pm, eprparadocs(a)gmail.com wrote: > >If I use TCP and stick to the serial, synchronized semantics of RPC, > doing > >one call at a time, I have only a few ways to solve the problem. Do > one call > >at a time, repeat N times, and that could take quite a while. I could > do M > >spawnProcesses and have each do N/M RPC calls. Or I could use M > threads and > >do it that way. Granted I have M sockets open at a time, it is > possible for > >this to take quite a while to execute. Performance would be terrible (and > >yes I want an approach that has good to very good performance. After > all who > >would want poor to terrible performance?) > > Let's just focus on this one thing, ignoring other resource issues for > now, because I think it needs to be clarified. Maybe this isn't the > case, but it looks like you're totally misunderstanding how asynchronous > I/O works. > > Here is an important thing: You can deal with multiple requests at the > same time even with TCP. You don't need to wait for one result before > you can send the next request. Send a bunch of requests at once, > asynchronously, and then deal with each response as it comes in. This > Just Works with multiple TCP connections in Twisted. No need for > threads, no need for Broadcast or Multicast. > I understand about the state machine of Twisted and how it can do the requests. The problem is that I still get down to issuing 1000s of TCP requests (and on top of that XML-RPC, SOAP, PB or whatever). It does beg the question has anyone really used a twisted server (or client) to invoke thousands of simultaneous RPC requests? Chaz

2 3