On 07:12 pm, rspringob(a)vmware.com wrote:
>I am designing a scheduling system and am wondering if twister is the right
>tool for the job. The typical model is to have a server answering many
>clients. I am looking at the inverse of having a solitary dispatcher (client)
>contact many different hosts (servers) in order to pass them a job. Looking
>at the examples in the documentation, it looks like the normal procedure is
>to have one persistent connection per reactor. Am I trying to do a bad thing
No, Twiste*d* lets you use multiple connections with the same reactor; in fact, you can only have one reactor instance in your program. The reactor is an event loop; it handles many event sources at once.
I am designing a scheduling system and am wondering if twister is the right
tool for the job. The typical model is to have a server answering many
clients. I am looking at the inverse of having a solitary dispatcher (client)
contact many different hosts (servers) in order to pass them a job. Looking
at the examples in the documentation, it looks like the normal procedure is
to have one persistent connection per reactor. Am I trying to do a bad thing
I didn't want to ramble, so let me know if you need more info in order to
understand what I am trying to do here.
>>>>> "glyph" == glyph <glyph(a)divmod.com> writes:
glyph> If you think you can hide Python code by compiling it to bytecodes,
glyph> have a look at http://www.crazy-compilers.com/decompyle/ -- that
glyph> should quickly disabuse you of that notion.
No, I have no intention of doing that - rather I assume the opposite (that
code and tests will all be available), that's what I thought I was saying.
glyph> Moreover, any security scheme that relies, in any way, upon the fact
glyph> that your code is hidden is broken by design. From wikipedia:
glyph> "computer science professors consider security through obscurity so
glyph> obviously wrong that the matter doesn't need proof or study".
Yes, I agree that's it's almost always an error (and those that choose it
reflexively are also particularly prone to the attendant false sense of
security. One would hope that some form of Darwinian selection might kick
in, but it never does).
glyph> However, even disregarding that, _some_ code in your system is
glyph> instantiating authenticated users and calling methods on them.
glyph> Assuming that your code is packaged and distributed (tests included,
glyph> so as to make testing of deployments easier), anyone who can read
glyph> the tests can read the rest of the code, and discover the same
Right, that's what I was saying.
glyph> So, I disagree in the strongest possible terms: it *isn't* an issue,
I don't understand how the above sentence follows from the one before (and
no, I'm not trolling either!).
glyph> and nobody should ever consider it as such when writing tests. The
glyph> correct way to write unit tests is one unit at a time, and it should
glyph> not be polluted by misguided security concerns.
glyph> Apologies if this sounds too harsh. I'm not just trolling, I
glyph> promise :). We have a lot of tests in the Twisted suite which bend
glyph> over backwards to instantiate things through obtuse
glyph> higher-than-unit-level interfaces (which often aren't really
glyph> themselves public), rather than testing functionality directly.
glyph> These tests are all overcomplex, error prone, and generally contain
glyph> race conditions and fail intermittently for unrelated reasons.
OK, I think we have the same aim: I'm trying to NOT write obtuse and
complex tests. That's why I was originally asking if there's an easy way to
make tests go through the front door (via twisted.cred) to get something
they can start making calls on. I wanted to do that precisely to _avoid_
bending over backwards etc. I have a couple of hundred unit tests (a very
small number, I know), of which only a few are more than about 4 lines. I'm
trying to keep it that way.
I have the feeling that my original mail should probably have been written
more clearly. And it's time for a coffee...
>>>>> "Andrew" == Andrew Bennetts <andrew-twisted(a)puzzling.org> writes:
Andrew> I'm guessing a significant proportion of your code you want to
Andrew> test, probably the majority of it, has nothing directly to do with
Andrew> authentication or permissions. Also, *unit* tests (as opposed to
Andrew> functional tests or integration tests or end-to-end tests or
Andrew> whatever other terms you prefer to use) test one thing at a time,
Andrew> so tests for this code shouldn't also be testing authentication.
Andrew> If *every* test has to authenticate, then you're just wasting
Andrew> effort: the tests run slower, there's more junk to step over in a
Andrew> debugger, and you gain no extra testing value by testing the exact
Andrew> same thing in 50 (or 500...) different tests.
That's why I'd use a setUp method: to log in to the server and store the
result for use in the unit tests.
Andrew> No, it's quite common, and sensible, for tests to go directly to
Andrew> the units (i.e. objects and methods) they want to test, so that
Andrew> they test precisely what they should test.
Right, I can do that (and do do that in the to-be-converted code). I
suppose I'll just make myself an authenticated user object and then call
into internal methods.
It does make me wonder though, still. What if you're writing a Twisted app
that you intend to license for commercial companies to run on their
intranets: You install the Python source and presumably the unit test code
gets run at some point, and maybe remains on the server. Then you're
showing the world how to access the innards of your service without going
through the approved authentication. Same comments apply to simply putting
the code online and letting people see how to get in to any instance they
can get local (user) access to. What can be done to prevent this? The most
obvious thing seems to be to verify the uid of the process trying to create
an authenticated user, and have both your unit tests and twistd run under
this uid. If you don't do something about this, a normal user on the
system can simply import your module, create an instance, and start making
method calls (possibly unsuccessfully, due to file system permissions on
databases, etc). Maybe they wont get far, but it's an issue: especially if
the system has been built to allow this, and the unit tests show precisely
how to pull it off.
>>>>> "JP" == Jean-Paul Calderone <exarkun(a)divmod.com> writes:
JP> Aside from the rest of your question and my response, this sounds
JP> inside out. The general idea is that the user object itself has
JP> methods which implement the actions which the protocol allows to be
JP> taken. If a user lacks permission to perform a particular action, or
JP> has restrictions on the particulars with which they may invoke that
JP> action, or whatever other logic may be necessary on a per-user basis,
JP> that is represented by a different implementation of the login
OK, thanks for this explanation. As for the inside-out part, I do want
things to be that way. E.g., there are methods that anyone can call and
what you get back depends on the set of permissions on the things you
request and also on your own preferences. So it's not known a priori what
the user can and can't do - at least not at that fine grained level. At
higher levels, sure, I do understand about returning user objects that
simply do not have access to prohibited methods.
After sending my mail I realized I should have gone to look through the
top-level docs before mailing. There I found mention of Trial. I was going
to follow up, but decided to see what people would suggest. You don't
mention Trial however, and nor has anyone else yet. (Go on, tell me it's
JP> It may not necessarily be the case that factoring your code as I've
JP> just described would be better than what you are doing now, but if you
JP> haven't considered it before, it may be worth doing so now.
I will certainly do it to some extent.
JP> Your server is started up from a .tac file.
JP> The .tac file creates some objects and gives them particular
Just one instance of my top-level class. That class knows how to provide
JP> The relationships created in the .tac file are necessary for correct
JP> operation of the server.
No. The main thing that's needed after my last round of changes is that API
methods are called in the context of an authenticated user.
JP> The unit tests have no way to run the code in the .tac file in order to
JP> duplicate these relationships.
Right, except if I hack an end run around twisted.cred. But I don't really
_want_ to do that. And if I've started a server via myApp.tac, then it
feels (though it may not be the case) like I'm asking for trouble by simply
instantiating and using another instance of my top-level class. Although
the point is to allow highly parallel access, I was intending to do that
through (e.g., a thread pool) created by a single twistd application. I'm
not sure I want to have possibly many of them running at once (though
perhaps I should, since this is part of the point). Does that make sense?
I could configure the unit tests to just use a different setup (different
DB etc) so it has nothing to do with the server version. That's in fact
what I used to do. I was mainly asking: now that I've built a nice new
front door using twisted.cred, how do I convert my existing unit tests to
all go through the door, changing as little as possible.
>> 1) Fake it. Call directly into my code to make myself an authenticated
>> user object, and then start passing it in to methods on my own
>> class. This is ugly though, plus if the service is already running it's
>> a bad solution because it goes behind the back of twisted and asks for
JP> This actually sounds a little like what I described above. What makes
JP> you say it is ugly? To me, it sounds like organizing your code so that
JP> you can test it a unit at a time. The tests which do this are testing
JP> the behavior of your code when authentication has already succeeded.
JP> You should also have tests for your authentication code, of course -
JP> but that's a separate unit.
It feels ugly in the way that Trial (if I understand it) feels right.
There's an approved way to access the service, and robust tests should go
through it. I don't fully believe this - I guess some combination, as you
suggest, is probably going to be best. It feels ugly because, as I said,
having built a new front door, I immediately hack my way around it.
>> 2) Make all the methods of my class available via RPC and have all the
>> unit test code use them. This seems like a major pain, though I suppose
>> it would work.
JP> In the past I have done this, and I have regretted it. This kind of
JP> test is unnecessarily expensive to run and unnecessarily difficult to
>> I'd much rather just have the unit test code connect as a special local
>> user and get back the instance of MyClass created by my .tac file and
>> call its methods. Is that easy to do?
JP> If you move the code responsible for instantiating MyClass out of the
JP> .tac file into a real module, then it is easy :)
Yes, modulo having multiple instances. I can deal with that though.
>> OK, sorry for another typically long posting....
JP> No problem. I hope this was of some help.
Yes, it's great, thanks for answering. I feel like such a babe in the woods
with Twisted! Part of it, I guess, is only really getting to work on this
stuff when I'm not doing what I'm actually supposed to be doing.
Here's one more question, about cred. Given your initial comment and my
emerging understanding of what goes on, what you call a user object
(perhaps only because I called it that) is what's elsewhere called an
avatar. If so, it seems an avatar can be anything at all and that generally
an avatar is an instance of some class, and that that class has methods
which the connecting user is allowed to call. I almost want to say "don't
bother answering, I know I must be right", but I won't delete this
paragraph just in case there are any other fundamental comments anyone
I'm working on a custom network protocol (application layer in TCP/IP
model) for a custom network. Please don't try to talk me out of this.
I'm limiting many of the other protocols, because this is in a highly
secure network, and latency/bandwidth is key. The packet sizes are
minimized by the protocol design, which is ideal. Anyway. I'm having
some problems with figuring out what to include in the Protocol class,
and what to handle in the server class. Basically, I'm using Telnet,
SMTP, and FTP protocol python files as a reference, and each have
different strutures and methods of doing similar things. Basically,
the protocol I'm developing is very similar to Telnet, so I used this
code as a base. It is necessary to have a state machine, but there is
dynamic information to send as a response to some received messages.
For example, the server may need to get information from a DB, and send
it in a packet as a reponse to a request. I'll give you a sample
implementation, and the remainder of the specific questions below:
server = a
client = b
1a = NEED, 1 - NEED is a request to send data from client to server,
1 is the option, or type of data
2b = HAVE, 1 - HAVE is a positive response
3a = NEED, 1 - re-iterated for the purpose of being a server. The
line "1a" is initiated only by a UI command. The line "3a" is
initiated by as a response to the request "2b".
4b = PASV, 1 - same as FTP. It's basically, "Well, I have data...what
port can I send it on?"
4a = PORT, 1, 51000 - not the same as FTP. It's the command, option,
and port number
5b = ACK - acknowledge
ok...now they both disconnect, or stop communication on this port, and
server dynamically opens port 51000
on port 51000:
6b - <<data stream>>
7a - <<md5 or something other to verify correct and complete>>
Questions I have:
This is a state machine. The server has a class derived from the
Protocol class whose methods are called prior to the Protocol's. For
example, since the Server is using the derived Protocol class (let's
say it's PProto), whenever a "dataReceived" is fired, PProto picks it
up, and has to call Protocol's dataReceived handler explicitly. Where
should all of this message handling take place? Should I split up the
duties, like anything that's a singular response to one line should be
in the Protocol class, and anything dynamic can be in the PProto class?
What about dynamic ports? Where should this be implemented? The
alternative is to have many connections on the one server port. Would
this cause any issues, so that I could eliminate the PASV command
I need to read the data being sent by the client. How can I correctly
intercept this data without effecting the operation of the Protocol,
and also so I can port the Protocol class to other applications without
rewriting the server?
I'm a newbie :).
Thanks for your help.
Here's a hopefully not-too-dumb question...
I have a class that provides a service that I plan to make accessible via
Twisted. I wrote the class and various unit tests. So I could formerly do
simple things like:
self.c = MyClass()
self.assert_(self.c.X() == 4)
I recently added users, passwords, and permissions to the system. So now
many method calls need a requesting user object passed to them, and check
that the user has permission to carry out the action. I added simple
twisted.cred class to provide realms, avatars, portal with checkers, etc.
In the myApp.tac file, I create an instance of my top-level class (MyClass
above). In the case of providing access to my service via Nevow, I
instantiate a NevowSite and pass it the portal. That is all, I think, as it
But now I'm left wondering what happens to the old unit testing code. It
obviously cannot just create a MyClass instance. The unit testing code
should be using the service by authenticating itself just like any other
client. But I'm not sure how best to do that. The unit test code wants to
perform all manner of tests, and preferably this would be done by just
making available the instance of MyClass created in my app's .tac file,
allowing the test code to call its methods. But somehow that no longer
feels quite right - the connection is in fact local, but it's coming in
somehow through twisted.cred.
I hope I'm being more or less clear about my dilemma, and I suppose this
must be a situation people have to deal with all the time.
Two solutions come to mind:
1) Fake it. Call directly into my code to make myself an authenticated
user object, and then start passing it in to methods on my own
class. This is ugly though, plus if the service is already running
it's a bad solution because it goes behind the back of twisted and
asks for trouble.
2) Make all the methods of my class available via RPC and have all the
unit test code use them. This seems like a major pain, though I
suppose it would work.
I'd much rather just have the unit test code connect as a special local
user and get back the instance of MyClass created by my .tac file and call
its methods. Is that easy to do?
OK, sorry for another typically long posting....
Thanks for any help. Regards,
> radix(a)twistedmatrix.com wrote:
>> On 03:49 pm, eprparadocs(a)gmail.com wrote:
>> >I have a XMLPRC client that works well under TCP and I am now trying
>> to get
>> >it to work under Multicast conditions.
>> It is unlikely that you will be able to get the XMLRPC protocol to
>> work over multicast, given that multicast is an unreliable transport,
>> like UDP. HTTP and XMLRPC don't know how to deal with that.
>> >Here is the "TCP" code I used:
>> >class StorageService(internet.TCPServer):
>> > def __init__(self,tcpPort,configInfo):
>> > r = StoragePeer(configInfo)
>> > xmlrpc.addIntrospection(r)
>> > internet.TCPServer.__init__(self,tcpPort,server.Site(r))
>> Subclassing the thinks in twisted.application.internet is not really
>> how those classes are meant to be used. Why did you do this instead of
>> just instantiating a TCPServer with the appropriate port and factory?
>> >I changed the call to TCPServer to:
>> >class StorageService(internet.TCPServer):
>> > def __init__(self,tcpPort,configInfo):
>> > r = StoragePeer(configInfo)
>> > xmlrpc.addIntrospection(r)
>> > internet.MulticastServer.__init__(self,tcpPort,server.Site(r))
>> >I thought this would work since, but it doesn't. What I get returned
>> is the
>> >following error message:
>> >Failed to load application: unbound method __init__() must be called
>> >MulticastServer instance as first argument (got StorageService instance
>> This is a trivial error in your Python. You can't call methods of
>> classes directly unless the first argument is an instance of that class.
> I know how much work I will need to do to get any UDP-like protocol to
> work with EXACTLY-ONCE semantics.
It seems my one problem was that in the definition of my class...
I should have used
That solved my immediate problem, though I did find out that XMLRPC does
in fact assume that you have a connection oriented protocol underneath
it. Now I will just have to fix that problem.
Also for those of you that said you can't do:
I would suggest you are wrong. In fact that is exactly how subclassing
works in Python. But that is for another time.
Once again thanks!
> On 06:43 pm, eprparadocs(a)gmail.com wrote:
> >If I use TCP and stick to the serial, synchronized semantics of RPC,
> >one call at a time, I have only a few ways to solve the problem. Do
> one call
> >at a time, repeat N times, and that could take quite a while. I could
> do M
> >spawnProcesses and have each do N/M RPC calls. Or I could use M
> threads and
> >do it that way. Granted I have M sockets open at a time, it is
> possible for
> >this to take quite a while to execute. Performance would be terrible (and
> >yes I want an approach that has good to very good performance. After
> all who
> >would want poor to terrible performance?)
> Let's just focus on this one thing, ignoring other resource issues for
> now, because I think it needs to be clarified. Maybe this isn't the
> case, but it looks like you're totally misunderstanding how asynchronous
> I/O works.
> Here is an important thing: You can deal with multiple requests at the
> same time even with TCP. You don't need to wait for one result before
> you can send the next request. Send a bunch of requests at once,
> asynchronously, and then deal with each response as it comes in. This
> Just Works with multiple TCP connections in Twisted. No need for
> threads, no need for Broadcast or Multicast.
I understand about the state machine of Twisted and how it can do the
requests. The problem is that I still get down to issuing 1000s of TCP
requests (and on top of that XML-RPC, SOAP, PB or whatever). It does beg
the question has anyone really used a twisted server (or client) to
invoke thousands of simultaneous RPC requests?