[Twisted-Python] Need a mother?
I am evaluating twisted matrix as the framework for an enterprise solution, a workflow/inventory/point of sale, does-all-kinds-of-things system. It needs to be highly secure, supporting SSL and encryption, and some decent Auth utilities. Our inclination is toward an n-tier solution for each store, with some connectivity between stores in a chain or under some kind of central management. Because there are many pieces to this puzzle, lots of optional services a system may need to provide and some specialized interfaces, my thought has been to take a component approach. This way we can drop in new components or update components as needed. We would like some way to centrally manage and load balance those components, within a single host as well as over multiple hosts. I am beginning to grasp how PB works, some. What I don't see is any kind of central registry for those components. It looks like the client needs to know the host and port of the component it wants in order to be able to use it. It looks to me like twisted.sister is meant to fulfill both of these functions, centralizing login, load balancing, name services, connecting clients to components. I think a mother is just the thing I need. Is it? If so, then how should I go about learning to use it. (I can create a parent tap easily enough with mktap, but I am uncertain how to customize it for our own authentication and encryption needs.) Anyone have examples of twisted sister in action? How far off are we from a stable implementation of twisted.sister? Our goal for the system we are developing is to roll it out in about a year. It would be nice if the framework components we used were at least semi-stable. Based on this very rough description, does twisted sound like the framework for me? Thanks for any assistance! -Stephen
On Sat, Mar 15, 2003 at 11:13:45AM -0600, Stephen Figgins wrote:
Since no one else has jumped in yet, I'll tell you what I know (not a lot). As it stands now, Twisted.sister has a problem with authorizers. Some stuff it depends on got changed around, and no one brought Sister up to date. I posted a patch that fixes it to the list a while back, but it seemed like a bad hack around a more basic problem. No one responded, so I never checked it in (I can make it available if you want and can't find it in the archives). This is just one symptom of the real problem, I think: Twisted.sister has no real users right now. Twisted practices XP development most of the time, which means the code that no one really has a use case for can sometimes get into pretty bad shape. I might be overstating the problems with Sister - as I said, I don't know too much about it - but in any case, I think many of the other features Twisted offers *do* make it suited for your goal. With a real user with real use cases, Sister would definitely see improvement (You'd yell at us when it broke, or if you're more proactive than that, you'd give us unit tests for it ;). So what's my point? Ahh, Twisted is good for this, but Sister might not be ready for what you need it to do, because no one has asked Sister to do *anything* before. Give it a nudge in the right direction and it will probably turn into what you need, or if it doesn't, doing what you need "from scratch" (on top of Twisted) shouldn't be much more effort. Hope this helps, Jp
A little history about the sister code... I (along with Glyph) was the original developer of this code for a game project. At one stage, it was fully integrated with authorizers and enterprise and I had distributed logins working, spawning of "game servers" on multiple machines in a cluster, object loading from a database, and centralized locking of resources across the distributed system. I had a set of twisted servers running my game world across three different machines on different operating systems almost transparently to the application code. Unfortunately, I am no longer at that company on that project and no-one is using or really maintaining the sister code. Jp is right in saying that it needs users - it really also needs developers. It's probably not a huge task to bring it back up to date, but i admit I haven't looked at the code in about four or five months and don't have the time or motivation to spend on it these days, but i would be willing to answer any questions if that is useful. Also, it's original purpose was actually more to distribute a single type of application rather than to connect a set of different applications as the origin poster suggests. Sean. -----Original Message----- From: twisted-python-admin@twistedmatrix.com [mailto:twisted-python-admin@twistedmatrix.com]On Behalf Of Jp Calderone Sent: Sunday, March 16, 2003 12:54 PM To: twisted-python@twistedmatrix.com Subject: Re: [Twisted-Python] Need a mother? On Sat, Mar 15, 2003 at 11:13:45AM -0600, Stephen Figgins wrote:
Since no one else has jumped in yet, I'll tell you what I know (not a lot). As it stands now, Twisted.sister has a problem with authorizers. Some stuff it depends on got changed around, and no one brought Sister up to date. I posted a patch that fixes it to the list a while back, but it seemed like a bad hack around a more basic problem. No one responded, so I never checked it in (I can make it available if you want and can't find it in the archives). This is just one symptom of the real problem, I think: Twisted.sister has no real users right now. Twisted practices XP development most of the time, which means the code that no one really has a use case for can sometimes get into pretty bad shape. I might be overstating the problems with Sister - as I said, I don't know too much about it - but in any case, I think many of the other features Twisted offers *do* make it suited for your goal. With a real user with real use cases, Sister would definitely see improvement (You'd yell at us when it broke, or if you're more proactive than that, you'd give us unit tests for it ;). So what's my point? Ahh, Twisted is good for this, but Sister might not be ready for what you need it to do, because no one has asked Sister to do *anything* before. Give it a nudge in the right direction and it will probably turn into what you need, or if it doesn't, doing what you need "from scratch" (on top of Twisted) shouldn't be much more effort. Hope this helps, Jp
Thanks for your responses, Jp and Sean. I hear you saying that if we need twisted.sister, we'll need to do some work on it first. It's a fixer-upper. If we use it, and contribute to its development, we might get some help from the twisted developers, who have largely neglected it because no one is currently using it. It also sounds like the primary task I want to put it towards, a central naming service/message queue sort of thing may not be the right match for twisted.sister. Though if fixed up, I sounds like it could load balance such a registry or an overtaxed component on my system. I am beginning to question my model of what the system should look like. This function seems basic to a distributed component system with a dozen or more components. Is anyone using Twisted.spread to implement such a system? Maybe my thoughts on this are too patterned on Corba. I am seeing Twisted as a lightweight asynchronous event-driven alternative to Corba, so I keep trying to map it to that: where is the POA? Where is the name server? I would sure appreciate any help wrapping my head around the twisted way of doing things. The problem set is: I want to create an n-tier application to manage stores in a vertical market that has many changing yet strict federally mandated security requirements, many interfaces to a variety of equipment (signature capture devices, dispensing devices, etc.), and interoperate with different B2B business partners. There is a limited set of services all stores will need, but each store/chain will have its own service needs. All stores/chains will need a common subset of services, but will also need a specialized set of extended services depending on their equipment, workflow and business partners. The clients need to be GUI and curses applications on Windows/Linux though possibly with an option for some thin web browser clients. Instead of thousands of clients hitting a central application intermittently (like a web server) Each store will have maybe a couple dozen clients that will interact with the server almost constantly throughout the day. It would be nice if the system could scale up (to multi-processor systems) as well as out (to a server farm) for larger chains which may want central hosting (though probably serving no more than a couple hundred stores.) What would be the twisted approach to this? Thanks, Stephen
On Mon, 17 Mar 2003 08:47:28 -0600 "Stephen R. Figgins" <fig@monitor.net> wrote:
Standard Twisted Feature Request Answer: Basically, no one has written one yet. Unfortanutely, we don't have the time to write every single a thing a distributed system might need, this being an open source project. So when someone eventually needs one, it will get written. Probably by them (i.e. by you?) although you could also pay someone to implement it. Otherwise you could just wait until one of the developers feels the need for a lookup service. Oh, and I'd love if it could use zeroconf in addition to a PB based lookup mechanism. -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting ***> Stop the war: http://unitedforpeace.org/ <***
On Monday, Mar 17, 2003, at 11:07 America/New_York, Itamar Shtull-Trauring wrote:
Oh, and I'd love if it could use zeroconf in addition to a PB based lookup mechanism.
Yeah, zeroconf would be great.. It's already really easy to publish and discover services using the PyObjC bridge, but only on OS X of course. -bob
Bob Ippolito wrote:
Yeah, zeroconf would be great.. It's already really easy to publish and discover services using the PyObjC bridge, but only on OS X of course.
There's a nice ZeroConf discovery implementation in pure python located at http://radio.weblogs.com/0105002/stories/2003/01/06/multicastDnsServiceDisco... I've been playing with it and it seems to work quite well. rich
On Mon, 17 Mar 2003 12:37:08 -0500 Rich Cavanaugh <rich@enflyer.com> wrote:
Except that it's licensed under "non-commercial use only", and uses threads, so it's not much help for us (other than being a nicer way to grok the protocol.)
Itamar Shtull-Trauring wrote:
I hadn't meant those questions as requests for features. I'm think my expecting these things to exist may be what is tripping me up. Instead of trying to make twisted fit my mental model of distributed computing, I want to know the twisted approach to my problem. Should I consider adding the functionality that fit's my model or is there another approach I should use instead? By the way, I will let the company I am working with know about Zoteca. HIPAA is an issue they need to work with on this project, and if they do go with a twisted solution, they might be interested in ZBE. -Stephen
On 3/18/03 11:29 AM, "Tommi Virtanen" <tv@twistedmatrix.com> wrote:
BINGO!
(sorry)
I would rather you elaborate than apologize. When you read my description and saw words like "n-tier" and "B2B", were you skeptical that I might need them? That they may be more buzzword than useful? Particularly when I use terminology like "n-tier" or perhaps just my use of "B2B"? Or were those terms not clear enough? Since I haven't received an answer to my questions yet, I worry that either nobody knows or you are all feeling annoyed reading my barrage of ignorant questions. Maybe I just haven't asked clearly enough, so I will try once more to be clearer in my request. The solution needn't be buzzword compliant. I just want to know the twisted approach. Considering what I have described, would you write one server and throw in all your basic components as modules to that server? Would you use Twisted's plug-in technology to add new features as you need them? Would you write components as servers/servants, and have them interact with each other via PB in a more DO approach? Would you write a front end server as a façade for those components? It looks like I could do any of the above and much more with Twisted. Is one way better than others? What has worked well for other large scale solutions? In a couple of articles on Python and Twisted, Aaron Trauring has made the point that multiple Twisted services (web, im, email, etc.) can all run in a single process. Is the single process approach desirable over a multiple component approach? Or is that only desirable when your needs are small and your application is unlikely to become CPU bound? There is so much here I am having a hard time making sense of it all. I sure would appreciate some direction from someone who understands twisted much better than I do. -Stephen
On Tue, Mar 18, 2003 at 03:03:53PM -0600, Stephen R. Figgins wrote:
Don't mind Tv's troll. Admittedly it was a litte amusing to see a buzzword-filled post, but nothing any of us would really get annoyed with. :-)
I think what most needs to be defined here is the interactions that are required between the different services. What hosts/ports the clients connect to the services with are pretty much irrelevant (i.e., facade). Once you have the interactions figured out, you can just use PB to have the different services talk to each other (if they need to be on separate hosts), as well as for the clients to talk to the services. Also, the plugin/component stuff is mostly fluff. Plugins are only useful when you want 3rd parties to add functionality to your service through some application-specific API, and I gather that's not going to be a requirement for you. Just know that it's probably simpler than you think. A few python servers that use PB to talk to each other. :)
If you need to be able to put the services on separate hosts, then yeah, you'll need to use separate processes and use PB to talk between them. If you don't, however, it's much easier and simpler to just let the services live in the same process and have synchronous access to all the objects in the system. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
On 3/18/03 3:40 PM, "Christopher Armstrong" <radix@twistedmatrix.com> wrote:
Just know that it's probably simpler than you think. A few python servers that use PB to talk to each other. :)
Thanks, Christopher. I think that is where I was headed, grouping related components into just a few servers, rather than having 50 different very specific components all communicating through some coordinating component. I can see the flexibility in small components, but the complexity seemed overwhelming, particularly without a central POA sort of server.
My concern here is that multiple processor systems are becoming much more affordable. Already dual Pentium systems are as common as dirt. It would be nice to have a design that could take advantage of such a system. Any ideas on how to make Twisted applications that can scale up to multiple processor systems? Maybe that is where we would need to work on getting Twisted Sister updated - to provide some kind of load balancing between multiple servers on the same host. -Stephen
On Tue, Mar 18, 2003 at 04:03:16PM -0600, Stephen R. Figgins wrote:
If you write the service such that it can use multiple processes (communicating via PB), then you can both take advantage of multi-CPU hosts as well as multiple distributed hosts. You would just run multiple nodes on one multi-CPU hosts, or one node per single-CPU host. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
On Tuesday, Mar 18, 2003, at 18:16 America/New_York, Christopher Armstrong wrote:
and once we get zeroconf into Twisted they can all find each other effortlessly on the LAN (and/or loopback).. how nice would that be :) -bob
On Sat, Mar 15, 2003 at 11:13:45AM -0600, Stephen Figgins wrote:
Since no one else has jumped in yet, I'll tell you what I know (not a lot). As it stands now, Twisted.sister has a problem with authorizers. Some stuff it depends on got changed around, and no one brought Sister up to date. I posted a patch that fixes it to the list a while back, but it seemed like a bad hack around a more basic problem. No one responded, so I never checked it in (I can make it available if you want and can't find it in the archives). This is just one symptom of the real problem, I think: Twisted.sister has no real users right now. Twisted practices XP development most of the time, which means the code that no one really has a use case for can sometimes get into pretty bad shape. I might be overstating the problems with Sister - as I said, I don't know too much about it - but in any case, I think many of the other features Twisted offers *do* make it suited for your goal. With a real user with real use cases, Sister would definitely see improvement (You'd yell at us when it broke, or if you're more proactive than that, you'd give us unit tests for it ;). So what's my point? Ahh, Twisted is good for this, but Sister might not be ready for what you need it to do, because no one has asked Sister to do *anything* before. Give it a nudge in the right direction and it will probably turn into what you need, or if it doesn't, doing what you need "from scratch" (on top of Twisted) shouldn't be much more effort. Hope this helps, Jp
A little history about the sister code... I (along with Glyph) was the original developer of this code for a game project. At one stage, it was fully integrated with authorizers and enterprise and I had distributed logins working, spawning of "game servers" on multiple machines in a cluster, object loading from a database, and centralized locking of resources across the distributed system. I had a set of twisted servers running my game world across three different machines on different operating systems almost transparently to the application code. Unfortunately, I am no longer at that company on that project and no-one is using or really maintaining the sister code. Jp is right in saying that it needs users - it really also needs developers. It's probably not a huge task to bring it back up to date, but i admit I haven't looked at the code in about four or five months and don't have the time or motivation to spend on it these days, but i would be willing to answer any questions if that is useful. Also, it's original purpose was actually more to distribute a single type of application rather than to connect a set of different applications as the origin poster suggests. Sean. -----Original Message----- From: twisted-python-admin@twistedmatrix.com [mailto:twisted-python-admin@twistedmatrix.com]On Behalf Of Jp Calderone Sent: Sunday, March 16, 2003 12:54 PM To: twisted-python@twistedmatrix.com Subject: Re: [Twisted-Python] Need a mother? On Sat, Mar 15, 2003 at 11:13:45AM -0600, Stephen Figgins wrote:
Since no one else has jumped in yet, I'll tell you what I know (not a lot). As it stands now, Twisted.sister has a problem with authorizers. Some stuff it depends on got changed around, and no one brought Sister up to date. I posted a patch that fixes it to the list a while back, but it seemed like a bad hack around a more basic problem. No one responded, so I never checked it in (I can make it available if you want and can't find it in the archives). This is just one symptom of the real problem, I think: Twisted.sister has no real users right now. Twisted practices XP development most of the time, which means the code that no one really has a use case for can sometimes get into pretty bad shape. I might be overstating the problems with Sister - as I said, I don't know too much about it - but in any case, I think many of the other features Twisted offers *do* make it suited for your goal. With a real user with real use cases, Sister would definitely see improvement (You'd yell at us when it broke, or if you're more proactive than that, you'd give us unit tests for it ;). So what's my point? Ahh, Twisted is good for this, but Sister might not be ready for what you need it to do, because no one has asked Sister to do *anything* before. Give it a nudge in the right direction and it will probably turn into what you need, or if it doesn't, doing what you need "from scratch" (on top of Twisted) shouldn't be much more effort. Hope this helps, Jp
Thanks for your responses, Jp and Sean. I hear you saying that if we need twisted.sister, we'll need to do some work on it first. It's a fixer-upper. If we use it, and contribute to its development, we might get some help from the twisted developers, who have largely neglected it because no one is currently using it. It also sounds like the primary task I want to put it towards, a central naming service/message queue sort of thing may not be the right match for twisted.sister. Though if fixed up, I sounds like it could load balance such a registry or an overtaxed component on my system. I am beginning to question my model of what the system should look like. This function seems basic to a distributed component system with a dozen or more components. Is anyone using Twisted.spread to implement such a system? Maybe my thoughts on this are too patterned on Corba. I am seeing Twisted as a lightweight asynchronous event-driven alternative to Corba, so I keep trying to map it to that: where is the POA? Where is the name server? I would sure appreciate any help wrapping my head around the twisted way of doing things. The problem set is: I want to create an n-tier application to manage stores in a vertical market that has many changing yet strict federally mandated security requirements, many interfaces to a variety of equipment (signature capture devices, dispensing devices, etc.), and interoperate with different B2B business partners. There is a limited set of services all stores will need, but each store/chain will have its own service needs. All stores/chains will need a common subset of services, but will also need a specialized set of extended services depending on their equipment, workflow and business partners. The clients need to be GUI and curses applications on Windows/Linux though possibly with an option for some thin web browser clients. Instead of thousands of clients hitting a central application intermittently (like a web server) Each store will have maybe a couple dozen clients that will interact with the server almost constantly throughout the day. It would be nice if the system could scale up (to multi-processor systems) as well as out (to a server farm) for larger chains which may want central hosting (though probably serving no more than a couple hundred stores.) What would be the twisted approach to this? Thanks, Stephen
On Mon, 17 Mar 2003 08:47:28 -0600 "Stephen R. Figgins" <fig@monitor.net> wrote:
Standard Twisted Feature Request Answer: Basically, no one has written one yet. Unfortanutely, we don't have the time to write every single a thing a distributed system might need, this being an open source project. So when someone eventually needs one, it will get written. Probably by them (i.e. by you?) although you could also pay someone to implement it. Otherwise you could just wait until one of the developers feels the need for a lookup service. Oh, and I'd love if it could use zeroconf in addition to a PB based lookup mechanism. -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting ***> Stop the war: http://unitedforpeace.org/ <***
On Monday, Mar 17, 2003, at 11:07 America/New_York, Itamar Shtull-Trauring wrote:
Oh, and I'd love if it could use zeroconf in addition to a PB based lookup mechanism.
Yeah, zeroconf would be great.. It's already really easy to publish and discover services using the PyObjC bridge, but only on OS X of course. -bob
Bob Ippolito wrote:
Yeah, zeroconf would be great.. It's already really easy to publish and discover services using the PyObjC bridge, but only on OS X of course.
There's a nice ZeroConf discovery implementation in pure python located at http://radio.weblogs.com/0105002/stories/2003/01/06/multicastDnsServiceDisco... I've been playing with it and it seems to work quite well. rich
On Mon, 17 Mar 2003 12:37:08 -0500 Rich Cavanaugh <rich@enflyer.com> wrote:
Except that it's licensed under "non-commercial use only", and uses threads, so it's not much help for us (other than being a nicer way to grok the protocol.)
Itamar Shtull-Trauring wrote:
I hadn't meant those questions as requests for features. I'm think my expecting these things to exist may be what is tripping me up. Instead of trying to make twisted fit my mental model of distributed computing, I want to know the twisted approach to my problem. Should I consider adding the functionality that fit's my model or is there another approach I should use instead? By the way, I will let the company I am working with know about Zoteca. HIPAA is an issue they need to work with on this project, and if they do go with a twisted solution, they might be interested in ZBE. -Stephen
On 3/18/03 11:29 AM, "Tommi Virtanen" <tv@twistedmatrix.com> wrote:
BINGO!
(sorry)
I would rather you elaborate than apologize. When you read my description and saw words like "n-tier" and "B2B", were you skeptical that I might need them? That they may be more buzzword than useful? Particularly when I use terminology like "n-tier" or perhaps just my use of "B2B"? Or were those terms not clear enough? Since I haven't received an answer to my questions yet, I worry that either nobody knows or you are all feeling annoyed reading my barrage of ignorant questions. Maybe I just haven't asked clearly enough, so I will try once more to be clearer in my request. The solution needn't be buzzword compliant. I just want to know the twisted approach. Considering what I have described, would you write one server and throw in all your basic components as modules to that server? Would you use Twisted's plug-in technology to add new features as you need them? Would you write components as servers/servants, and have them interact with each other via PB in a more DO approach? Would you write a front end server as a façade for those components? It looks like I could do any of the above and much more with Twisted. Is one way better than others? What has worked well for other large scale solutions? In a couple of articles on Python and Twisted, Aaron Trauring has made the point that multiple Twisted services (web, im, email, etc.) can all run in a single process. Is the single process approach desirable over a multiple component approach? Or is that only desirable when your needs are small and your application is unlikely to become CPU bound? There is so much here I am having a hard time making sense of it all. I sure would appreciate some direction from someone who understands twisted much better than I do. -Stephen
On Tue, Mar 18, 2003 at 03:03:53PM -0600, Stephen R. Figgins wrote:
Don't mind Tv's troll. Admittedly it was a litte amusing to see a buzzword-filled post, but nothing any of us would really get annoyed with. :-)
I think what most needs to be defined here is the interactions that are required between the different services. What hosts/ports the clients connect to the services with are pretty much irrelevant (i.e., facade). Once you have the interactions figured out, you can just use PB to have the different services talk to each other (if they need to be on separate hosts), as well as for the clients to talk to the services. Also, the plugin/component stuff is mostly fluff. Plugins are only useful when you want 3rd parties to add functionality to your service through some application-specific API, and I gather that's not going to be a requirement for you. Just know that it's probably simpler than you think. A few python servers that use PB to talk to each other. :)
If you need to be able to put the services on separate hosts, then yeah, you'll need to use separate processes and use PB to talk between them. If you don't, however, it's much easier and simpler to just let the services live in the same process and have synchronous access to all the objects in the system. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
On 3/18/03 3:40 PM, "Christopher Armstrong" <radix@twistedmatrix.com> wrote:
Just know that it's probably simpler than you think. A few python servers that use PB to talk to each other. :)
Thanks, Christopher. I think that is where I was headed, grouping related components into just a few servers, rather than having 50 different very specific components all communicating through some coordinating component. I can see the flexibility in small components, but the complexity seemed overwhelming, particularly without a central POA sort of server.
My concern here is that multiple processor systems are becoming much more affordable. Already dual Pentium systems are as common as dirt. It would be nice to have a design that could take advantage of such a system. Any ideas on how to make Twisted applications that can scale up to multiple processor systems? Maybe that is where we would need to work on getting Twisted Sister updated - to provide some kind of load balancing between multiple servers on the same host. -Stephen
On Tue, Mar 18, 2003 at 04:03:16PM -0600, Stephen R. Figgins wrote:
If you write the service such that it can use multiple processes (communicating via PB), then you can both take advantage of multi-CPU hosts as well as multiple distributed hosts. You would just run multiple nodes on one multi-CPU hosts, or one node per single-CPU host. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
On Tuesday, Mar 18, 2003, at 18:16 America/New_York, Christopher Armstrong wrote:
and once we get zeroconf into Twisted they can all find each other effortlessly on the LAN (and/or loopback).. how nice would that be :) -bob
participants (10)
-
Bob Ippolito -
Christopher Armstrong -
Itamar Shtull-Trauring -
Jp Calderone -
Rich Cavanaugh -
Samuel Bronson -
Sean Riley -
Stephen Figgins -
Stephen R. Figgins
-
Tommi Virtanen