Thanks for your reply. I already thought about that but I'll have other problems if I do so: - I need some variables to be common between the instances (Maybe it's not so difficult with RPC or a DB or shared memory). - I also need to keep a pool of all the outgoing the connections (to perform some real time stats and to limit the number of connection per user) But if I do not have other choices, it's certainly what I'll have to do... Yann. On Wed, Jun 24, 2009 at 21:08, Duncan M. McGreggor<duncan.mcgreggor@gmail.com> wrote:
Yann Pomarede wrote:
Hello,
I used Twisted to write some kind of proxy and it's working great, it's so easy to develop and understand.
But I have a problem. I use SSL (on both side of this app) and the bandwidth available is limited to the power of one of my CPU due to the high cost of the SSL encryption. I would like to use both core of my CPU to increase the bandwidth of this application but it does not seem easy to do.
# Here is the server definition in myapp.tac from twisted.application import internet, service application = service.Application('MyApp', uid=1001, gid=1001) internet.SSLServer(80, serverFactory, MySSLContext(), 50, 'xxx.xxx.xxx.xxx').setServiceParent(service.IServiceCollection(application))
# And here the clients connections from twisted.internet import reactor, ssl mySSLContext = ssl.ClientContextFactory() mySSLContext.method = SSL.SSLv23_METHOD reactor.connectSSL(self.HOST, self.PORT, self, mySSLContext)
Does anyone already have this kind of problem? Is there an easy way to multithread the SSL encryption part?
You could run two instances and load balance them...
d
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
2009/6/24 Yann Pomarede <yann.pomarede_twisted@gadz.org>:
Thanks for your reply. I already thought about that but I'll have other problems if I do so: - I need some variables to be common between the instances (Maybe it's not so difficult with RPC or a DB or shared memory). If you need full ACI (ACID without durability) this can be a performance issue.
- I also need to keep a pool of all the outgoing the connections (to perform some real time stats and to limit the number of connection per user)
The load balancer can limit number of connections (without decipher ssl) or agregate real time stats from web servers (for easier management). You can upgrade code in your web app without lost service too ;-)
But if I do not have other choices, it's certainly what I'll have to do...
Threads are not the solution. Python have a GIL (Global Inter-Lock) [1] and don't use multicore facilities. I don't know if you can do multiprocessing with twisted. A load balancer sound more easy (and maybe more horizontal scalable). [1] http://www.dabeaz.com/python/GIL.pdf Excuse my poor english. Regargs, Javi
Yann.
On Wed, Jun 24, 2009 at 21:08, Duncan M. McGreggor<duncan.mcgreggor@gmail.com> wrote:
Yann Pomarede wrote:
Hello,
I used Twisted to write some kind of proxy and it's working great, it's so easy to develop and understand.
But I have a problem. I use SSL (on both side of this app) and the bandwidth available is limited to the power of one of my CPU due to the high cost of the SSL encryption. I would like to use both core of my CPU to increase the bandwidth of this application but it does not seem easy to do.
# Here is the server definition in myapp.tac from twisted.application import internet, service application = service.Application('MyApp', uid=1001, gid=1001) internet.SSLServer(80, serverFactory, MySSLContext(), 50, 'xxx.xxx.xxx.xxx').setServiceParent(service.IServiceCollection(application))
# And here the clients connections from twisted.internet import reactor, ssl mySSLContext = ssl.ClientContextFactory() mySSLContext.method = SSL.SSLv23_METHOD reactor.connectSSL(self.HOST, self.PORT, self, mySSLContext)
Does anyone already have this kind of problem? Is there an easy way to multithread the SSL encryption part?
You could run two instances and load balance them...
d
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On Wed, Jun 24, 2009 at 4:00 PM, lasizoillo<lasizoillo@gmail.com> wrote:
2009/6/24 Yann Pomarede <yann.pomarede_twisted@gadz.org>:
Thanks for your reply. I already thought about that but I'll have other problems if I do so: - I need some variables to be common between the instances (Maybe it's not so difficult with RPC or a DB or shared memory). If you need full ACI (ACID without durability) this can be a performance issue.
- I also need to keep a pool of all the outgoing the connections (to perform some real time stats and to limit the number of connection per user)
The load balancer can limit number of connections (without decipher ssl) or agregate real time stats from web servers (for easier management). You can upgrade code in your web app without lost service too ;-)
But if I do not have other choices, it's certainly what I'll have to do...
Threads are not the solution. Python have a GIL (Global Inter-Lock) [1] and don't use multicore facilities. I don't know if you can do multiprocessing with twisted. A load balancer sound more easy (and maybe more horizontal scalable).
[1] http://www.dabeaz.com/python/GIL.pdf
Excuse my poor english.
A little bit of knowledge... Threads are indeed a possibility, because OpenSSL would be the thing doing the SSL work, and given that you use OpenSSL via a C extension, the C extension can release the GIL. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ http://canonical.com/
What about ampoule, https://launchpad.net/ampoule , can you check it? -- m 2009/6/24 lasizoillo <lasizoillo@gmail.com>
2009/6/24 Yann Pomarede <yann.pomarede_twisted@gadz.org>:
Thanks for your reply. I already thought about that but I'll have other problems if I do so: - I need some variables to be common between the instances (Maybe it's not so difficult with RPC or a DB or shared memory). If you need full ACI (ACID without durability) this can be a performance issue.
- I also need to keep a pool of all the outgoing the connections (to perform some real time stats and to limit the number of connection per user)
The load balancer can limit number of connections (without decipher ssl) or agregate real time stats from web servers (for easier management). You can upgrade code in your web app without lost service too ;-)
But if I do not have other choices, it's certainly what I'll have to do...
Threads are not the solution. Python have a GIL (Global Inter-Lock) [1] and don't use multicore facilities. I don't know if you can do multiprocessing with twisted. A load balancer sound more easy (and maybe more horizontal scalable).
[1] http://www.dabeaz.com/python/GIL.pdf
Excuse my poor english.
Regargs,
Javi
Yann.
On Wed, Jun 24, 2009 at 21:08, Duncan M. McGreggor<duncan.mcgreggor@gmail.com> wrote:
Yann Pomarede wrote:
Hello,
I used Twisted to write some kind of proxy and it's working great, it's so easy to develop and understand.
But I have a problem. I use SSL (on both side of this app) and the bandwidth available is limited to the power of one of my CPU due to the high cost of the SSL encryption. I would like to use both core of my CPU to increase the bandwidth of this application but it does not seem easy to do.
# Here is the server definition in myapp.tac from twisted.application import internet, service application = service.Application('MyApp', uid=1001, gid=1001) internet.SSLServer(80, serverFactory, MySSLContext(), 50,
'xxx.xxx.xxx.xxx').setServiceParent(service.IServiceCollection(application))
# And here the clients connections from twisted.internet import reactor, ssl mySSLContext = ssl.ClientContextFactory() mySSLContext.method = SSL.SSLv23_METHOD reactor.connectSSL(self.HOST, self.PORT, self, mySSLContext)
Does anyone already have this kind of problem? Is there an easy way to multithread the SSL encryption part?
You could run two instances and load balance them...
d
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
participants (4)
-
Christopher Armstrong -
lasizoillo -
Michal Pasternak -
Yann Pomarede