[Twisted-Python] New user questions
Well as the docs say, it's kind of difficult to know where to start...
I have an application that is client/server based using a simple custom protocol. The client sends one line of data, starting with the ascii STX and ending with ETX. The server processes the data and returns a response with the same STX [data] EXT format. Communications are done with SSL. The ability to handle high concurrency (200-300 TPS) is the main objective as this is a transaction processing environment. The application also uses Berkeley DB through the bsddb3 module as a data store.
I looked through the sections on writing servers, the application framework, and writing plugins. What approach sounds right for this particular application? Right now I am using mod python with apache, which works fine but is a bit of a resource hog and I'd also like to get away from having the extra dependency of apache + mod python.
Chris
Firstly, you need to set your real name on your email account.
On Feb 5, 2005, at 8:36 PM, snacktime wrote:
I have an application that is client/server based using a simple custom protocol. The client sends one line of data, starting with the ascii STX and ending with ETX. The server processes the data and returns a response with the same STX [data] EXT format. Communications are done with SSL. The ability to handle high concurrency (200-300 TPS) is the main objective as this is a transaction processing environment. The application also uses Berkeley DB through the bsddb3 module as a data store.
This sounds like a good fit for twisted.
I looked through the sections on writing servers, the application framework, and writing plugins. What approach sounds right for this particular application? Right now I am using mod python with apache, which works fine but is a bit of a resource hog and I'd also like to get away from having the extra dependency of apache + mod python.
You need to look at the section on writing protocols. The finger tutorial is a good thing to start with. You can completely ignore the parts about plugins. As for the application framework, you can pretty much just copy&paste example code, you probably don't need to get into the details.
James
You need to look at the section on writing protocols. The finger tutorial is a good thing to start with. You can completely ignore the parts about plugins. As for the application framework, you can pretty much just copy&paste example code, you probably don't need to get into the details.
I looked at the docs for writing servers and applications, that part was pretty easy. What I don't understand yet is how to write a server that calls a function that blocks. I'm assuming I will have to use Deferreds? Will have to spend some time looking at the examples on this and then I might have some more questions...:)
Chris
On Sat, 2005-02-05 at 21:05 -0800, snacktime wrote:
I looked at the docs for writing servers and applications, that part was pretty easy. What I don't understand yet is how to write a server that calls a function that blocks.
Either: 1. Restructure it to not block. For network code this tends to be a side effect of using Twisted. E.g. twisted.protocols.smtp.sendmail does a single logical operation, "send an email", but it does not block because actions internally are split up based on networking events driven by the event loop - socket readable, socket writable.
2. Actions that can't be broken up, typically a C extension call that may take a long time, can be run in a thread pool. See e.g. how twisted.enterprise.adbapi works.
bsddb may or may not count as blocking depending how you use it.
I looked at the docs for writing servers and applications, that part was pretty easy. What I don't understand yet is how to write a server that calls a function that blocks.
Either:
- Restructure it to not block. For network code this tends to be a side
effect of using Twisted. E.g. twisted.protocols.smtp.sendmail does a single logical operation, "send an email", but it does not block because actions internally are split up based on networking events driven by the event loop - socket readable, socket writable.
- Actions that can't be broken up, typically a C extension call that
may take a long time, can be run in a thread pool. See e.g. how twisted.enterprise.adbapi works.
bsddb may or may not count as blocking depending how you use it.
The problem I am having is getting my head around async io and using callbacks. I've just never had to deal with it before on this level.
Maybe someone could show me the basic logic of how I would structure the following to be non blocking?
1. Client sends message to server 2. Server does an https POST to another server and gets a response 3. Server sends a message back to the client and closes the connection.
On Sun, 2005-02-06 at 12:12 -0800, snacktime wrote:
Maybe someone could show me the basic logic of how I would structure the following to be non blocking?
- Client sends message to server
- Server does an https POST to another server and gets a response
- Server sends a message back to the client and closes the connection.
Server receives data from client -> server sends POST to another server. Server receives response from another server -> server sends message back to client and closes connection.
Server sending POST would probably use twisted.web.client and be a single function call, but internally be: Open connection to another server. Connection made -> send HTTP request. HTTP response receoved -> do something.
Receiving data is: Received data -> check if we've received enough to process protocol message, if so process message, otherwise buffer data.
On Sun, 06 Feb 2005 16:17:17 -0500, Itamar Shtull-Trauring itamar@itamarst.org wrote:
On Sun, 2005-02-06 at 12:12 -0800, snacktime wrote:
Maybe someone could show me the basic logic of how I would structure the following to be non blocking?
- Client sends message to server
- Server does an https POST to another server and gets a response
- Server sends a message back to the client and closes the connection.
Server receives data from client -> server sends POST to another server. Server receives response from another server -> server sends message back to client and closes connection.
Server sending POST would probably use twisted.web.client and be a single function call, but internally be: Open connection to another server. Connection made -> send HTTP request. HTTP response receoved -> do something.
Receiving data is: Received data -> check if we've received enough to process protocol message, if so process message, otherwise buffer data.
Ok, so basically I have to write my application so it does not block, which means anything that does block has to be rewritten using appropriate twisted components that don't block. For instance, there isn't a way to use urllib2 to do the POST and make it not block, correct?
So I guess the main problem that arises is when calling some code that blocks, but there is no twisted component that you can use to make the code non blocking. In which case you have to resort to using threads or processes correct?
Chris
On Sun, 6 Feb 2005 13:30:55 -0800, snacktime snacktime@gmail.com wrote:
On Sun, 06 Feb 2005 16:17:17 -0500, Itamar Shtull-Trauring itamar@itamarst.org wrote:
On Sun, 2005-02-06 at 12:12 -0800, snacktime wrote:
Maybe someone could show me the basic logic of how I would structure the following to be non blocking?
- Client sends message to server
- Server does an https POST to another server and gets a response
- Server sends a message back to the client and closes the
connection.
Server receives data from client -> server sends POST to another server. Server receives response from another server -> server sends message back to client and closes connection.
Server sending POST would probably use twisted.web.client and be a single function call, but internally be: Open connection to another server. Connection made -> send HTTP request. HTTP response receoved -> do something.
Receiving data is: Received data -> check if we've received enough to process protocol message, if so process message, otherwise buffer data.
Ok, so basically I have to write my application so it does not block, which means anything that does block has to be rewritten using appropriate twisted components that don't block. For instance, there isn't a way to use urllib2 to do the POST and make it not block, correct?
Correct.
So I guess the main problem that arises is when calling some code that blocks, but there is no twisted component that you can use to make the code non blocking. In which case you have to resort to using threads or processes correct?
Correct. http://twistedmatrix.com/documents/current/howto/threading
(or it should be possible to use Stackless Python with radix's Deferred support code, in the sandbox; it's experimental of course)
-Eric
Chris
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, 07 Feb 2005 10:08:03 -0600, Eric Mangold teratorn@twistedmatrix.com wrote:
On Sun, 6 Feb 2005 13:30:55 -0800, snacktime snacktime@gmail.com wrote:
So I guess the main problem that arises is when calling some code that blocks, but there is no twisted component that you can use to make the code non blocking. In which case you have to resort to using threads or processes correct?
Correct. http://twistedmatrix.com/documents/current/howto/threading
(or it should be possible to use Stackless Python with radix's Deferred support code, in the sandbox; it's experimental of course)
Not quite; stackless, greenlets, and so forth won't prevent you from blocking on blocking code (by which I mean code that blocks on system calls or synchronous CPU computations); they're essentially a way to make asynchronous stuff look synchronous. They may make it easier to rewrite such code to be asynchronous, but you'd still need to modify the code.
(Let's ignore Pavel's evil "LettuceWrapper" hack).
I'm having some problems playing with Deferreds and I think it's due to using threads. In the following code, will printResult be called in the thread? printResult is inside the protocol handler for a simple echo server, and when calling transport.write it doesn't seem to write anything, and I'm wondering if it's because it's being called inside a thread?
d = threads.deferToThread(doLongCalculation) d.addCallback(printResult)
It's not being called inside a thread; the callbacks of deferToThread are run in the main thread.
On Sun, 6 Feb 2005 18:28:57 -0800, snacktime snacktime@gmail.com wrote:
I'm having some problems playing with Deferreds and I think it's due to using threads. In the following code, will printResult be called in the thread? printResult is inside the protocol handler for a simple echo server, and when calling transport.write it doesn't seem to write anything, and I'm wondering if it's because it's being called inside a thread?
d = threads.deferToThread(doLongCalculation) d.addCallback(printResult)
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Sun, Feb 06, 2005 at 06:28:57PM -0800, snacktime wrote:
I'm having some problems playing with Deferreds and I think it's due to using threads. In the following code, will printResult be called in the thread? printResult is inside the protocol handler for a simple echo server, and when calling transport.write it doesn't seem to write anything, and I'm wondering if it's because it's being called inside a thread?
d = threads.deferToThread(doLongCalculation) d.addCallback(printResult)
In this code, printResult will not be called in a thread.
-Andrew.
participants (6)
-
Andrew Bennetts -
Christopher Armstrong -
Eric Mangold -
Itamar Shtull-Trauring -
James Y Knight -
snacktime