Hi, Mike C. Fletcher wrote:

http://blog.vrplumber.com/1353

Thanks, nice article. And I've tried that code snipped, but it didn't help much in my case, as response times are below 1 second. Please see these measurements, taken with apache bench. I'm testing response times on a simple status resource, which doesn't take long to calculate (i.e. doesn't even throw a deferred, but returns a response immediately): Document Path: /status Document Length: 202 bytes Concurrency Level: 10 Time taken for tests: 0.202266 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 38400 bytes HTML transferred: 20200 bytes Requests per second: 494.40 [#/sec] (mean) Time per request: 20.227 [ms] (mean) Time per request: 2.023 [ms] (mean, across all concurrent requests) Transfer rate: 182.93 [Kbytes/sec] received Now, measured while the server is under very small load: Document Path: /status Document Length: 202 bytes Concurrency Level: 10 Time taken for tests: 4.103465 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 38400 bytes HTML transferred: 20200 bytes Requests per second: 24.37 [#/sec] (mean) Time per request: 410.347 [ms] (mean) Time per request: 41.035 [ms] (mean, across all concurrent requests) Transfer rate: 9.02 [Kbytes/sec] received When putting the server under real load, those response times climb up to two seconds, so there must be something wrong. Can I somehow get the reactor's state, i.e. how many deferreds are waiting in the queue, how many threads are running concurrently, etc? How good is the idea of deferring File I/O to threads, i.e. threads.deferToThread(self.filehandle.write, data)? Another possibly blocking module might be the database api, but I'm using twisted's enterprise adbapi, which should be async, AFAICT. Maybe copying data around takes time. I'm sending around chunks of 64k size (streaming from the database to an external programm). Reducing chunk size to 1k helps somewhat (i.e. response time is seldom over 150ms, but can still climb up to > 0.5 seconds). Hum... external program.... maybe it's the self.transport.write() call which blocks several 100ms? Is it safe to write: d = threads.deferToThread(self.transport.write, dataChunk) (i.e. call transport.write from a thread?) How much resources do these deferToThread() deferreds eat? AFAICT, the reactor prepares a thread pool, which leads me to think that it's a well optimized implementation... Regards Markus

On 02:32 pm, markus@bluegap.ch wrote:

Requests per second: 494.40 [#/sec] (mean)

Now, measured while the server is under very small load:

Requests per second: 24.37 [#/sec] (mean)

When putting the server under real load, those response times climb up to

What is 'real load'? Are you talking about things in process with, but not related to, the web server?

two seconds, so there must be something wrong.

Maybe your server is slow? :)

Can I somehow get the reactor's state, i.e. how many deferreds are waiting in the queue, how many threads are running concurrently, etc?

There is no queue of Deferreds. They don't actually have anything to do with the reactor.

How good is the idea of deferring File I/O to threads, i.e. threads.deferToThread(self.filehandle.write, data)?

If you do indeed discover that you are waiting a long time to write your particular stuff to files, then that might help. It might also randomly interleave the data and corrupt your files, if 'data' is big enough.

Another possibly blocking module might be the database api, but I'm using twisted's enterprise adbapi, which should be async, AFAICT.

It does the operations in threads, yes. However, the threadpool will eventually fill up; the concurrency is fairly limited. (The default is 10 or so workers, I think).

Maybe copying data around takes time. I'm sending around chunks of 64k size (streaming from the database to an external programm). Reducing chunk size to 1k helps somewhat (i.e. response time is seldom over 150ms, but can still climb up to > 0.5 seconds).

That's a possibility that the "--profile" option to twistd which JP suggested might help you with. You'll see the functions copying data taking a lot of CPU time in that case.

Hum... external program.... maybe it's the self.transport.write() call which blocks several 100ms? Is it safe to write:

d = threads.deferToThread(self.transport.write, dataChunk)

(i.e. call transport.write from a thread?)

No. _All_ Twisted APIs are not thread safe. This call does not block though, and it is extremely, vanishingly unlikely that it is causing your problems. It just sticks some data into the outgoing queue and returns immediately.

How much resources do these deferToThread() deferreds eat? AFAICT, the reactor prepares a thread pool, which leads me to think that it's a well optimized implementation...

It's not particularly "optimized", in the sense that we haven't measured the performance or improved it much, but it's also not doing very much; put a value into a queue, get it out in a thread, do it: that's about all. It would certainly surprise me if that operation were taking 100ms. One quick measurement you can do to determine what might be causing this performance loss is to examine the server in 'top' while it is allegedly under load. Is it taking up 100% CPU? If not, then it's probably blocked on some kind of I/O in your application, or perhaps writing the log. If so, then there's some inefficient application code (or Twisted code) that you need to profile and optimize. The output of "strace -T" on your Twisted server *might* be useful if you discover that you're blocking on some kind of I/O.

On Mon, 02 Jul 2007 11:04:56 +0200, Markus Schiltknecht wrote:

[snip] Anyway, with very simple timing measures within the twisted server itself, I've figured out what was causing the delays: reactor.spawnProcess() takes more than a second. I knew that fork() was expensive, but that expensive?

This isn't normal.

What I'm doing now feels very dirty: I'm calling reactor.spawnProcess() from a thread. (Yes, I'm taking care that only one thread can spawn a process at any time.) At least on my Linux Dev-Box, that seems to work - and resolves my issue. But... calling fork() from a thread???

Er. That's not supported. The only Twisted API you can call from a thread is reactor.callFromThread. Jean-Paul

Hi, glyph@divmod.com wrote:

Markus, can you instrument spawnProcess and figure out which *part* of it is taking so long?

I give it a try. Especially, because I need to fiddle with process.Process anyway, see the mail I'm going to send in a minute or two...

Like exarkun says: this is not normal. (I wonder if we are unwittingly calling some blocking API in there? I don't think so, but...)

I'll try to do some more benchmarking.

In other words, this may appear to work, but if you suddenly burst into flame unexpectedly, don't blame *us*. And you probably will, eventually, if you keep doing that.

Yup, I'm aware of that. It's what I meant when calling it ugly or dirty or whatever I said. Regards Markus

Hi, Maarten ter Huurne wrote:

Does your server process use a lot of memory?

Thanks, but no, that's certainly not the problem here. reactor.spawnProcess() was it. See my other mail. Regards Markus