[Twisted-Python] Twisted receiving buffers swamped?
Hi, I am doing network performance tests using netperf on a trivial Twisted TCP echo server (code at the end). One of the tests that netperf offers is throughput, and I am running into an issue with this. When running the test (loopback) for 10 seconds on the test box I get a throughput of 11.5 Gb/s (which is not bad): [oberstet@brummer1 ~]$ netperf -N -H 127.0.0.1 -t TCP_STREAM -l 10 -- -P 9000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 9000 AF_INET to 127.0.0.1 () port 9000 AF_INET : no control : histogram : interval : dirty data : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 0 32768 32768 10.02 11517.89 [Sidenote: when running against the netperf server, the box does 46Gb/s on that test. More results here: https://github.com/crossbario/crossbar/wiki/Stream-Testee#netperf] However, when I run the test for 60 s (changing to "-l 60" in above will do), the test server is killed by the OS due to out-of-memory. This screenshot http://picpaste.com/pics/Clipboard03-DllXR7QE.1420107551.png shows that the server at the time immediately before of killing allocated >30GB RAM. In fact, memory also runs away with 10 sec test .. it's just that the machine has enough RAM to cope with that. So it's a "general" issue. I tested with: * CPython 2.7.9 and PyPy 2.4 * select, poll and kqueue reactors all on FreeBSD 10.1. Same behavior for all combinations. === Now, my suspicion is that Twisted is reading off the TCP stack from the kernel and buffering in userspace faster than the echo server is pushing out stuff to the TCP stack into the kernel. Hence, no TCP backpressure results, netperf happily sends more and more, and the memory of the Twisted process runs away. I am aware of http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html, but that seems to cover the sending side only. What's the cause? What can I do? How do I prevent Twisted to read off sockets from kernel as the userspace buffer grows? E.g. can I set a limit on the userspace buffer, so Twisted won't read out the sockets until the app has consumed more of the already buffered stuff? Any hints appreciated, Cheers, /Tobias TCP Echo Server used ====> from twisted.internet import kqreactor kqreactor.install() #from twisted.internet import selectreactor #selectreactor.install() #from twisted.internet import pollreactor #pollreactor.install() from twisted.internet import protocol, reactor, endpoints class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) class EchoFactory(protocol.Factory): def buildProtocol(self, addr): return Echo() endpoints.serverFromString(reactor, "tcp:9000").listen(EchoFactory()) print "running on ", reactor.__class__ reactor.run()
On Jan 1, 2015, at 2:21 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
I am aware of http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html <http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html>, but that seems to cover the sending side only.
It covers the receiving side as well. If you pauseProducing() on a transport, it stops calling dataReceived on its transport.
What's the cause? What can I do?
My initial hypothesis is that netperf is sending traffic but not bothering to receive it. If this hypothesis is correct, then self.transport.registerProducer(self.transport) should solve the problem. Presuming that there is no problem with crossing the streams - I don't think i've ever done that particular incantation, and I'm almost shocked it's taken this long to come up :). -glyph
I am aware of http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html, but that seems to cover the sending side only.
It covers the receiving side as well. If you pauseProducing() on a transport, it stops calling dataReceived on its transport.
Not sure I understand that. But you say, this will stop Twisted reading incoming data from a socket into userspace? And hence TCP backpressure results?
What's the cause? What can I do?
My initial hypothesis is that netperf is sending traffic but not bothering to receive it.
I haven't looked through the netperf sources .. but I guess netperf will send as fast as the receiving side can digest .. only throttle down because of TCP backpressure, not app-level flow-control.
If this hypothesis is correct, then self.transport.registerProducer(self.transport) should solve the problem. Presuming that there is no problem with crossing the streams - I don't think i've ever done that particular incantation, >and I'm almost shocked it's taken this long to come up :).
Unfortunately, it doesn't seem to work (the problem persists): https://github.com/oberstet/scratchbox/blob/master/python/asyncio/tcp_echo_s... http://picpaste.com/pics/Clipboard07-HlqQmTW0.1420188656.png Btw: the problem also arises when running over real network .. at least fast networks. I tested on fully switched 10GbE. And: asyncio (Trollius that is), has the same issue. The target server just gets swamped .. and then killed. Cheers, /Tobias
On Jan 2, 2015, at 2:25 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
Unfortunately, it doesn't seem to work (the problem persists):
Your "streaming" flag is wrong. A TCP transport is an IPushProducer (it will produce data without being asked). Try setting it to True and see if that helps? -g
Unfortunately, it doesn't seem to work (the problem persists): Your "streaming" flag is wrong. A TCP transport is an IPushProducer (it will produce data without being asked). Try setting it to True and see if that helps?
With streaming == True and cProfile added https://github.com/oberstet/scratchbox/blob/master/python/asyncio/tcp_echo_s... I get strange results. Sluggish performance: [oberstet@brummer1 ~]$ netperf -N -H 127.0.0.1 -t TCP_STREAM -l 10 -- -P 9000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 9000 AF_INET to 127.0.0.1 () port 9000 AF_INET : no control : histogram : interval : dirty data : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 0 32768 32768 10.03 0.31 === That is 310kb/s. Which is totally slow. The native C based netserver that comes with netperf does 46Gb/s on this test. With streaming == False, I get something like 11Gb/s with that Twisted server (until it collapsed due to OOM) The CPU load is very low (near 0%). The memory stays flat. Here is the profile: https://github.com/oberstet/scratchbox/blob/master/python/asyncio/profile.lo... /Tobias
Hi glyph,
I get strange results. Sluggish performance:
Did you ever diagnose this further? This seems like the sort of thing that we should start having a performance test for.
Not yet. I didn't reply again since you gave me enough homework already: - Run the producer/consumer variant on Linux (bisecting BSD/kqueue) - Do the memory profiling with non-producer/consumer (tracking down _where_ memory runs away) Other stuff interrupted me again, and my impression is, that it might be significant effort to really track this down. No surprise here: really pushing things often means "issues" pop up. I absolutely agree: we should have repeatable, comparable, standard performance tests. Like we have with trial/buildbot, but for performance, not functional tests. FWIW, here are my thoughts on this: 1) A simple Twisted based "TCP echo server" (maybe in non-producer/consumer and producer/consumer variants) as a testee will already allow us to do a _lot_. We can come up with more testees later (e.g. Twisted Web with static resource, ...). 2) It might be wise to use a non-Twisted, standard test load generator like netperf, instead of a Twisted based one. - having the load generator written in Twisted creates a cyclic dependency (e.g. rgd. interpreting results) - it allows to compare results to non-Twisted setups and allows others to repeat against their stuff 3) We should include at least 2 operating systems (FreeBSD / Linux). This allows to quickly bisect OS or Twisted reactor specific issues. 4) We should run this on real, physical, non-virtualized, dedicated hardware and networking gear. I can't stress enough how important this is in my experience: Any form of virtualization brings a whole own dimension of factors/variability into the game. Testing in VMs on a shared hypervisor on a public cloud: you never really know, you never really can repeat. Repeatability is absolutely crucial. 5) The load generator and the testee should run on 2 separate boxes, connected via real network (e.g. switched ether). Testing via loopback is often misleading, and practically often irrelevant (too far away from production deployments). 6) We should test on both CPython and PyPy. Because this is where stuff actually runs later in production. And for bisecting Python implementation specifics. 7) It should be automated. 8) The results should be stored in a long term archive (a database) so we can compare results over time / setups. 9) We should collect monitoring parameters (CPU load ...) on both the load generator and testee boxes during test runs. Like, "same network perf., but one triggers double the CPU load" .. === Because of 3/4/5, this requires 4 boxes to begin with. Those should be absolutely _identical_. Currently, we (Tavendo) have a setup dedicated to performance tests consisting of 2 boxes with dual port 10GbE and a 8 port 10GbE switch. Buying 2 more identical boxes and adding those would be technically possible. 7/8/9 and setting this all up is work. I would need to somehow justify/book these investments. I have "ideas" about that, but step by step: what do you think about above? /Tobias
On Jan 10, 2015, at 12:33 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
I would need to somehow justify/book these investments. I have "ideas" about that, but step by step: what do you think about above?
It sounds like http://speed.twistedmatrix.com <http://speed.twistedmatrix.com/> but far more ambitious :). Are you familiar with that site, and the benchmarks repository that powers it? It's nowhere near as comprehensive as what you'd like, but it is a good place to start. I'm particularly curious about the specific performance case we were talking about though; wire speed with flow-control is an important use-case for Twisted and it sounds to me like we may be falling very far short. -glyph
I would need to somehow justify/book these investments. I have "ideas" about that, but step by step: what do you think about above?
It sounds like http://speed.twistedmatrix.com but far more ambitious :). Are you familiar with that site, and the benchmarks repository that powers it? It's nowhere near as comprehensive as what you'd like, but it is a good >place to start.
I have stumbled across this before, but haven't looked into much. I am looking right now .. On the one hand: it has a _lot_ of testee scenarios, not just TCP echo, but e.g. also TLS, which I am definitely interested in. I was only proposing to have a trivial "TCP echo", essentially the couple of lines from the Twisted homepage - not very ambitious;) However, as far as I can see, speed.twistedmatrix.com runs all tests over loopback with both the load generator and the testee in Twisted - even in 1 process. http://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/view/head:... => uses producer/consumer https://github.com/twisted/twisted/blob/trunk/twisted/protocols/wire.py#L17 => does not use producer/consumer (like you suggested https://github.com/oberstet/scratchbox/blob/master/python/asyncio/tcp_echo_s...) For various reasons, I don't think this would have been able to demonstrate the issue I ran into .. swamping. You need a very fast sender (netperf) to even be able to swamp the receiver. Then I am not interested (much) in loopback. It is these performance test "setup/system things" were my proposal is indeed far more ambitious. And for me this is crucial: I want hard numbers which are _representative_ of what happens in production (at least close to).
I'm particularly curious about the specific performance case we were talking about though; wire speed with flow-control is an important use-case for Twisted and it sounds to me like we may be falling very far short.
Yes, I understand. Note that we can't get to 10GbE wirespeed in a throughput test with Twisted even leaving "swamping" aside. And I verified that the hardware/OS is not the bottleneck (I can saturate the link at 9.94Gb/s using netperf-to-netserver easily). /Tobias
It sounds like http://speed.twistedmatrix.com but far more ambitious :). Are you familiar with that site, and the benchmarks repository that powers it? It's nowhere near as comprehensive as what you'd like, but it is a good place to start.
I've look into it a little. I am confused;) E.g. take "SSL throughput big writes": http://picpaste.com/pics/Clipboard01-0isEvjph.1420884805.png There is a big dropoff in commit 43146. It's cool to see a history of performance correlated with commits. Now, if I dig into that commit, I see: http://picpaste.com/pics/Clipboard02-HV47NdTC.1420884887.png The commit seems to be a "doc only" commit. No actual code changes at all. How should I interpret that? Probably the test machine was changed, a new version of OpenSSL or pyOpenSSL, or something else? I'd say: the infrastructure aspects when doing performance tests do matter. To the degree that performance results are of very limited value at all, if the former aspects are not accounted for. /Tobias
On Jan 10, 2015, at 02:22, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
It sounds like http://speed.twistedmatrix.com but far more ambitious :). Are you familiar with that site, and the benchmarks repository that powers it? It's nowhere near as comprehensive as what you'd like, but it is a good place to start.
I've look into it a little. I am confused;)
E.g. take "SSL throughput big writes":
http://picpaste.com/pics/Clipboard01-0isEvjph.1420884805.png
There is a big dropoff in commit 43146.
It's cool to see a history of performance correlated with commits.
Now, if I dig into that commit, I see:
http://picpaste.com/pics/Clipboard02-HV47NdTC.1420884887.png
The commit seems to be a "doc only" commit. No actual code changes at all.
How should I interpret that?
Probably the test machine was changed, a new version of OpenSSL or pyOpenSSL, or something else?
One of those things. There is no infrastructure in place for identifying events which impact the performance testing infrastructure. The only performance testing environment is a very old mac mini still running Snow Leopard, which is probably the environment which we care the *least* about performance in, so it's not in great shape ;).
I'd say: the infrastructure aspects when doing performance tests do matter. To the degree that performance results are of very limited value at all, if the former aspects are not accounted for.
I don't think the results that we have presently are worth much at all. My point was mostly that there is some infrastructure which is halfway usable, and so you don't have to start from scratch. If you could take over this project (I am pretty sure at this point there is nobody to take it over *from*, exarkun did some work a long time ago and hasn't given it a second look in years) it would be highly appreciated! (And if you care a lot about performance in a particular environment you could set it up in that environment and get attention for it :)). You should also have a look at the existing benchmark suite and potentially look at maintaining / expanding that as well. Thoughts? -glyph
As someone partially responsible for the infrastructure Mozilla uses to do its performance benchmarking, I can say that it's *really* hard. Getting live operating systems to sit still and behave is a mess, and then *keeping* them still over months and years (while attending to necessary security upgrades, hardware migrations, and so on) is even worse. One of the smarter things we've figured out how to do is to "phase in" potentially disruptive changes so that we can either see that there's no impact, or estimate a correction factor for comparing results before and after the change. Dustin On Sat, Jan 10, 2015 at 6:47 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jan 10, 2015, at 02:22, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
It sounds like http://speed.twistedmatrix.com but far more ambitious :). Are you familiar with that site, and the benchmarks repository that powers it? It's nowhere near as comprehensive as what you'd like, but it is a good place to start.
I've look into it a little. I am confused;)
E.g. take "SSL throughput big writes":
http://picpaste.com/pics/Clipboard01-0isEvjph.1420884805.png
There is a big dropoff in commit 43146.
It's cool to see a history of performance correlated with commits.
Now, if I dig into that commit, I see:
http://picpaste.com/pics/Clipboard02-HV47NdTC.1420884887.png
The commit seems to be a "doc only" commit. No actual code changes at all.
How should I interpret that?
Probably the test machine was changed, a new version of OpenSSL or pyOpenSSL, or something else?
One of those things. There is no infrastructure in place for identifying events which impact the performance testing infrastructure. The only performance testing environment is a very old mac mini still running Snow Leopard, which is probably the environment which we care the *least* about performance in, so it's not in great shape ;).
I'd say: the infrastructure aspects when doing performance tests do matter. To the degree that performance results are of very limited value at all, if the former aspects are not accounted for.
I don't think the results that we have presently are worth much at all. My point was mostly that there is some infrastructure which is halfway usable, and so you don't have to start from scratch. If you could take over this project (I am pretty sure at this point there is nobody to take it over *from*, exarkun did some work a long time ago and hasn't given it a second look in years) it would be highly appreciated!
(And if you care a lot about performance in a particular environment you could set it up in that environment and get attention for it :)).
You should also have a look at the existing benchmark suite and potentially look at maintaining / expanding that as well.
Thoughts?
-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Jan 10, 2015, at 3:59 PM, Dustin J. Mitchell <dustin@v.igoro.us> wrote:
As someone partially responsible for the infrastructure Mozilla uses to do its performance benchmarking, I can say that it's *really* hard. Getting live operating systems to sit still and behave is a mess, and then *keeping* them still over months and years (while attending to necessary security upgrades, hardware migrations, and so on) is even worse.
One of the smarter things we've figured out how to do is to "phase in" potentially disruptive changes so that we can either see that there's no impact, or estimate a correction factor for comparing results before and after the change.
In my unfortunately somewhat uninformed opinion, one thing that can really help is not to commit to long-term stability, but rather to just have a clearly documented log of operations performed on the monitoring cluster. Twisted has far less intense performance-analysis requirements than Mozilla, I should hope, and a lot less data to deal with, so just the ability to see events on the X axis could be enough to tell contributors what's going on with performance deltas. I should point out that the main reason we need a performance testing rig is not continuous performance monitoring over time, but rather, clear performance tracking of individual changes, ideally before they land. One of the things I'm unhappy about with speed center (and a big reason it's basically unmaintained) is that it's very hard to tell it to build a branch and to get a good picture of the aggregate effect of that branch on the benchmarks. -glyph
Probably the test machine was changed, a new version of OpenSSL or pyOpenSSL, or something else?
One of those things. There is no infrastructure in place for identifying events which impact the performance testing infrastructure. The only performance
Yes, this is an important point: track changes in infrastructure (everything that might have an influence, but is outside the tested code).
testing environment is a very old mac mini still running Snow Leopard, which
omg;)
I'd say: the infrastructure aspects when doing performance tests do matter. To the degree that performance results are of very limited value at all, if the former aspects are not accounted for.
I don't think the results that we have presently are worth much at all. My point was mostly that there is some infrastructure which is halfway usable, and so you don't have to start from scratch. If you could take over this
You mean taking over the code "as is" http://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/files or the task in general (Twisted benchmarking)?
project (I am pretty sure at this point there is nobody to take it over *from*,
We are currently developing performance test infrastructure for Crossbar.io - naturally, it is eating it's own dog food: the infrastructure is based on Crossbar.io and WAMP to orchestrate and wire up things in a distributed test setup. We could extend that to test at the Twisted(-only) level. Need to think about how that fits into "overall strategy", as the Crossbar.io perf. test stuff isn't open-source. The testing hardware above (mac, no real network) is insufficient for what I need. I'm thinking about buying and setting up 2 more boxes for Linux. Rgd. Codespeed (https://github.com/tobami/codespeed), which seems to be used by speedcenter.twistedmatrix.com: I have issues here as well. E.g. I need latency histograms, but this seems unsupported (benchmark results can only have avg/min/max/stddev). For me, this isn't "nice to have", but essential. Throughput is one thing. Constistent low latency a completely different. The latter is much much harder. But what is the "interface" between test cases from "twisted-benchmarks" to codespeed? This https://github.com/tobami/codespeed#saving-data seems to suggest, performance test results are HTTP/POSTed as JSON to codespeed. And codespeed is then only responsible for visualization and web hosting, right? I think we can find something better for that part.
(And if you care a lot about performance in a particular environment you could set it up in that environment and get attention for it :)).
Yes, in particular that very last one is a factor to justify efforts;) Anything like having a promo logo or similar - that would be an argument to invest time and material. I will seriously contemplate .. need to align with strategy/available time. We already host FreeBSD buildslaves for both Twisted and PyPy. That might be another synergy (hosting the latter on that same boxes).
You should also have a look at the existing benchmark suite and potentially look at maintaining / expanding that as well.
I will try to integrate some of this into our upcoming perf. infrastructure. /Tobias
Thoughts?
-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 08:49 pm, tobias.oberstein@tavendo.de wrote:
Rgd. Codespeed (https://github.com/tobami/codespeed), which seems to be used by speedcenter.twistedmatrix.com: I have issues here as well.
E.g. I need latency histograms, but this seems unsupported (benchmark results can only have avg/min/max/stddev). For me, this isn't "nice to have", but essential. Throughput is one thing. Constistent low latency a completely different. The latter is much much harder.
Codespeed is terrible. But this is not one of the ways in which it is terrible. Codespeed doesn't care if you label your measurement "latency". I think you've just noticed that what the existing benchmarks measure is mostly (entirely?) throughput. If you wanted to write a latency benchmark, I don't think anything's stopping you.
I think we can find something better for that part.
I suggest more fully understanding the capabilities and limitations of of codespeed before embarking on a project to replace it. Jean-Paul
On Jan 12, 2015, at 1:08 PM, exarkun@twistedmatrix.com wrote:
E.g. I need latency histograms, but this seems unsupported (benchmark results can only have avg/min/max/stddev). For me, this isn't "nice to have", but essential. Throughput is one thing. Constistent low latency a completely different. The latter is much much harder.
Codespeed is terrible. But this is not one of the ways in which it is terrible. Codespeed doesn't care if you label your measurement "latency". I think you've just noticed that what the existing benchmarks measure is mostly (entirely?) throughput. If you wanted to write a latency benchmark, I don't think anything's stopping you.
I believe Tobias was not saying "codespeed can't have a measurement called 'latency'" but rather "codespeed can't do histograms of measurements, which we need for measurement of latency and you don't need for measurement of throughput". Is that accurate? I don't know if there's a histogram feature hidden in the UI somewhere. It would be nice to at least try a little bit to contribute things (like a histogram feature) to codespeed before charging off in a completely different direction. -glyph
On 03:21 am, glyph@twistedmatrix.com wrote:
On Jan 12, 2015, at 1:08 PM, exarkun@twistedmatrix.com wrote:
E.g. I need latency histograms, but this seems unsupported (benchmark results can only have avg/min/max/stddev). For me, this isn't "nice to have", but essential. Throughput is one thing. Constistent low latency a completely different. The latter is much much harder.
Codespeed is terrible. But this is not one of the ways in which it is terrible. Codespeed doesn't care if you label your measurement "latency". I think you've just noticed that what the existing benchmarks measure is mostly (entirely?) throughput. If you wanted to write a latency benchmark, I don't think anything's stopping you.
I believe Tobias was not saying "codespeed can't have a measurement called 'latency'" but rather "codespeed can't do histograms of measurements, which we need for measurement of latency and you don't need for measurement of throughput". Is that accurate? I don't know if there's a histogram feature hidden in the UI somewhere.
It would be nice to at least try a little bit to contribute things (like a histogram feature) to codespeed before charging off in a completely different direction.
I wasn't suggesting it would be a good idea to contribute to codespeed. I think codespeed should be thrown in the trash. It was a great demonstration of a concept and we should thank it for that. However, as the basis of future development - no, it's an awful piece of unmaintained software. I was just trying to say that work towards replacing it should learn what it can from codespeed to try to avoid creating another piece of awful, ultimately unmaintained software. Jean-Paul
First, sorry for sluggish response time ..
E.g. I need latency histograms, but this seems unsupported (benchmark results can only have avg/min/max/stddev). For me, this isn't "nice to have", but essential. Throughput is one thing. Constistent low latency a completely different. The latter is much much harder.
Codespeed is terrible. But this is not one of the ways in which it is terrible. Codespeed doesn't care if you label your measurement "latency". I think you've just noticed that what the existing benchmarks measure is mostly (entirely?) throughput. If you wanted to >>write a latency benchmark, I don't think anything's stopping you.
I believe Tobias was not saying "codespeed can't have a measurement called 'latency'" but rather "codespeed can't do histograms of measurements, which we need for measurement of latency and you don't need for measurement of throughput". Is that accurate? I don't know if there's a histogram feature hidden in the UI somewhere.
Yes, exactly. That's what I meant. As an example of the kind of analysis and visualization I am after, please have a look here: http://www.brendangregg.com/HeatMaps/latency.html In particular, latency heatmaps are an incredible useful visualization. http://www.brendangregg.com/perf.html#HeatMaps
It would be nice to at least try a little bit to contribute things (like a histogram feature) to codespeed before charging off in a completely different direction.
It is Django and canvas for gfx. The former I have no know-how and no use/motivation for. Rendering HTML on the server isn't something we do anymore (WebSocket talking WAMP to Crossbar.io, anything else is just static HTML/JS/CSS/Images). For the latter, there is D3 (http://d3js.org/) which is awesome and vector graphics. For heatmaps, canvas _might_ be fine, but for virtually anything chart like, D3 has a lot to bring to the table. In general, I can follow the argument of "contributing instead of reinventing" and "paying back", but in this particular case, I can't justify sinking time into this (codespeed). Cheers, /Tobias
On Jan 12, 2015, at 12:49 PM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
Probably the test machine was changed, a new version of OpenSSL or pyOpenSSL, or something else?
One of those things. There is no infrastructure in place for identifying events which impact the performance testing infrastructure. The only performance
Yes, this is an important point: track changes in infrastructure (everything that might have an influence, but is outside the tested code).
testing environment is a very old mac mini still running Snow Leopard, which
omg;)
I'd say: the infrastructure aspects when doing performance tests do matter. To the degree that performance results are of very limited value at all, if the former aspects are not accounted for.
I don't think the results that we have presently are worth much at all. My point was mostly that there is some infrastructure which is halfway usable, and so you don't have to start from scratch. If you could take over this
You mean taking over the code "as is"
http://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/files
or the task in general (Twisted benchmarking)?
Both, I think. I'd really prefer it if you could start with the existing benchmarks, incrementally maintain them (perhaps beginning by porting to Github, we seem to be having pretty good success with travis) and probably eventually replace them wholesale, than replace them from the beginning.
project (I am pretty sure at this point there is nobody to take it over *from*,
We are currently developing performance test infrastructure for Crossbar.io - naturally, it is eating it's own dog food: the infrastructure is based on Crossbar.io and WAMP to orchestrate and wire up things in a distributed test setup.
We could extend that to test at the Twisted(-only) level. Need to think about how that fits into "overall strategy", as the Crossbar.io perf. test stuff isn't open-source.
The testing hardware above (mac, no real network) is insufficient for what I need. I'm thinking about buying and setting up 2 more boxes for Linux.
Keep in mind that a performance testing environment should be scalable. Others may have different environments they care about. Building your specific environment would be tremendously useful, but it would be even more useful to build it in a way that others can compare in their own hardware setups.
Rgd. Codespeed (https://github.com/tobami/codespeed), which seems to be used by speedcenter.twistedmatrix.com: I have issues here as well.
E.g. I need latency histograms, but this seems unsupported (benchmark results can only have avg/min/max/stddev). For me, this isn't "nice to have", but essential. Throughput is one thing. Constistent low latency a completely different. The latter is much much harder.
But what is the "interface" between test cases from "twisted-benchmarks" to codespeed?
Codespeed runs the benchmark, and they print out this stuff: https://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/view/head... POSTing them via JSON would be nicer, structured data is great.
This
https://github.com/tobami/codespeed#saving-data
seems to suggest, performance test results are HTTP/POSTed as JSON to codespeed.
And codespeed is then only responsible for visualization and web hosting, right?
I think we can find something better for that part.
(And if you care a lot about performance in a particular environment you could set it up in that environment and get attention for it :)).
Yes, in particular that very last one is a factor to justify efforts;) Anything like having a promo logo or similar - that would be an argument to invest time and material. I will seriously contemplate .. need to align with strategy/available time.
You should probably contact tsf@ with these concerns :).
We already host FreeBSD buildslaves for both Twisted and PyPy. That might be another synergy (hosting the latter on that same boxes).
You should also have a look at the existing benchmark suite and potentially look at maintaining / expanding that as well.
I will try to integrate some of this into our upcoming perf. infrastructure.
/Tobias
Thoughts?
-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 03:18 am, glyph@twistedmatrix.com wrote:
On Jan 12, 2015, at 12:49 PM, Tobias Oberstein But what is the "interface" between test cases from "twisted- benchmarks" to codespeed?
Codespeed runs the benchmark, and they print out this stuff: https://bazaar.launchpad.net/~twisted-dev/twisted- benchmarks/trunk/view/head:/benchlib.py#L12
POSTing them via JSON would be nicer, structured data is great.
Nothing parses that output. It's just for humans. The code you're looking for is: https://bazaar.launchpad.net/~twisted-dev/twisted- benchmarks/trunk/view/head:/speedcenter.py which POSTs structured data (though urlencoded, not JSON) to the codespeed server. Jean-Paul
On Jan 12, 2015, at 12:49 PM, Tobias Oberstein But what is the "interface" between test cases from "twisted- benchmarks" to codespeed?
Codespeed runs the benchmark, and they print out this stuff: https://bazaar.launchpad.net/~twisted-dev/twisted- benchmarks/trunk/view/head:/benchlib.py#L12
POSTing them via JSON would be nicer, structured data is great.
Nothing parses that output. It's just for humans.
The code you're looking for is:
https://bazaar.launchpad.net/~twisted-dev/twisted- benchmarks/trunk/view/head:/speedcenter.py
which POSTs structured data (though urlencoded, not JSON) to the codespeed server.
I see. And codespeed parsest that, stores it in a database and produces graphics? It seems, reporting the results via a WAMP RPC to a backend should be quite easy to add in speedcenter.py How is Twisted speedcenter orchestrated / triggered? I mean, a new commit to Twisted repo will trigger rerunning all speed tests? If so, how does that work? /Tobias
On 05:00 pm, tobias.oberstein@tavendo.de wrote:
On Jan 12, 2015, at 12:49 PM, Tobias Oberstein But what is the "interface" between test cases from "twisted- benchmarks" to codespeed?
Codespeed runs the benchmark, and they print out this stuff: https://bazaar.launchpad.net/~twisted-dev/twisted- benchmarks/trunk/view/head:/benchlib.py#L12
POSTing them via JSON would be nicer, structured data is great.
Nothing parses that output. It's just for humans.
The code you're looking for is:
https://bazaar.launchpad.net/~twisted-dev/twisted- benchmarks/trunk/view/head:/speedcenter.py
which POSTs structured data (though urlencoded, not JSON) to the codespeed server.
I see. And codespeed parsest that, stores it in a database and produces graphics?
It seems, reporting the results via a WAMP RPC to a backend should be quite easy to add in speedcenter.py
WAMP? I don't think so. I understand you work in that space a lot but I think you might want to take a step back. The extra complexity of WebSockets is pointless for this project. Just HTTP POST some JSON. This can be really simple. Your time is almost certainly better spent elsewhere. Jean-Paul
How is Twisted speedcenter orchestrated / triggered?
I mean, a new commit to Twisted repo will trigger rerunning all speed tests? If so, how does that work?
/Tobias
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
which POSTs structured data (though urlencoded, not JSON) to the codespeed server.
I see. And codespeed parsest that, stores it in a database and produces graphics?
It seems, reporting the results via a WAMP RPC to a backend should be quite easy to add in speedcenter.py
WAMP? I don't think so. I understand you work in that space a lot but I think you might want to take a step back. The extra complexity of WebSockets is pointless for this project. Just HTTP POST some JSON.
All the extra complexity is hidden using WAMP and AutobahnJS.
This can be really simple. Your time is almost certainly better spent elsewhere.
No. Sinking time into Django/codespeed is lost time for me .. /Tobias
On 12:15 pm, tobias.oberstein@tavendo.de wrote:
which POSTs structured data (though urlencoded, not JSON) to the codespeed server.
I see. And codespeed parsest that, stores it in a database and produces graphics?
It seems, reporting the results via a WAMP RPC to a backend should be quite easy to add in speedcenter.py
WAMP? I don't think so. I understand you work in that space a lot but I think you might want to take a step back. The extra complexity of WebSockets is pointless for this project. Just HTTP POST some JSON.
All the extra complexity is hidden using WAMP and AutobahnJS.
Oh well. I'm not going to spend much more effort to convince you that this is a bad idea. Maybe someone else will.
This can be really simple. Your time is almost certainly better spent elsewhere.
No. Sinking time into Django/codespeed is lost time for me ..
If you read my emails, you'll see that I explicitly recommended *against* contributing to codespeed. If you're not going to read what I write then I'll stop writing, I guess. Jean-Paul
It seems, reporting the results via a WAMP RPC to a backend should be quite easy to add in speedcenter.py
WAMP? I don't think so. I understand you work in that space a lot but I think you might want to take a step back. The extra complexity of WebSockets is pointless for this project. Just HTTP POST some JSON.
All the extra complexity is hidden using WAMP and AutobahnJS.
Oh well. I'm not going to spend much more effort to convince you that this is a bad idea. Maybe someone else will.
That's ok for me. I guess we will create something in-house that fits what we need. /Tobias
On Jan 18, 2015, at 4:46 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
All the extra complexity is hidden using WAMP and AutobahnJS.
Oh well. I'm not going to spend much more effort to convince you that this is a bad idea. Maybe someone else will.
That's ok for me. I guess we will create something in-house that fits what we need.
I don't think I am following this conversation. What do you mean by "in-house"? -glyph
All the extra complexity is hidden using WAMP and AutobahnJS.
Oh well. I'm not going to spend much more effort to convince you that this is a bad idea. Maybe someone else will.
That's ok for me. I guess we will create something in-house that fits what we need.
I don't think I am following this conversation. What do you mean by "in-house"?
From how I understood Jean-Paul, he thinks using WAMP to hook up test infrastructure components (like load probes, test orchestrators and test database backend) is a bad idea, and HTTP/REST should be used instead.
I have a different view on this for technical reasons - but, admitted, also because I am affiliated with WAMP and have zero time to invest in stuff that I am not interested in / have no need for - that is HTTP/REST, and the server bits to make that fly. It'll be _more_ work on HTTP/REST, and less capable. Anyway. I think WAMP is a great choice to hook up components of a distributed test system - which is what I am after (e.g. I want to orchestrate 10 TCP load probes running on different machines, stressing a target TCP echo server). This difference in opinion might be because we have different _scopes/requirements_ to start from. Or not. I don't know. So I thought, for the time being, it might be better if we (Tavendo) develop something for internal use / privately ("in-house"), and probably come again / show something when we actually have it running. === Regarding the "charting sub-issue": I came across https://plot.ly/ This is kinda cool and very quick to get started: http://picpaste.com/pics/Clipboard01-i4Karh0D.1421692351.png It does histograms and tons of fancy stuff and hosts everything for free. Cheers, /Tobias
On Jan 19, 2015, at 10:44 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
All the extra complexity is hidden using WAMP and AutobahnJS.
Oh well. I'm not going to spend much more effort to convince you that this is a bad idea. Maybe someone else will.
That's ok for me. I guess we will create something in-house that fits what we need.
I don't think I am following this conversation. What do you mean by "in-house"?
From how I understood Jean-Paul, he thinks using WAMP to hook up test infrastructure components (like load probes, test orchestrators and test database backend) is a bad idea, and HTTP/REST should be used instead.
I have a different view on this for technical reasons - but, admitted, also because I am affiliated with WAMP and have zero time to invest in stuff that I am not interested in / have no need for - that is HTTP/REST, and the server bits to make that fly. It'll be _more_ work on HTTP/REST, and less capable.
Anyway. I think WAMP is a great choice to hook up components of a distributed test system - which is what I am after (e.g. I want to orchestrate 10 TCP load probes running on different machines, stressing a target TCP echo server).
This difference in opinion might be because we have different _scopes/requirements_ to start from. Or not. I don't know.
This is Jean-Paul's personal opinion on the implementation choices involved. This is only really relevant if JP is going to be working on the performance testing framework directly or needs to interact with the WAMP bits directly. If you're the one doing the work, you get to make the technology choices and, within certain reasonable constraints like "no PHP", I'm sure that we would be happy to run whatever performance testing system you come up with if it satisfies our requirements and has a reasonable API.
So I thought, for the time being, it might be better if we (Tavendo) develop something for internal use / privately ("in-house"), and probably come again / show something when we actually have it running.
Well, we'd want to see something that actually works before officially standardizing Twisted on it anyway, but please feel free to share it early. I guarantee you that we will not reject something simply for using WAMP :-).
Regarding the "charting sub-issue":
I came across https://plot.ly/
This is kinda cool and very quick to get started:
http://picpaste.com/pics/Clipboard01-i4Karh0D.1421692351.png
It does histograms and tons of fancy stuff and hosts everything for free.
This is hosted-service only? Not that that's a dealbreaker, I guess, we are pretty sure we're going to move everything to Github anyway... -glyph
Anyway. I think WAMP is a great choice to hook up components of a distributed test system - which is what I am after (e.g. I want to orchestrate 10 TCP load probes running on different machines, stressing a target TCP echo server).
This difference in opinion might be because we have different _scopes/requirements_ to start from. Or not. I don't know.
This is Jean-Paul's personal opinion on the implementation choices involved. This is only really relevant if JP is going to be working on the performance testing framework directly or needs to interact with the WAMP bits directly.
Ok. Yes, WAMP is an implementation detail here. @Jean-Paul: I don't want to leave a bad impression here: you have been helping me so often, and I learned so much from your hints, code, suggestions and opinions. I do value that very much, and I have read everything you wrote. I swear;)
If you're the one doing the work, you get to make the technology choices and, within certain reasonable constraints like "no PHP", I'm sure that we
omg, no;)
would be happy to run whatever performance testing system you come up with if it satisfies our requirements and has a reasonable API.
Alright. We want to contribute, and if that is welcome, awesome! "our requirements": This is very important, as we need to fold in Twisted project specific requirements into the list we have. If this is to be used not only in-house, but be of real use/value to Twisted as a project. I think there will be a big overlap, but I want to make sure we have it consolidated _before_ starting with code. Can we collect those requirements from a Twisted project perspective? I have skimmed through this thread and dumped it to: https://github.com/oberstet/scratchbox/blob/master/python/twisted/speed/requ... E.g. 10. + 11. + 12. + 15. from Glyph's comments. If anyone has more to add to this list, please reply, I'll append it, and clean it up /structure it in the end.
So I thought, for the time being, it might be better if we (Tavendo) develop something for internal use / privately ("in-house"), and probably come again / show something when we actually have it running.
Well, we'd want to see something that actually works before officially standardizing Twisted on it anyway, but please feel free to share it early. I
Yes. Will do. I hate _talking_, instead of showing.
guarantee you that we will not reject something simply for using WAMP :-).
=)
Regarding the "charting sub-issue":
I came across https://plot.ly/
This is kinda cool and very quick to get started:
http://picpaste.com/pics/Clipboard01-i4Karh0D.1421692351.png
It does histograms and tons of fancy stuff and hosts everything for free.
This is hosted-service only? Not that that's a dealbreaker, I guess, we are
Yes, this is hosted only. Which is part of why it's attractive. But: we don't want to be locked into this too much. Means: the test result _data_ should stay in a database open and accessible to the Twisted project. We could host that database. Or it could reside on TSF. The data is where the value is and the Twisted project needs to own/control that. The data from that DB is then just read and requests to above service issued to produce graphs. But if we at some point want to kick that service, and generate graphs on our own or whatever, that should definitely be possible. /Tobias
pretty sure we're going to move everything to Github anyway...
-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 1 Jan, 10:21 am, tobias.oberstein@tavendo.de wrote:
Hi,
I am doing network performance tests using netperf on a trivial Twisted TCP echo server (code at the end).
One of the tests that netperf offers is throughput, and I am running into an issue with this. ? [snip]
Now, my suspicion is that Twisted is reading off the TCP stack from the kernel and buffering in userspace faster than the echo server is pushing out stuff to the TCP stack into the kernel. Hence, no TCP backpressure results, netperf happily sends more and more, and the memory of the Twisted process runs away.
What you said here about "buffering in userspace" is ambiguous. It's not clear if you meant data is being buffered in userspace on the read side before your protocol gets a chance to handle it or if you meant that data being written to the transport by the protocol is being buffered in userspace. The former doesn't happen. There are no no userspace read buffers in Twisted between the transport and the protocol. Bytes are read from the socket and then passed to dataReceived. The latter would be addressed by using producer/consumer APIs as Glyph suggested. Jean-Paul
Now, my suspicion is that Twisted is reading off the TCP stack from the kernel and buffering in userspace faster than the echo server is pushing out stuff to the TCP stack into the kernel. Hence, no TCP backpressure results, netperf happily sends more and more, and the memory of the Twisted process runs away.
What you said here about "buffering in userspace" is ambiguous. It's not clear if you meant data is being buffered in userspace on the read side before your protocol gets a chance to handle it
Yes. That's what I meant. Buffering in userspace inside Twisted, and before data is handled by the app in dataReceived.
.. or if you meant that data being written to the transport by the protocol is being buffered in userspace.
Nope, I didn't meant that. That's the sending side.
The former doesn't happen. There are no no userspace read buffers in Twisted between the transport and the protocol. Bytes are read from the
Ok.
socket and then passed to dataReceived.
The latter would be addressed by using producer/consumer APIs as Glyph suggested.
Mmh. Fact is: somehow memory runs away. How do I track down _where_ exactly the mem is spent? Probably that leads to the "why" then .. /Tobias
On 12:06 pm, tobias.oberstein@tavendo.de wrote:
Fact is: somehow memory runs away.
How do I track down _where_ exactly the mem is spent? Probably that leads to the "why" then ..
There are memory profiling tools for Python. For example, memory_profiler: https://pypi.python.org/pypi/memory_profiler As far as I know, there are none that are specific to Twisted. Jean-Paul
On Jan 2, 2015, at 4:06 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
Fact is: somehow memory runs away.
How do I track down _where_ exactly the mem is spent? Probably that leads to the "why" then ..
The first place to look - since sometimes looking in a specific place makes memory profilers easier to use - would be twisted.internet.tcp.Server._tempDataBuffer. The fact that it's extremely slow when you turn on consumer/producer logic in this way makes sense to me. twisted.internet.abstract.FileDescriptor.bufferSize is hard-coded to 65,536 bytes; every time the write side outpaces the read side by that buffer size, it will result in a call to (in your case) twisted.internet.kqreactor.KQueueReactor._updateRegistration, which makes the kcontrol syscall. One thing you might try is to run with the environment variable PYPYLOG=jit-summary:- set. This will give you a bunch of statistics about what the JIT did at the end of the run, on standard out; compare the fast (and run out of memory) to the slow (and work right) run to see what the differences are. -g
participants (5)
-
Dustin J. Mitchell -
exarkun＠twistedmatrix.com -
Glyph -
Glyph Lefkowitz
-
Tobias Oberstein