On 10-12-21 02:24 PM, exarkun@twistedmatrix.com wrote:
On 07:02 pm, psanchez@fosstel.com wrote:
On 10-12-21 12:47 PM, exarkun@twistedmatrix.com wrote:
On 04:43 pm, psanchez@fosstel.com wrote:
OK, I guess I'm being slow :-( Here's another version, same results.
import os from twisted.internet import reactor from twisted.web.server import Site from twisted.web.resource import Resource
CHUNK_SIZE = 32*1024 data = os.urandom(10*1024*1024) chunks = []
def make_chunks(): s = 0 for chunk in iter(lambda: data[s:s+CHUNK_SIZE], ''): chunks.append(chunk) s = s + CHUNK_SIZE
class TestPage(Resource): isLeaf = True
def render_GET(self, request): for chunk in chunks: request.write(chunk)
make_chunks() root = Resource() root.putChild('test', TestPage()) reactor.listenTCP(8880, Site(root)) reactor.run()
This version has flat memory usage on my system - 33MB all the way through. Jean-Paul
Did you try this running the clients on a different machine than the one running the server? I find that if I run both the server and the clients (using httperf) on the same machine things behave differently. I can only see the memory consumption problem consistently when running the clients on a separate machine (which would be my real use case).
BTW, I'm running these test on Ubuntu 10.04, Python 2.6.5, twisted 10.0.0-2ubuntu2.
Argh. I just ran it over loopback, so I wasn't accurately testing the case you described. James Knight just reminded me that the transport will still find a way to copy data in this case, so you're right that this still isn't a working solution. Instead, you have to go all the way to producers/consumers, and only write more data to the transport buffer when it has finished dealing with what you previously gave it.
Request implements IConsumer, which means you can pass an IProducer provider to its registerProducer method. When the send buffer gets close to empty, the IProducer's resumeProducing method will be called and you can write some more bytes to the Request.
There are some more complications and subtleties in the IProducer/IConsumer interfaces, which you can read about at http://twistedmatrix.com/documents/current/core/howto/producers.html.
Basically, you'll end up with something like:
from twisted.python.log import err from twisted.protocols.basic import FileSender from twisted.web.server import NOT_DONE_YET
def render_GET(self, request): producer = FileSender() d = producer.beginFileTransfer(StringIO(chunk), request) def finished(ignored): request.finish() d.addErrback(err, "Streaming data to client failed") d.addCallback(finished) return NOT_DONE_YET
Sorry for the earlier mis-information.
Jean-Paul
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Thanks again Jean-Paul, This one will take me some time to digest and prototype. I'll give it a whirl to try to implement a solution and I'll post back my results. -- Pedro