Ok, I've pimped up the test

https://github.com/oberstet/scratchbox/blob/master/python/twisted/masterchild/streaming_master.py

https://github.com/oberstet/scratchbox/blob/master/python/twisted/masterchild/streaming_child.py

and running on Ubuntu LTS 12.04, 64 Bit, Core i7 3.4GHz, epoll reactor, I was able to get it up to

Master => Child : 450MB/s

[94%/100% CPU load)

Master <=> Child (full-duplex echo): 200MB/s

[95%/100% CPU load)

Nifty.

Now, it might be that I can push even higher by adjusting the pipe buffer sizes (there is a Linux fcntl for that .. F_SETPIPE_SZ).

Q: How do I get at the FDs of the underlying pipes to the spawned process in the parent?

reactor.spawnProcess returns an object implementing IProcessTransport

However, that does not seem to provide access to the underlying pipes (and their FDs in the parent)

Looking at

http://twistedmatrix.com/trac/browser/tags/releases/twisted-13.1.0/twisted/internet/process.py

I can get it, but not via official API, right?

===

Windows: It does not work at all. Tested "select" and "iocp" reactors. It just hangs doing nothing.

Von: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] Im Auftrag von Tobias Oberstein
Gesendet: Mittwoch, 23. Oktober 2013 00:58
An: Twisted discussion
Betreff: Re: [Twisted-Python] Flow-control for Pipes / Unix Domains Sockets?

Hi Itamar,

once again Twisted surprises me: it just works!

I have tested a master/child combo over pipe (both Twisted) with a push producer on the sending leg.

It can push around 70MB/s raw binary (16k write size) to the child. Load is evenly on 2 CPU cores each at 80% and stays roughly constant. Memory is flat for both.

This is on some years old Macbook (kqueue reactor .. a little faster than select) - tomorrow I repeat on a beefy system/OS.

I will do more systematic throughput and also latency measurements .. but this is already not bad at all.

Moreso: I really am curious now how that works under the hood, since the pausing/resuming seems to be upper/lower watermark controlled .. it'll resume the producer before the consumer starves. As it should be;)

Background: this is part of experiments in preparation for a multi-core capable Autobahn based message broker ..

Takeaway: Next time I don't waste time on the internet reading half-baked posts, but just hack away;)

/Tobias

PS: The following is actually slower than above master/slave pair .. which I also didn't expect:

tobias-obersteins-macbook-pro-2:masterchild oberstet$ time dd if=/dev/zero bs=1k count=1000000 | wc -c
1000000+0 records in
1000000+0 records out
1024000000 bytes transferred in 17.454178 secs (58667902 bytes/sec)
 1024000000

real    0m17.460s
user    0m18.389s
sys    0m7.729s



On 22.10.13 02:00, "Itamar Turner-Trauring" <itamar@itamarst.org> wrote:

On 10/21/2013 04:48 PM, Tobias Oberstein wrote:

Flow-control for Pipes / Unix Domains Sockets? Hi,

 with TCP (either remote or loopback) I can have flow-control using the producer-consumer machinery that Twisted provides.

 Is that (flow-control / producer-consumer) also available (and practically usable/recommended) with:

 a) Unix Domain sockets
 b) Pipes (spawnProcess)

 I would expect transport.registerProducer to work with both.