One is a language. The other is an event loop. I'm not sure how we are supposed to compare the two. If he would've said E and Twisted, perhaps it'd be a more interesting comparison :-) Also, what Steve said jumped right into my eye as well. Don't get me wrong. I like Erlang -- it's functional, it's robust, it's very easy to make your programs execute in parallel. RabbitMQ and Scalaris are two examples of *excellent* Erlang software. I don't really think Erlang was a bad choice -- I just think they don't know enough about Twisted to judge :-) Laurens

It became very painful once we started getting CPU bound on 32 core CPU machines. We had to run multiple instances of our Twisted server implementation to saturate the machine. I have since "ported" the same application to Erlang/OTP. It is about 1/5 the amount of code. And it scales 1:1 horizontally (adding more CPUs local or remote ) and vertically (adding FASTER cpus ) with no code changes.

The broad statement that Twisted doesn't possess the capabilities of Erlang/OTP is pretty much accurate. Each solves a different problem. Granted Erlang/OTP is definately a super-set of what Twisted does.

Scaling horizontally, i.e. parallelization of twisted servers over multiple cores is a difficult problem. We run this on 8 and 16 core CPU machines, one process per core, and I use UDP multicasts on internal LAN for messaging between the servers (both inside the same physical machine, or to remote nodes). This is of course less than ideal. It would be very nice if Twisted had some inherent capabilities for scaling horizontally.

vertically (adding FASTER cpus ) with no code changes.

When you ported, were you able to serve to less or more clients on the same speed CPU core? The biggest advantage of Twisted is a shorter development time, and the availability of Python libraries. But I wonder if running something of the scale of Facebook chat would be feasible at all, or if there are people on this list that have run many parallelized twisted servers that saturated many multi-core machines? In other words, are there practical examples of large-scale Twisted deployment?

I think we are accounting way too much time to the unfounded unaccountable random blathering of a troll :-)

As you can see from the %CPU column, I have my reasons for concern ;) This is current copy and paste from a node with 2x quad core xeon L5420 @ 2.50GHz - 1 twistd process per core. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24448 nobody 20 0 991m 607m 2452 R 99 4.3 24764:37 twistd 28553 nobody 20 0 909m 453m 2412 R 95 3.2 1346:51 twistd 24640 nobody 20 0 1092m 676m 2452 R 93 4.8 32750:14 twistd 29900 nobody 20 0 802m 362m 2412 R 93 2.6 1180:53 twistd 24279 nobody 20 0 761m 277m 2424 R 42 2.0 13891:58 twistd 32422 nobody 20 0 1381m 962m 2372 R 10 6.9 13241:54 twistd 24210 nobody 20 0 599m 236m 2396 S 4 1.7 9063:29 twistd 29862 nobody 20 0 323m 14m 2384 S 2 0.1 71:53.50 twistd

-----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python- bounces@twistedmatrix.com] On Behalf Of Laurens Van Houtven Sent: Tuesday, September 29, 2009 12:28 PM To: Twisted general discussion Subject: Re: [Twisted-Python] I found an interisting comment about Twisted vs. Erlang

I think we are accounting way too much time to the unfounded unaccountable random blathering of a troll :-)

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

What was your application doing at the time? Was it idle, heavily loaded, somewhere in the middle? What is the QoS for each client connecting to your service? Are requests being handled in a timely fashion? Do the users of your service perceive a performance problem or do you just not like seeing big numbers in the %CPU column? Are performance problems really in Twisted, or are they because of suboptimal decisions in application logic/factoring?

Posting a list like that and suggesting that something is wrong (or right, even) with Twisted isn't really productive: there's nowhere obvious to go from here to make it better.

My first glance at those numbers made me think Twisted is awesome because it lets you write an application that can actually make full use of the CPU power you have, but I'm guessing that's not what you're getting at. :)

I am actually not suggesting the performance is bad, it's quite decent. Each process is handling about 30,000 clients, using epoll reactor, and is heavily loaded as you can see from S column, which shows "R". The app broadcasts the messages sent from a client to groups of other clients, and has some other logic. I posted this twice by mistake, but as I wrote in my second post, the debugging is pretty hard, esp memory leaks... Also, performance debugging- say how much time is spent in each function? FB engineers seems to say that debugging of E is transparent, but I do not have the first hand knowledge.

-----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python- bounces@twistedmatrix.com] On Behalf Of Jamu Kakar Sent: Tuesday, September 29, 2009 12:48 PM To: Twisted general discussion Subject: Re: [Twisted-Python] I found an interisting comment about Twisted vs. Erlang

Hi Alec,

On Tue, Sep 29, 2009 at 12:36 PM, Alec Matusis wrote:

...

As you can see from the %CPU column, I have my reasons for concern ;) This is current copy and paste from a node with 2x quad core xeon L5420 @ 2.50GHz - 1 twistd process per core.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

24448 nobody 20 0 991m 607m 2452 R 99 4.3 24764:37 twistd

28553 nobody 20 0 909m 453m 2412 R 95 3.2 1346:51 twistd

24640 nobody 20 0 1092m 676m 2452 R 93 4.8 32750:14 twistd

29900 nobody 20 0 802m 362m 2412 R 93 2.6 1180:53 twistd

24279 nobody 20 0 761m 277m 2424 R 42 2.0 13891:58 twistd

32422 nobody 20 0 1381m 962m 2372 R 10 6.9 13241:54 twistd

24210 nobody 20 0 599m 236m 2396 S 4 1.7 9063:29 twistd

29862 nobody 20 0 323m 14m 2384 S 2 0.1 71:53.50 twistd

What was your application doing at the time? Was it idle, heavily loaded, somewhere in the middle? What is the QoS for each client connecting to your service? Are requests being handled in a timely fashion? Do the users of your service perceive a performance problem or do you just not like seeing big numbers in the %CPU column? Are performance problems really in Twisted, or are they because of suboptimal decisions in application logic/factoring?

Posting a list like that and suggesting that something is wrong (or right, even) with Twisted isn't really productive: there's nowhere obvious to go from here to make it better.

My first glance at those numbers made me think Twisted is awesome because it lets you write an application that can actually make full use of the CPU power you have, but I'm guessing that's not what you're getting at. :)

Thanks, J.

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python