[Twisted-Python] Win32 IOCPReactor?
I'm new to Twisted. Perusing the documentation, I came across IOCPReactor. It's marked as 'extremely experimental', and it appears it only provides support for TCP events. It seems the last check in to anything IOCPReactor-related was back in 2005. Is it dead? Is anyone using it? Is anyone planning to pick development back up? Were there specific issues with IOCP that impeded development, or just a general lack of people with time, inclination, or Win32-fu? Assuming the IOCPReactor's not feasible for high load production deployment, can anyone comment on how the default select()-based reactor for Win32 fares under high load? Trent.
IOCPReactor isn't production-ready for most people's definitions of "production". And yes, it's been stagnant for a long time.. AFAIK it's mostly just lack of attention. It's a lot of work, and so far nobody has been able to commit to finishing it. As for the default select-based reactor it should work fine. There aren't any outstanding performance concerns that I am aware of. You will have to test for yourself if it meets your requirements. Hope that helps, -- Eric Mangold Twisted/Win32 Maintainer
I'm new to Twisted. Perusing the documentation, I came across IOCPReactor. It's marked as 'extremely experimental', and it appears it only provides support for TCP events.
It seems the last check in to anything IOCPReactor-related was back in 2005. Is it dead? Is anyone using it? Is anyone planning to pick development back up? Were there specific issues with IOCP that impeded development, or just a general lack of people with time, inclination, or Win32-fu?
Assuming the IOCPReactor's not feasible for high load production deployment, can anyone comment on how the default select()-based reactor for Win32 fares under high load?
Trent.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
We use the iocp reactor on our Windows servers for our MMORPG Minions of Mirth. The select reactor on Windows is *extremely* slow and bombs when you hit between 32-64 connections. ------------------------------------ -Josh Ritter President Prairie Games, Inc http://www.prairiegames.com ----- Original Message ----- From: "Eric Mangold" <teratorn@twistedmatrix.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Friday, May 18, 2007 12:58 PM Subject: Re: [Twisted-Python] Win32 IOCPReactor?
IOCPReactor isn't production-ready for most people's definitions of "production".
And yes, it's been stagnant for a long time.. AFAIK it's mostly just lack of attention. It's a lot of work, and so far nobody has been able to commit to finishing it.
As for the default select-based reactor it should work fine. There aren't any outstanding performance concerns that I am aware of. You will have to test for yourself if it meets your requirements.
Hope that helps, -- Eric Mangold Twisted/Win32 Maintainer
I'm new to Twisted. Perusing the documentation, I came across IOCPReactor. It's marked as 'extremely experimental', and it appears it only provides support for TCP events.
It seems the last check in to anything IOCPReactor-related was back in 2005. Is it dead? Is anyone using it? Is anyone planning to pick development back up? Were there specific issues with IOCP that impeded development, or just a general lack of people with time, inclination, or Win32-fu?
Assuming the IOCPReactor's not feasible for high load production deployment, can anyone comment on how the default select()-based reactor for Win32 fares under high load?
Trent.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
We've never used the event reactor. It has been awhile since using the select reactor in Windows. At the time, we wouldn't of had nearly 512 simultaneous connections... It would bomb well before 512 connections. It is possible that it would allow up to 512 connections to be made (not simultaneous) before the issue raised itself. Though, once the problem happened no more connections were possible, regardless of some dropping, etc The performance was also terrible. -JR ----- Original Message ----- From: "Itamar Shtull-Trauring" <itamar@itamarst.org> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Saturday, May 19, 2007 4:11 PM Subject: Re: [Twisted-Python] Win32 IOCPReactor?
On Sat, 2007-05-19 at 15:59 -0500, Josh Ritter wrote:
The select reactor on Windows is *extremely* slow and bombs when you hit between 32-64 connections.
The select reactor can go up to 512 connections... are you sure you're not thinking of the win32eventreactor?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Next question - who runs servers on *Windows*? :)
I am new to Twisted, and I am considering porting my application to Twisted, (for performance and scalability reasons, and to take advantage of Comet). *All* our customers are running Windows. This discussion thread sounds scary! What works and what does not work in IOCP reactor? Will I be able to run PB and web server using IOCP reactor? Is select reactor really broken on Windows? Should I be looking at other server products instead? Please, if someone has experience using Twisted for distributed applications with web interface on Windows, could you guide a newbie? Thank you very much! Volodymyr Orlenko
On Sun, 2007-05-20 at 16:06 -0700, Volodymyr Orlenko wrote:
I am new to Twisted, and I am considering porting my application to Twisted, (for performance and scalability reasons, and to take advantage of Comet). *All* our customers are running Windows. This discussion thread sounds scary! What works and what does not work in IOCP reactor?
Basic TCP should work. There's a rewrite of the IOCP reactor in a branch which is likely better.
Will I be able to run PB and web server using IOCP reactor?
Probably; some people at least seem to be able to use it.
Is select reactor really broken on Windows?
All tests pass in the buildbot: http://twistedmatrix.com/buildbot/ and we have no major bug reports against it as far as I know. I would trust it more than the IOCP reactor (as long as you're OK with only 512 connections.)
Should I be looking at other server products instead?
Well... it's probably hard to find something as powerful and extensive as Twisted. So if your choice is building all this infrastructure on some other framework, or just helping improve Twisted's IOCP support (if you find it necessary), why not do the later? :)
Itamar Shtull-Trauring wrote:
Well... it's probably hard to find something as powerful and extensive as Twisted. So if your choice is building all this infrastructure on some other framework, or just helping improve Twisted's IOCP support (if you find it necessary), why not do the later? :)
I guess you are right. I've worked with IOCP back in my C++ days, and I really liked the performance. Maybe what I should do is put together a simple application and do some benchmarking, to compare performance of IOCP reactor and select reactor on Windows (and of my existing application, which uses TurboGears/CherryPy and Python's built-in XMLRPC server). I'll keep you posted. Is this <http://twistedmatrix.com/trac/browser/branches/iocpreactor-1760-3> the branch you mentioned?
At 5:55 PM -0400 5/19/07, Itamar Shtull-Trauring wrote:
Next question - who runs servers on *Windows*? :)
We do [Raises hand]. Twisted built with Py2Exe and running as an NT service. Have delivered several different applications in this configuration for clients. --Ray -- Raymond Cote Appropriate Solutions, Inc. PO Box 458 ~ Peterborough, NH 03458-0458 Phone: 603.924.6079 ~ Fax: 603.924.8668 rgacote(at)AppropriateSolutions.com www.AppropriateSolutions.com
On Mon, 2007-05-21 at 08:13 -0400, Ray Cote wrote:
At 5:55 PM -0400 5/19/07, Itamar Shtull-Trauring wrote:
Next question - who runs servers on *Windows*? :)
We do [Raises hand]. Twisted built with Py2Exe and running as an NT service. Have delivered several different applications in this configuration for clients. --Ray
Oh, I certainly understand running on Windows when clients pay you to do so. I was just surprised people do so on internal servers when they do have a choice. Anyway, since this is bringing users out into the open - I'm sure we'd all like to hear what Twisted is being used for in different companies, and under which platforms. Do tell! My company uses Twisted for a variety of uses, including reliable multicast data distribution, messaging, monitoring and managing complex processes, in the airline industry. (http://itasoftware.com) How about you all?
Basic TCP should work. There's a rewrite of the IOCP reactor in a branch which is likely better.
I believe Volodymyr already pointed it out, but can I just confirm you're referring to this branch: svn://svn.twistedmatrix.com/svn/Twisted/branches/iocpreactor-1760-3/ Can anyone comment on the status of these two branches? iocp-producer-1657/ iocp_syshandle-2021/
How about you all?
High-throughput in-house network management (now libsnmp-based, formerly bespoke code). Lots of little glue services e.g. tail a log using "tail -f" and a process protocol, extract stats, answer SNMP queries for those stats via a unix socket passthrough from the net-snmp snmpd daemon. A couple of web servers; including one that parses a FreeRadius SQL accounting database and issues either a PB-based, SNMP-based or Radius packet-of-death based disconnect for kicking newly-banned users. A couple of misc daemons; e.g. one on our mail relays that will run an MD5 of a file in the filesystem on request from a Unix socket (in a pyrex GIL-release thread using the OpenSSL functions) so we can block all instances of a specific attachment by MD5 at SMTP-DATA-time. A very clever app that a colleague wrote that uses the QUEUE target of iptables to get TCP port 443 SYN packets from a walled garden network; then connects to that SSL service and gets the certificate; then permits or allows the original SYN based on a name-based wildcard of the certificate CN. Loads of other misc stuff mostly in this vein. It's a great.
On Mon, 2007-05-21 at 16:02 +0200, Trent Nelson wrote:
Basic TCP should work. There's a rewrite of the IOCP reactor in a branch which is likely better.
I believe Volodymyr already pointed it out, but can I just confirm you're referring to this branch:
svn://svn.twistedmatrix.com/svn/Twisted/branches/iocpreactor-1760-3/
Yes.
Can anyone comment on the status of these two branches?
iocp-producer-1657/ iocp_syshandle-2021/
I think these involve bugs in the current IOCP code.
At 8:34 AM -0400 5/21/07, Itamar Shtull-Trauring wrote:
Anyway, since this is bringing users out into the open - I'm sure we'd all like to hear what Twisted is being used for in different companies, and under which platforms. Do tell!
Windows: - Web services front end for an Oracle database in the food/nutrition management industry. <http://www.AppropriateSolutions.com> - Gateway to various payment processing services (still in development). <http://www.AuricSystems.com> -- Raymond Cote Appropriate Solutions, Inc. PO Box 458 ~ Peterborough, NH 03458-0458 Phone: 603.924.6079 ~ Fax: 603.924.8668 rgacote(at)AppropriateSolutions.com www.AppropriateSolutions.com
Allmydata, Inc. uses Twisted in its current proprietary consumer backup product [1] as well as in its free-software distributed storage grid project [2]. Regards, Zooko [1] http://allmydata.com/ [2] http://allmydata.org/
We run a successful independent MMORPG called Minions of Mirth. We use Twisted for our master, character, GM, world servers, and client. The game has more than 40,000 registered players. We're also working on licensing the technology. There's more information on this available here: http://www.mmoworkshop.com ----- Original Message ----- From: "Itamar Shtull-Trauring" <itamar@itamarst.org> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Monday, May 21, 2007 7:34 AM Subject: Tell us how you're using Twisted (was Re: [Twisted-Python] Win32IOCPReactor?)
On Mon, 2007-05-21 at 08:13 -0400, Ray Cote wrote:
At 5:55 PM -0400 5/19/07, Itamar Shtull-Trauring wrote:
Next question - who runs servers on *Windows*? :)
We do [Raises hand]. Twisted built with Py2Exe and running as an NT service. Have delivered several different applications in this configuration for clients. --Ray
Oh, I certainly understand running on Windows when clients pay you to do so. I was just surprised people do so on internal servers when they do have a choice.
Anyway, since this is bringing users out into the open - I'm sure we'd all like to hear what Twisted is being used for in different companies, and under which platforms. Do tell!
My company uses Twisted for a variety of uses, including reliable multicast data distribution, messaging, monitoring and managing complex processes, in the airline industry. (http://itasoftware.com)
How about you all?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I wrote:
Allmydata, Inc. uses Twisted in its current proprietary consumer backup product [1] as well as in its free-software distributed storage grid project [2].
...
Oh, and apropos the discussion about Twisted-on-Windows and iocp reactor, I should mention that the current consumer-usable product [1] runs exclusively on Windows (although the non-GUI, server part of it also runs on our Linux servers). The new free-software project [2] runs on Linux, Mac, Windows, Cygwin. We use the select reactor exclusively. Every couple of months we experiment again with switching to iocp reactor, and we hit a bug that forces us to switch back to the select reactor, e.g. [3]. The good news is that it's a new bug each time... Regards, Zooko [3] http://allmydata.org/trac/tahoe/changeset/524
I use it in a peer-to-peer content addressible storage application for managing petabytes of store (twistedstorage.org). Chuck Wegrzyn On 5/21/07, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
On Mon, 2007-05-21 at 08:13 -0400, Ray Cote wrote:
At 5:55 PM -0400 5/19/07, Itamar Shtull-Trauring wrote:
Next question - who runs servers on *Windows*? :)
We do [Raises hand]. Twisted built with Py2Exe and running as an NT service. Have delivered several different applications in this configuration for clients. --Ray
Oh, I certainly understand running on Windows when clients pay you to do so. I was just surprised people do so on internal servers when they do have a choice.
Anyway, since this is bringing users out into the open - I'm sure we'd all like to hear what Twisted is being used for in different companies, and under which platforms. Do tell!
My company uses Twisted for a variety of uses, including reliable multicast data distribution, messaging, monitoring and managing complex processes, in the airline industry. (http://itasoftware.com)
How about you all?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
My company (pacecocorp.com) is using twisted for various services using Windows Server 2003 for the purpose of tracking and controlling containers, cranes and trucks for container tracking in ship and rail yards around the world. Currently, we're using it to send images and data from an embedded machine which has OCR algorithms for container identification. Regards, Ben Henry
-----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python- bounces@twistedmatrix.com] On Behalf Of Itamar Shtull-Trauring Sent: Monday, May 21, 2007 5:35 AM To: Twisted general discussion Subject: Tell us how you're using Twisted (was Re: [Twisted-Python] Win32IOCPReactor?)
On Mon, 2007-05-21 at 08:13 -0400, Ray Cote wrote:
At 5:55 PM -0400 5/19/07, Itamar Shtull-Trauring wrote:
Next question - who runs servers on *Windows*? :)
We do [Raises hand]. Twisted built with Py2Exe and running as an NT service. Have delivered several different applications in this configuration for clients. --Ray
Oh, I certainly understand running on Windows when clients pay you to do so. I was just surprised people do so on internal servers when they do have a choice.
Anyway, since this is bringing users out into the open - I'm sure we'd all like to hear what Twisted is being used for in different companies, and under which platforms. Do tell!
My company uses Twisted for a variety of uses, including reliable multicast data distribution, messaging, monitoring and managing complex processes, in the airline industry. (http://itasoftware.com)
How about you all?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, 2007-05-21 at 08:34 -0400, Itamar Shtull-Trauring wrote:
How about you all?
Software to monitor anything connected to a computer, mostly via SNMP, but also TCP, ICMP, and any custom program. [1] The BEEPy pure Python BEEP implementation. [2] libsnmp, a pure Python SNMP library.[3] Though not purely twisted code, we use it with twisted in seafelt, and there are backburner plans to merge it into twisted. (Thanks also to Phil Mayers for the libsnmp mention. I'm really pleased that's worked out so well for you). [1] http://www.seafelt.com [2] http://beepy.sourceforge.net/ [3] http://www.seafelt.com/libsnmp -- Justin Warren <daedalus@eigenmagic.com>
We do most development on MSW and also have some non-productive testsites running for demonstration purposes. One is an implementation of the most popular Swiss card game 'Jass' where you can play against three 'computer players' on jass.thie.ch (use login/pass: sf.tv when propmted. The other is a demo for a webapp for looking up phone numbers and address details for all registered customers of Swiss Telecom. I also married some example code (serial GPS stuff) with Athena/LivePages thus providing a browser interface for a serially connected industrial SPS. Another Athena/Livepage mini example I wrote involved a standard IXUS Canon camera (USB connected), providing access to the cams functions via the browser. Werner Itamar Shtull-Trauring wrote:
On Mon, 2007-05-21 at 08:13 -0400, Ray Cote wrote:
At 5:55 PM -0400 5/19/07, Itamar Shtull-Trauring wrote:
Next question - who runs servers on *Windows*? :) We do [Raises hand]. Twisted built with Py2Exe and running as an NT service. Have delivered several different applications in this configuration for clients. --Ray
Oh, I certainly understand running on Windows when clients pay you to do so. I was just surprised people do so on internal servers when they do have a choice.
Anyway, since this is bringing users out into the open - I'm sure we'd all like to hear what Twisted is being used for in different companies, and under which platforms. Do tell!
My company uses Twisted for a variety of uses, including reliable multicast data distribution, messaging, monitoring and managing complex processes, in the airline industry. (http://itasoftware.com)
How about you all?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
There is an IOCP buildbot that illustrates the current state of the IOCP reator against the twisted test suite. Just look at the win32-py2.5-iocp column on http://twistedmatrix.com/buildbot/ Happy coding, Mike Itamar Shtull-Trauring wrote:
On Mon, 2007-05-21 at 16:02 +0200, Trent Nelson wrote:
Basic TCP should work. There's a rewrite of the IOCP reactor in a
branch
which is likely better.
I believe Volodymyr already pointed it out, but can I just confirm you're referring to this branch:
svn://svn.twistedmatrix.com/svn/Twisted/branches/iocpreactor-1760-3/
Yes.
Can anyone comment on the status of these two branches?
iocp-producer-1657/ iocp_syshandle-2021/
I think these involve bugs in the current IOCP code.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
There is an IOCP buildbot that illustrates the current state of the IOCP reator against the twisted test suite.
Just look at the win32-py2.5-iocp column on http://twistedmatrix.com/buildbot/
Happy coding, Mike
Great link, thanks. Least I've got a starting point now (address test failures) if I want to start digging around in the code }:> Trent.
Trent, You can kick off a buildbot run against any brach. Feel free do trigger a run against the IOCP branch if you want check the status of the latest IOCP fixes. By default, buildbot runs against trunk Happy Coding, Mike Trent Nelson wrote:
There is an IOCP buildbot that illustrates the current state of the
IOCP
reator against the twisted test suite.
Just look at the win32-py2.5-iocp column on http://twistedmatrix.com/buildbot/
Happy coding, Mike
Great link, thanks. Least I've got a starting point now (address test failures) if I want to start digging around in the code }:>
Trent.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Il giorno lun, 21/05/2007 alle 08.34 -0400, Itamar Shtull-Trauring ha scritto:
Oh, I certainly understand running on Windows when clients pay you to do so. I was just surprised people do so on internal servers when they do have a choice.
Anyway, since this is bringing users out into the open - I'm sure we'd all like to hear what Twisted is being used for in different companies, and under which platforms. Do tell!
My company uses Twisted for a variety of uses, including reliable multicast data distribution, messaging, monitoring and managing complex processes, in the airline industry. (http://itasoftware.com)
How about you all?
I used twisted to code HL7 message transport protocol. I'm in the process to improve the protocol to be more secure. Bye sc -- Stefano Canepa aka sc: sc@linux.it http://www.stefanocanepa.it Three great virtues of a programmer: laziness, impatience and hubris. Le tre grandi virtù di un programmatore: pigrizia, impazienza e arroganza. (Larry Wall)
On 6/21/07, Stefano Canepa <sc@linux.it> wrote:
Anyway, since this is bringing users out into the open - I'm sure we'd all like to hear what Twisted is being used for in different companies, and under which platforms. Do tell!
I'm planning to use Twisted to monitor events from an Asterisk PBX and feed multiple Flex/AIR clients on extension status, voicemail, etc. etc. I'll probably use Linux 2.6 - but whatever works best. Arnar
[ Itamar Shtull-Trauring ]: ----------------------------- |Anyway, since this is bringing users out into the open - I'm sure we'd |all like to hear what Twisted is being used for in different companies, |and under which platforms. Do tell! | |My company uses Twisted for a variety of uses, including reliable |multicast data distribution, messaging, monitoring and managing complex |processes, in the airline industry. (http://itasoftware.com) | |How about you all? We did several things with this great great great tool, but the biggest one was to implement an MRCP [1] server that binds with a Text-To-Speech (TTS) engine for Brazilian Portuguese. It runs on Linux, Windows and Solaris. Unfortunately the company [2] that commissioned the project from me did not gave me permission to open up the code. Nevertheless, I am available to comment about it and help any endeavour to make a open version of it. Sorry about the time warp. [1] http://en.wikipedia.org/wiki/Media_Resource_Control_Protocol [2] http://www.cpqd.com.br/site/ContentView.php?cd=3774 cheers, Rod Senra --- http://rodrigo.senra.nom.br
Itamar Shtull-Trauring napisał(a):
Anyway, since this is bringing users out into the open - I'm sure we'd all like to hear what Twisted is being used for in different companies, and under which platforms. Do tell!
My company uses Twisted for a variety of uses, including reliable multicast data distribution, messaging, monitoring and managing complex processes, in the airline industry. (http://itasoftware.com)
How about you all?
In Sensisoft we use Twisted to build webservice (REST) parts of systems we sell, i.e. full-text search/document indexing servers. -- Jarek Zgoda Skype: jzgoda | GTalk: zgoda@jabber.aster.pl | voice: +48228430101 "We read Knuth so you don't have to." (Tim Peters)
Hi! We (Acat) use Twisted to implement interfaces to hotel PMSs (Property Management Systems). Great tool, thanks a lot. -- Bye, Andreas
I'm building xooChat live help service and it uses Twisted as the backend. On 9/4/07, Andreas Poisel <a.poisel@acat.cc> wrote:
Hi!
We (Acat) use Twisted to implement interfaces to hotel PMSs (Property Management Systems).
Great tool, thanks a lot.
-- Bye, Andreas
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Good luck, Pavel Bastov xooChat Evangelist and Team Leader http://www.xoochat.com/
participants (19)
-
Andreas Poisel -
Arnar Birgisson -
Benjamin Henry -
Chas. -
Eric Mangold -
Itamar Shtull-Trauring -
Jarek Zgoda -
Josh Ritter -
Justin Warren -
Michael Schneider -
Pavel Bastov -
Phil Mayers -
Ray Cote -
Rodrigo Senra -
Stefano Canepa -
Trent Nelson -
Volodymyr Orlenko -
Werner Thie -
zooko＠zooko.com