[Twisted-Python] epollreactor for pyicq-0.6-t
Hi, I hope some one could offer me some help on this. I am quite new to both python and twisted. The thing I do may sounds a bit wierd. I am using a pyicq-0.6-t. As pyicq-0.6-t only works with twisted-2.2 or earlier, so I chose to install twisted-2.2. However, I wished to use epollreactor as the default selectreactor only supports 1024 file descriptors. So I copied the relevant files of epollreactor which is installed on twisted-2.5 to the site-packages/twisted-2.2 directory. I copied 5 files in the total, epollreactor.py, epoll..pyx to the twisted/internet directory, _epoll.py _epoll.pyc and _epoll.so to twisted/python directory. I can't remember the exact file names. I copied it with a hope that it works. I am not sure whether I am doing thing stupid here. correct me if I am. The thing is, after I changed the reactor.py to import epollreactor, pyicq-0.6-t seems to be using epollreactor but it still can only supports 1024 open sockets. I thought the problem is only in selectreactor? I figured this out by doing a netstat -a -n on linux and found that pyicq-0.6-t only opened about 1017 maximum concurrent sockets. I presume that would mean only around 1000 connections can be made. Although strange, but I do have a reason to use pyicq-0.6-t which is an outdated version. thanks a lot. regards, Steven Notice The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying or distribution of the message, or any action taken by you in reliance on it, is prohibited and may be unlawful. If you have received this message in error, please delete it and contact the sender immediately. Thank you.
On 02:50 pm, steven_zhou@ctl.creative.com wrote:
The thing I do may sounds a bit wierd. I am using a pyicq-0.6-t. As pyicq-0.6-t only works with twisted-2.2 or earlier, so I chose to install twisted-2.2. However, I wished to use epollreactor as the default selectreactor only supports 1024 file descriptors. So I copied the relevant files of epollreactor which is installed on twisted-2.5 to the site-packages/twisted-2.2 directory. I copied 5 files in the total, epollreactor.py, epoll..pyx to the twisted/internet directory, _epoll.py _epoll.pyc and _epoll.so to twisted/python directory. I can't remember the exact file names. I copied it with a hope that it works. I am not sure whether I am doing thing stupid here. correct me if I am.
"stupid" might be a harsh word, but what you're doing shouldn't be expected to work. I'm surprised that it has worked as well as you suggest that it has.
The thing is, after I changed the reactor.py to import epollreactor, pyicq-0.6-t seems to be using epollreactor but it still can only supports 1024 open sockets. I thought the problem is only in selectreactor?
You might want to check your ulimits. However, twisted 2.2 might have been calling select() in any number of places. And Python 2.4.3 had a bug in it which would have caused a problem regardless. And there might be a bug in the interaction of random bits of code from twisted 2.5 and 2.2. Whatever the problem is, you're on your own trying to sort it out until you can upgrade to a supported version :)
The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized.
Please do not attach notices like this to messages on this mailing list, given that the "addressee" in this case is the entire internet :).
On Tue, 8 Jan 2008 22:50:54 +0800, Steven ZHOU <steven_zhou@ctl.creative.com> wrote:
Hi,
I hope some one could offer me some help on this. I am quite new to both python and twisted.
The thing I do may sounds a bit wierd. I am using a pyicq-0.6-t. As pyicq-0.6-t only works with twisted-2.2 or earlier, so I chose to install twisted-2.2. However, I wished to use epollreactor as the default selectreactor only supports 1024 file descriptors. So I copied the relevant files of epollreactor which is installed on twisted-2.5 to the site-packages/twisted-2.2 directory. I copied 5 files in the total, epollreactor.py, epoll..pyx to the twisted/internet directory, _epoll.py _epoll.pyc and _epoll.so to twisted/python directory. I can't remember the exact file names. I copied it with a hope that it works. I am not sure whether I am doing thing stupid here. correct me if I am.
I don't know if this works. I've never tried it. The epoll reactor implementation shares code from other parts of Twisted. It may be the case that that code changed between Twisted 2.2 and Twisted 2.5 in a way which will prevent epollreactor from working. It may also be the case that it hasn't.
The thing is, after I changed the reactor.py to import epollreactor, pyicq-0.6-t seems to be using epollreactor but it still can only supports 1024 open sockets. I thought the problem is only in selectreactor? I figured this out by doing a netstat -a -n on linux and found that pyicq-0.6-t only opened about 1017 maximum concurrent sockets. I presume that would mean only around 1000 connections can be made.
Although strange, but I do have a reason to use pyicq-0.6-t which is an outdated version.
Are you using Python 2.4.3? This sounds like it may be related to the issue described here: http://jcalderone.livejournal.com/2006/05/26/ Jean-Paul
Hi Glyph, Thanks for the prompt help. I didn't expect it to work initially in fact. I did set ulimit -n if that's the only thing to set. Yes, I am having python 2.4, not sure whether it's 2.4.3, I am now at home and will check that when I am back to office. The reason I used an outdated version of pyicq-0.6-t is to find out the scalability of pyicqt, pymsnt etc. Unfortunately, I only managed to find a clone of ICQ server (IServerd) which supports older version of icq oscar protocols. So I can stress test with iserverd. Where can I find scalability report of twisted based applications (e.g. pymsnt)? That would be even of greater help. regards, Steven Notice The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying or distribution of the message, or any action taken by you in reliance on it, is prohibited and may be unlawful. If you have received this message in error, please delete it and contact the sender immediately. Thank you. glyph@divmod.com Sent by: twisted-python-bo To unces@twistedmatr Twisted general discussion ix.com <twisted-python@twistedmatrix.com> cc 01/08/2008 10:59 Subject PM Re: [Twisted-Python] epollreactor for pyicq-0.6-t Please respond to Twisted general discussion <twisted-python@t wistedmatrix.com> On 02:50 pm, steven_zhou@ctl.creative.com wrote:
The thing I do may sounds a bit wierd. I am using a pyicq-0.6-t. As pyicq-0.6-t only works with twisted-2.2 or earlier, so I chose to install twisted-2.2. However, I wished to use epollreactor as the default selectreactor only supports 1024 file descriptors. So I copied the relevant files of epollreactor which is installed on twisted-2.5 to the site-packages/twisted-2.2 directory. I copied 5 files in the total, epollreactor.py, epoll..pyx to the twisted/internet directory, _epoll.py _epoll.pyc and _epoll.so to twisted/python directory. I can't remember the exact file names. I copied it with a hope that it works. I am not sure whether I am doing thing stupid here. correct me if I am.
"stupid" might be a harsh word, but what you're doing shouldn't be expected to work. I'm surprised that it has worked as well as you suggest that it has.
The thing is, after I changed the reactor.py to import epollreactor, pyicq-0.6-t seems to be using epollreactor but it still can only supports 1024 open sockets. I thought the problem is only in selectreactor?
You might want to check your ulimits. However, twisted 2.2 might have been calling select() in any number of places. And Python 2.4.3 had a bug in it which would have caused a problem regardless. And there might be a bug in the interaction of random bits of code from twisted 2.5 and 2.2. Whatever the problem is, you're on your own trying to sort it out until you can upgrade to a supported version :)
The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized.
Please do not attach notices like this to messages on this mailing list, given that the "addressee" in this case is the entire internet :). _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ForwardSourceID:NT0001A012
Steven ZHOU wrote:
Hi,
I hope some one could offer me some help on this. I am quite new to both python and twisted.
The thing I do may sounds a bit wierd. I am using a pyicq-0.6-t. As pyicq-0.6-t only works with twisted-2.2 or earlier, so I chose to install twisted-2.2. However, I wished to use epollreactor as the default selectreactor only supports 1024 file descriptors. So I copied the relevant files of epollreactor which is installed on twisted-2.5 to the site-packages/twisted-2.2 directory. I copied 5 files in the total, epollreactor.py, epoll..pyx to the twisted/internet directory, _epoll.py _epoll.pyc and _epoll.so to twisted/python directory. I can't remember the exact file names. I copied it with a hope that it works. I am not sure whether I am doing thing stupid here. correct me if I am.
I dub thee "Frankenstein": "Hmm, let's see, a bit of 2.2, and if I use the same amount of 2.6 that should come out right around 2.4". If only that were true ... regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/
Hi Jean-Paul, Thanks for the help. It's indeed the python 2.4.3 bug. I changed it to 2.4.4 or 2.5. It works. The adhoc way I did to epollreactor seems to work. Maybe I am lucky. regards, Steven Notice The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying or distribution of the message, or any action taken by you in reliance on it, is prohibited and may be unlawful. If you have received this message in error, please delete it and contact the sender immediately. Thank you. Jean-Paul Calderone <exarkun@divmod.c To om> Twisted general discussion Sent by: <twisted-python@twistedmatrix.com> twisted-python-bo cc unces@twistedmatr ix.com Subject Re: [Twisted-Python] epollreactor for pyicq-0.6-t 01/08/2008 11:03 PM Please respond to Twisted general discussion <twisted-python@t wistedmatrix.com> On Tue, 8 Jan 2008 22:50:54 +0800, Steven ZHOU <steven_zhou@ctl.creative.com> wrote:
Hi,
I hope some one could offer me some help on this. I am quite new to both python and twisted.
The thing I do may sounds a bit wierd. I am using a pyicq-0.6-t. As pyicq-0.6-t only works with twisted-2.2 or earlier, so I chose to install twisted-2.2. However, I wished to use epollreactor as the default selectreactor only supports 1024 file descriptors. So I copied the
relevant
files of epollreactor which is installed on twisted-2.5 to the site-packages/twisted-2.2 directory. I copied 5 files in the total, epollreactor.py, epoll..pyx to the twisted/internet directory, _epoll.py _epoll.pyc and _epoll.so to twisted/python directory. I can't remember the exact file names. I copied it with a hope that it works. I am not sure whether I am doing thing stupid here. correct me if I am.
I don't know if this works. I've never tried it. The epoll reactor implementation shares code from other parts of Twisted. It may be the case that that code changed between Twisted 2.2 and Twisted 2.5 in a way which will prevent epollreactor from working. It may also be the case that it hasn't.
The thing is, after I changed the reactor.py to import epollreactor, pyicq-0.6-t seems to be using epollreactor but it still can only supports 1024 open sockets. I thought the problem is only in selectreactor? I figured this out by doing a netstat -a -n on linux and found that pyicq-0.6-t only opened about 1017 maximum concurrent sockets. I presume that would mean only around 1000 connections can be made.
Although strange, but I do have a reason to use pyicq-0.6-t which is an outdated version.
Are you using Python 2.4.3? This sounds like it may be related to the issue described here: http://jcalderone.livejournal.com/2006/05/26/ Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ForwardSourceID:NT0001A016
participants (4)
-
glyph＠divmod.com -
Jean-Paul Calderone -
Steve Holden -
Steven ZHOU