[Twisted-Python] Does anyone use ampoule in production?
Hi all.
just a few hours ago, I discovered ampoule is probably a right choice to run my analyze codes. I want to know whether your guys have used it for production purposes or not It seems the development has been frozen for a while and I noticed many bugs reported on the board. Some bugs looks like quite serious for example: a bug mentioned occasionally, startWorker may generate double process until the system hanging.
If those bugs still exist, i am gonna try to fix some of them if it is impossible.
Regards
GELIN YAN
2011/11/18 gelin yan dynamicgl@gmail.com:
Hi all. just a few hours ago, I discovered ampoule is probably a right choice to run my analyze codes. I want to know whether your guys have used it for production purposes or not It seems the development has been frozen for a while and I noticed many bugs reported on the board. Some bugs looks like quite serious for example: a bug mentioned occasionally, startWorker may generate double process until the system hanging. If those bugs still exist, i am gonna try to fix some of them if it is impossible. Regards GELIN YAN _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Good question! I've recently adopted our game server for processing client requests in several processes using ampoule as tool for managing process pool. New approach has significantly more performance(5-100% more queries per second in depends on type of query and the number of concurrent clients) and high scalability. Modern servers have a lot of processor cores, but GIL in python kills all opportunities to use multi-threading for heavy computation. But my solution is still not in production. I wanted to ask the same question, but other tasks distracted me from ampoule and multi-processing. I should also mention that my tests were rather artificial, I don't know how ampoule would work when many real players came to our server and started playing.
I have also several patches for ampoule, which allows me send many-many client's requests to several processes in pool(from the box ampoule allows to process one query per process).
I'm really keen on this question, thank you, GELIN YAN.
Hi Romanoff
Have you suffered from transfer large size of data to child process? I encountered a problem: it seems i am unable to send any chunks of data larger than 4000 bytes. I guess there is a limit.
On Sat, Nov 19, 2011 at 3:58 AM, Romanoff Alexei drednout.by@gmail.comwrote:
2011/11/18 gelin yan dynamicgl@gmail.com:
Hi all. just a few hours ago, I discovered ampoule is probably a right choice
to
run my analyze codes. I want to know whether your guys have used it for production purposes or not It seems the development has been frozen for a while and I noticed many bugs reported on the board. Some bugs looks like quite serious for example: a bug mentioned occasionally, startWorker may generate double process until the system hanging. If those bugs still exist, i am gonna try to fix some of them if it
is
impossible. Regards GELIN YAN _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Good question! I've recently adopted our game server for processing client requests in several processes using ampoule as tool for managing process pool. New approach has significantly more performance(5-100% more queries per second in depends on type of query and the number of concurrent clients) and high scalability. Modern servers have a lot of processor cores, but GIL in python kills all opportunities to use multi-threading for heavy computation. But my solution is still not in production. I wanted to ask the same question, but other tasks distracted me from ampoule and multi-processing. I should also mention that my tests were rather artificial, I don't know how ampoule would work when many real players came to our server and started playing.
I have also several patches for ampoule, which allows me send many-many client's requests to several processes in pool(from the box ampoule allows to process one query per process).
I'm really keen on this question, thank you, GELIN YAN.
Best regards, Alexei Romanoff software developer, Melesta
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Nov 21, 2011, at 1:25 AM, gelin yan wrote:
Have you suffered from transfer large size of data to child process? I encountered a problem: it seems i am unable to send any chunks of data larger than 4000 bytes. I guess there is a limit.
What do you mean by this? Do you have a test case that demonstrates this limit?
-glyph
Hi glyph
I attached a file for this purpose. there is a number 4073 inside the code. when i use any smaller value, the function return a correct result. I use win 7 & twisted 11.1 at the moment. I am not sure whether this limit comes from os or not.
thanks
gelin yan
On Mon, Nov 21, 2011 at 2:57 PM, Glyph glyph@twistedmatrix.com wrote:
On Nov 21, 2011, at 1:25 AM, gelin yan wrote:
Have you suffered from transfer large size of data to child process? Iencountered a problem: it seems i am unable to send any chunks of data larger than 4000 bytes. I guess there is a limit.
What do you mean by this? Do you have a test case that demonstrates this limit?
-glyph
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Sun, Nov 20, 2011 at 11:40 PM, gelin yan dynamicgl@gmail.com wrote:
Hi glyph I attached a file for this purpose. there is a number 4073 inside the code. when i use any smaller value, the function return a correct result. I use win 7 & twisted 11.1 at the moment. I am not sure whether this limit comes from os or not.
A few intrepid hackers working on Win32 platforms reported similar issues with Bravo, which uses Ampoule extensively. We chalked it up to Win32 weirdness and told people to use better operating systems when deploying Bravo. The only hack which seemed to alleviate the problem was changing Twisted's subprocess management to allocate massive (64KiB) buffers for the subprocess pipes instead of zeroing them out. I have no idea what the correct answer is.
~ C.
Hi
I am happy to see some other teams have used it in a production environment. If there are no serious bugs, I am gonna use it to implement my analyzer which need to take benefit from multiple cores. At the moment, ampoule seems is the most intuitive way to reap the power of multiple cores.
thanks
gelin yan
On Mon, Nov 21, 2011 at 4:39 PM, Corbin Simpson mostawesomedude@gmail.comwrote:
On Sun, Nov 20, 2011 at 11:40 PM, gelin yan dynamicgl@gmail.com wrote:
Hi glyph I attached a file for this purpose. there is a number 4073 inside
the
code. when i use any smaller value, the function return a correct
result. I
use win 7 & twisted 11.1 at the moment. I am not sure whether this limit comes from os or not.
A few intrepid hackers working on Win32 platforms reported similar issues with Bravo, which uses Ampoule extensively. We chalked it up to Win32 weirdness and told people to use better operating systems when deploying Bravo. The only hack which seemed to alleviate the problem was changing Twisted's subprocess management to allocate massive (64KiB) buffers for the subprocess pipes instead of zeroing them out. I have no idea what the correct answer is.
~ C.
-- When the facts change, I change my mind. What do you do, sir? ~ Keynes
Corbin Simpson MostAwesomeDude@gmail.com
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
gelin yan <dynamicgl <at> gmail.com> writes:
Hi glyph
I attached a file for this purpose. there is a number 4073 inside the
code. when i use any smaller value, the function return a correct result. I use win 7 & twisted 11.1 at the moment. I am not sure whether this limit comes from os or not.
hi, there is a limit, but it's 64kb [1][2], not 4k, in fact i can successfully run your program in linux until i use something bigger than 64k.
maybe there's an os-specific bug somewhere.
bye, flavio
[1] http://twistedmatrix.com/documents/current/api/twisted.protocols.amp.html (the notes section)
[2] http://twistedmatrix.com/pipermail/twisted-python/2009-February/019218.html (mail by ampoule author)
On 21.11.2011 8:40, gelin yan wrote:
Hi glyph
I attached a file for this purpose. there is a number 4073 inside thecode. when i use any smaller value, the function return a correct result. I use win 7& twisted 11.1 at the moment. I am not sure whether this limit comes from os or not.
thanks
gelin yan
That sounds like issue 5365: http://twistedmatrix.com/trac/ticket/5365
Does the patch there fix the issue for you?
Best regards, Ziga
Thanks for that... the patch you mentioned definitely solve this problem...it looks like the default buffer size is 4096 bytes on windows.
minor change of the patch for twisted 11.1
_pollingfile._PIPE_BUFFER_SIZE has been changed to _pollingfile.FULL_BUFFER_SIZE
* * thanks again for your guys help.
Regards
gelin yan
On Mon, Nov 21, 2011 at 7:02 PM, Žiga Seilnacht ziga.seilnacht@gmail.comwrote:
On 21.11.2011 8:40, gelin yan wrote:
Hi glyph
I attached a file for this purpose. there is a number 4073 insidethe
code. when i use any smaller value, the function return a correct
result. I
use win 7& twisted 11.1 at the moment. I am not sure whether this limit comes from os or not.
thanks
gelin yan
That sounds like issue 5365: http://twistedmatrix.com/trac/ticket/5365
Does the patch there fix the issue for you?
Best regards, Ziga
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (6)
-
Corbin Simpson -
Flavio Grossi -
gelin yan -
Glyph -
Romanoff Alexei -
Žiga Seilnacht