[Twisted-Python] is there way to baypass value limit of 64kb in twisted.protocols.amp
Hello! I'm currently trying to figure out, how to bypass amp value limit of 64kb. First tought was to split string into chunks <64kb, then join before processing. Problem is, how to dinamicaly generate amp.Command argument list. Maybe there is other well known method how to achieve this? Thank You!
On Nov 30, 2009, at 2:10 PM, MārisR wrote:
I'm currently trying to figure out, how to bypass amp value limit of 64kb. First tought was to split string into chunks <64kb, then join before processing. Problem is, how to dinamicaly generate amp.Command argument list. Maybe there is other well known method how to achieve this?
You can define your own Argument which reads multiple values, rather than dynamically generating the arguments list. This is what you want anyway, because your application code wants to receive one value; the fact that you are splitting/joining at the protocol level should not be important. There's an example of this implemented here: http://bazaar.launchpad.net/%7Eglyph/%2Bjunk/amphacks/annotate/head%3A/pytho... The reason that this isn't currently included in the core AMP implementation is that if you need to transfer values larger than this limit, perhaps you should be doing the transfer in some other way. Streaming the result? Doing the transfer out-of-band as (for example) an HTTP request? I think this might be an unhelpful way to ask people to consider this problem, so we may end up packaging something similar in Twisted eventually. One way to do this might be by doing this: <http://twistedmatrix.com/trac/ticket/2529>, another might be bundling the contents of mediumbox.py (or something like it) into amp.py.
Glyph Lefkowitz wrote:
On Nov 30, 2009, at 2:10 PM, MārisR wrote:
I'm currently trying to figure out, how to bypass amp value limit of 64kb. First tought was to split string into chunks <64kb, then join before processing. Problem is, how to dinamicaly generate amp.Command argument list. Maybe there is other well known method how to achieve this?
You can define your own Argument which reads multiple values, rather than dynamically generating the arguments list. This is what you want anyway, because your application code wants to receive one value; the fact that you are splitting/joining at the protocol level should not be important.
There's an example of this implemented here:
http://bazaar.launchpad.net/%7Eglyph/%2Bjunk/amphacks/annotate/head%3A/pytho...
The reason that this isn't currently included in the core AMP implementation is that if you need to transfer values larger than this limit, perhaps you should be doing the transfer in some other way. Streaming the result? Doing the transfer out-of-band as (for example) an HTTP request?
I think this might be an unhelpful way to ask people to consider this problem, so we may end up packaging something similar in Twisted eventually. One way to do this might be by doing this: <http://twistedmatrix.com/trac/ticket/2529>, another might be bundling the contents of mediumbox.py (or something like it) into amp.py.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
mediumbox done the trick with little patching to support utf8, Big thanks, for advises :)
participants (3)
-
Glyph Lefkowitz -
Maris Ruskulis -
MārisR