[Twisted-Python] Porting twisted.spread.banana to py3
I'm a gsoc student working on the buildbot project, porting it to py3. Part of this is looking at porting the dependencies buildbot has in twisted and try to port them.
twisted.spread is the most used twisted dependency, so I have started working on this. I started working on banana.py and I have encountered a question. In dataRecieved (banana.py:174). A chunk is added to the buffer. On python3, how should the distinction be made between string and bytes. Buffer seems to be a string, the chunk seems to be bytes, and sometimes opposite.
Any help would be appreciated
regards, Jørn Lomax
Am Mittwoch, 1. Juli 2015, 11:33:58 schrieb Jørn Lomax:
I'm a gsoc student working on the buildbot project, porting it to py3. Part of this is looking at porting the dependencies buildbot has in twisted and try to port them.
twisted.spread is the most used twisted dependency, so I have started working on this.
I would much appreciate seeing spread ported.
Last year I actually already did (all tests passed) but then I did not find the time to make separate tickets to be reviewed.
Actually porting spread is easier than getting the source into something acceptable by the core developers.
But I published what I got, see my mail on this list from Nov 3 2014:
there now is https://github.com/wrohdewald/twisted.git
branch spread-py3-7598
If I would continue with this now, I would probably first rebase that branch onto master (not sure if the twisted project still uses svn or if the migration to git has already been done).
You could also look for tickets opened by myself (wolfgang61), most of them should be about porting spread.
I started working on banana.py and I have encountered a question. In dataRecieved (banana.py:174). A chunk is added to the buffer. On python3, how should the distinction be made between string and bytes. Buffer seems to be a string, the chunk seems to be bytes, and sometimes opposite.
I believe that chunk is a banana-encoded string, so there should be no need to treat it as bytes.
On 2015-07-01 05:33, Jørn Lomax wrote:
I'm a gsoc student working on the buildbot project, porting it to py3. Part of this is looking at porting the dependencies buildbot has in twisted and try to port them.
twisted.spread is the most used twisted dependency, so I have started working on this. I started working on banana.py and I have encountered a question. In dataRecieved (banana.py:174). A chunk is added to the buffer. On python3, how should the distinction be made between string and bytes. Buffer seems to be a string, the chunk seems to be bytes, and sometimes opposite.
Banana calls bytes "string". So if you're using banana there is only bytes support. Python 3 vs 2 doesn't matter, it's always going to be bytes.
Thank you very much. That is the exact answer I wanted :)
I'm guessing maybe the tests need updating, because they seem to send use unicode data, which of course didn't make any difference in py2, but now it does.
On 01.07.2015 18:40, Itamar Turner-Trauring wrote:
On 2015-07-01 05:33, Jørn Lomax wrote:
I'm a gsoc student working on the buildbot project, porting it to py3. Part of this is looking at porting the dependencies buildbot has in twisted and try to port them.
twisted.spread is the most used twisted dependency, so I have started working on this. I started working on banana.py and I have encountered a question. In dataRecieved (banana.py:174). A chunk is added to the buffer. On python3, how should the distinction be made between string and bytes. Buffer seems to be a string, the chunk seems to be bytes, and sometimes opposite.
Banana calls bytes "string". So if you're using banana there is only bytes support. Python 3 vs 2 doesn't matter, it's always going to be bytes.
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Jul 1, 2015, at 9:46 AM, Jørn Lomax northlomax@gmail.com wrote:
Thank you very much. That is the exact answer I wanted :)
I'm guessing maybe the tests need updating, because they seem to send use unicode data, which of course didn't make any difference in py2, but now it does.
We may actually need to change the wire protocol to allow some applications to continue to work on py3, to allow for a 'text' type that corresponds to Python unicode. This should probably be done after the initial port though.
-glyph
On 2015-07-01 17:33, Glyph wrote:
We may actually need to change the wire protocol to allow some applications to continue to work on py3, to allow for a 'text' type that corresponds to Python unicode. This should probably be done after the initial port though.
Really? That seems like Jelly's job, not Banana's.
On Jul 2, 2015, at 6:13 AM, Itamar Turner-Trauring itamar@itamarst.org wrote:
On 2015-07-01 17:33, Glyph wrote:
We may actually need to change the wire protocol to allow some applications to continue to work on py3, to allow for a 'text' type that corresponds to Python unicode. This should probably be done after the initial port though.
Really? That seems like Jelly's job, not Banana's.
Yes, you're right.
-glyph
On Jul 2, 2015, at 11:12 AM, Glyph glyph@twistedmatrix.com wrote:
On Jul 2, 2015, at 6:13 AM, Itamar Turner-Trauring <itamar@itamarst.org mailto:itamar@itamarst.org> wrote:
On 2015-07-01 17:33, Glyph wrote:
We may actually need to change the wire protocol to allow some applications to continue to work on py3, to allow for a 'text' type that corresponds to Python unicode. This should probably be done after the initial port though.
Really? That seems like Jelly's job, not Banana's.
Yes, you're right.
Furthermore (I thought I remembered this!) Jelly already has support for unicode, so nothing will need to change :).
-glyph
I have found another issue I'm wondering a little how to handle. There are places in the bana_test file that sys.maxint is used. This no longer exists in python3, so how should it be handled. Should we make an artificial maxint to use for python3. I might want to note that it is not used for comparison, but to create an integer over a given size:
foo = [1, 2, [3, 4], [30.5, 40.2], 5, [b"six", b"seven", [b"eight", 9]], [10], # TODO: currently the C implementation's a bit buggy... sys.maxint * long(3), sys.maxint * long(2), sys.maxint * long(-2)]
regards, Jørn lomax
On 02.07.2015 21:21, Glyph Lefkowitz wrote:
On Jul 2, 2015, at 11:12 AM, Glyph <glyph@twistedmatrix.com mailto:glyph@twistedmatrix.com> wrote:
On Jul 2, 2015, at 6:13 AM, Itamar Turner-Trauring <itamar@itamarst.org mailto:itamar@itamarst.org> wrote:
On 2015-07-01 17:33, Glyph wrote:
We may actually need to change the wire protocol to allow some applications to continue to work on py3, to allow for a 'text' type that corresponds to Python unicode. This should probably be done after the initial port though.
Really? That seems like Jelly's job, not Banana's.
Yes, you're right.
Furthermore (I thought I remembered this!) Jelly already has support for unicode, so nothing will need to change :).
-glyph
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Jul 3, 2015, at 2:00 PM, Jørn Lomax northlomax@gmail.com wrote:
I have found another issue I'm wondering a little how to handle. There are places in the bana_test file that sys.maxint is used. This no longer exists in python3, so how should it be handled. Should we make an artificial maxint to use for python3. I might want to note that it is not used for comparison, but to create an integer over a given size:
foo = [1, 2, [3, 4], [30.5, 40.2], 5, [b"six", b"seven", [b"eight", 9]], [10], # TODO: currently the C implementation's a bit buggy... sys.maxint * long(3), sys.maxint * long(2), sys.maxint * long(-2)]
regards, Jørn lomax
These tests are just making sure that it's capable of serializing things that don't fit into a machine integer; sys.maxsize should be an acceptable substitute.
-glyph
Am Freitag, 3. Juli 2015, 23:00:45 schrieb Jørn Lomax:
I have found another issue I'm wondering a little how to handle. There are places in the bana_test file that sys.maxint is used. This no longer exists in python3, so how should it be handled. Should we make an artificial maxint to use for python3. I might want to note that it is not used for comparison, but to create an integer over a given size:
i5:~/tmp/twisted/twisted/test (spread-py3-7598) git show a0d35fa commit a0d35fa912b055bd65fb76b597de1d64675711d8 Author: Wolfgang Rohdewald wolfgang@rohdewald.de Date: Sun Nov 2 18:50:54 2014 +0100
twisted.test.test_banana: PY3 has no sys.maxint
diff --git a/twisted/test/test_banana.py b/twisted/test/test_banana.py index e0504d0..c43bdc8 100644 --- a/twisted/test/test_banana.py +++ b/twisted/test/test_banana.py @@ -455,10 +455,11 @@ class BananaTestCase(BananaTestBase): Test feeding the data byte per byte to the receiver. Normally data is not split. """ + maxint = sys.maxsize if _PY3 else sys.maxint foo = [1, 2, [3, 4], [30.5, 40.2], 5, [b"six", b"seven", [b"eight", 9]], [10], # TODO: currently the C implementation's a bit buggy... - sys.maxint * 3l, sys.maxint * 2l, sys.maxint * long(-2)] + maxint * long(3), maxint * long(2), maxint * long(-2)] self.enc.sendEncoded(foo) self.feed(self.io.getvalue()) assert self.result == foo, "%s!=%s" % (repr(self.result), repr(foo))
participants (5)
-
Glyph -
Glyph Lefkowitz
-
Itamar Turner-Trauring -
Jørn Lomax -
Wolfgang Rohdewald