[Twisted-Python] porting twisted.spread to Python3
![](https://secure.gravatar.com/avatar/d57bac257873fa3c0993069a9cde47a5.jpg?s=120&d=mm&r=g)
I now have a local git branch with about 70 commits, always rebased onto current trunk. Those test suites pass with Python2 and Python3: test_banana, test_jelly, test_pb, test_pbfailure, test_persisted I also have a real app using PB (kajongg, a KDE game). Running the server and the client with any combination of Python 2 and Python 3 works but more testing is needed there. Especially transferring failures with non-ascii test, so I need to inject random errors. spread.banana is now fully covered by tests (was at 85% before). There is still a lot of code in twisted.spread which has no tests yet. I do not think I want to add missing tests for everything, only for those regions where I do changes or where it is not too difficult. And of course if an application triggers a bug yet uncovered by tests. Can you point me to other PB applications I could use for testing? Preferrably ones that are already mostly compatible with PY3 themselves, but that is not a _must_, I would try porting them unless it turns out to be too much work. And preferrably ones developed by people with a non-ascii native language. I do not yet trust Failures and logging with non-ascii. Below is the current list of commits. Everything you see there is still much in flux. This includes shuffling code between commits, combining and splitting commits, changing order. I want to open one ticket per commit. Simple things hopefully not needing much discussion first. Estimating one week per review/commit to trunk, this process will take more than one year. Often I can only add a new ticket after the previous one has been integrated because they depend on each other. So if it really seems to take that long, at some point I might just decide to put what I have to github and not care anymore. Which of course means I might have to distribute a patched twisted version with my application. I really want to leave Python2 behind, and only twisted is in the way. Latest versions of KDE do not have Python2 bindings anymore (KDE Frameworks 5). This log of course includes things being in tickets up for review. git log --pretty='format:%s' --reverse trunk..pb3 spread.banana: selecting an unsupported dialect now raises an Exception spread.banana now raises BananaError when receiving pb messages without pb being the selected dialect spread.banana: specific error message about unicode objects spread.banana: test coverage goes up from 85% to 100% test.test_banana: eliminate duplicate code twisted.python.reflect.safe_str on Python 3 converts utf-8 encoded bytes to clean str instead of "b'a'" test.test_{failure,banana,persisted,reflect}: consistently name all test methods test_* test.test_jelly: add tests for yet untested code spread.jelly: support the sets module only for Python 2 twisted.spread: Assume that Python always has unicode support enabled Ticket 7653. spread.jelly: deprecate unjellying instances spread.jelly can now transfer objects with new style classes cred.credentials: implements->implementer cred.checkers: implements->implementer test.test_pb: implements->implementer persisted.styles: add new tests persisted.styles.unpickleMethod: remove dead code spread.flavors: fix import paths, making them work with Python 3 too spread: Python 3: add from __future__ do not use argument tuple unpacking anymore, not available in Python 3 spread.publish and test.test_pb: replace file by open test.test_pb: use a constant for "BRAINS!" python.compat: get_im_func, get_im_self, get_im_class and use them in spread persisted.styles: simplify unpickleMethod persisted.styles: Port to Python 3 python.compat: for PY3, define long=int and xrange=range spread.banana: replace StringIO.StringIO by io.BytesIO python.compat.networkChar: New, and use it in spread.banana python.compat.networkString: improve Exception message spread.banana: change literals from "" to b"" where appropriate test.test_banana: fix a wrong assertion message test.test_banana: use itertools.chain test.test_banana: Python 3 does not know type long spread: Python 3 does not have sys.maxint spread.banana and test: Complete port to Python 3: handling of data spread.jelly: Unjellying instance is not supported with Python 3 spread.jelly: unify usage of the types module spread.jelly: consistently use _atom identifiers spread.jelly: complete port to Python 3 spread.pb, test.test_pbfailure: new syntax for except spread.pb: do not use argument tuple unpacking anymore, not available in Python 3 test.test_jelly now passes all tests with Python 3 cred.credentials.updateHash accepts unicode add ported modules to python/dist3.py spread.flavors: consistently use _atom identifiers spread.flavors: Python 3: eliminate __cmp__. Only support __eq__ and __ne__ spread: Use networkString/nativeString where appropriate spread.flavors: switch atoms from str to b"" test.test_pb: improve test_concurrentLogin: use a non-ascii password spread.pb: eliminate __cmp__, using FancyEqMixin test.test_pb: add comment to test_tooManyRefs: too slow spread.pb: introduce *_atom constants spread.pb: Python 3: use list() around some generators spread.pb.Broker: improve Exception message spread.pb and test: Python 3: xrange = range spread.pb and test: Python 3: use byte literals spread.pb: failure2Copyable: use _newInstance from jelly spread.pb: replace types.ListType with list spread.pb and test: use networkChar where appropriate test.test_pb: change from StringIO to io.BytesIO spread.pb: Python 3: make *_atom bytes python.logger.formatWithCall: support non-ascii strings. Why exactly does kajongg need this? python.compat: networkIdentifier, nativeIdentifier. Not yet used. SQUASH spread.pb: CopiedFailure now handles non-ascii strings. TEST MISSING spread.flavors: RemoteCopy supports non-ascii identifiers. TEST MISSING. Why does kajongg need this? python.reflect: _safeFormat must handle unicode python.util: new debug function stack(). REMOVE AGAIN python.compat: nativeIdentifier accepts non-unicode. NEEDED WHY? TEST CASE MISSING test.test_{banana,jelly,pb,pbfailure,persisted}: from __future__ test.test_pbfailure: make literal strings bytes test.test_pbfailure: replace StringIO with io.BytesIO spread.flavors: translate keys of received dict into nativeString. INCOMPLETE -- Wolfgang
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 3 Oct, 11:09 pm, wolfgang.kde@rohdewald.de wrote:
I now have a local git branch with about 70 commits, always rebased onto current trunk.
It makes me sad to learn you're carrying so many patches to Twisted. It might be useful to the project if you could share why this development was easier to do outside the tree rather than contributing it to the project as you developed it.
For testing? Probably not random - instead, comprehensive.
The coding standard only requires you to add tests for code you're changing. Python 3 mixes things up a bit though - by changing everything. Calling a module "Python 3 compatible" without changing it at all still changes lots of things how about it works. I think that before the Twisted project wants to call a particular module ported, we want it to have test coverage that can run on Python 3.
I'll let other people volunteer projects they know of - but I strongly suspect there are few or no such applications.
If twisted.spread were completely supported on Python 3 in only another year I'll be very impressed. :)
That's too bad.
Again, it seems unfortunate that you have all of this work ... somewhere. Somewhere only you (as far as I can tell) can see it. Somewhere only you can test it. Somewhere only you can work on it and get it contributed to Twisted. Jean-Paul
![](https://secure.gravatar.com/avatar/d57bac257873fa3c0993069a9cde47a5.jpg?s=120&d=mm&r=g)
Am Sonntag, 5. Oktober 2014, 13:48:12 schrieb exarkun@twistedmatrix.com:
This is just my personal style of development. A bit chaotic. Sometimes trial and error. And really a lot of git rebase -i. Whenever I find some necessary or helpful change that could be done before porting, put that at the top of the patch list and readjust everything. The methodic part comes last: Look at what I changed, rethink why that is really needed, and look for similar places I overlooked. Then generate tickets for things that seem ready. BTW what about Ticket 7628, news extension "port"? I could soon start feeding porting tickets but if this extension is useful, I guess it should be applied first.
I was afraid you'd say that. https://twistedmatrix.com/trac/wiki/Plan/Python3 does not explicitly say so, maybe this should be changed. I can do that, but then I have no write permission for the wiki.
I'll let other people volunteer projects they know of - but I strongly suspect there are few or no such applications.
That makes me wonder if I really should have used PB at all. But now I do and have no plans to change that.
I believe I saw some mailing list posts where many years ago somebody said he has spread running with PY3 but as it seems nothing of that got into trunk. Be assured that I want to avoid that. What is the preferred place for Twisted public repositories? No svn please, only git. But first I want to do some more cleaning and reshuffling, I cannot really do that anymore with commits already pushed to a public repository. Maybe 2 or 3 weeks. -- Wolfgang
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 04:23 pm, wolfgang.kde@rohdewald.de wrote:
Looks like Kevin is working on it. I'll review it when it comes up.
Ah. I didn't think anyone was paying any attention to the plan anymore (and I've run out of energy to try to get people to stick to it). For example, following the plan, I would have expected to see perhaps two patches to port the banana module: one to add the missing test coverage and one to make any and all changes necessary to get the test suite to pass on Python 3. (Note, I'm just pointing this out for the sake of clarity. I'm not complaining about the tickets/patches you recently contributed. As I said, I'm out of energy for that). As far as wiki access goes, I'm not clear on how those permissions work anymore. Perhaps someone else can chime in. If not, consider asking on #twisted-dev.
Sorry about this. For anyone who's reading and thinking about starting a project, I personally recommend you not use PB. It's almost certainly not the right tool to solve the problem you have (if you think your problem is so unique and challenging it needs a tool like PB, maybe it is! Feel free to start a discussion about the specifics on this list. Maybe you've got one of the rare problems where PB makes sense.)
Anywhere public. Collaboration can't take place when things are private.
Pushing something isn't a promise not to abandoning, screw it up, rebase it, delete it later, whatever. I think that the notion that before something is shared it has to be carefully arranged and re-arranged to make it look like it was developed by an all-knowing, all-programming super being. Real software doesn't get built one perfect, self-contained step at a time. The construction of real software is messy. Jean-Paul
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 08:28 pm, wolfgang.kde@rohdewald.de wrote:
That doesn't mean it's impossible. It just means you may also have to shuffle some files around.
But I can of course reduce the number of tickets related to porting and put more than one patchfile into them.
Note that my point isn't that we need to reduce the number of tickets related to porting or that there should be more than one patch to resolve any one ticket (there shouldn't be). The point is that the documented porting plan is to port module by module, not Python 3 incompatible change by Python 3 incompatible change. Jean-Paul
![](https://secure.gravatar.com/avatar/d57bac257873fa3c0993069a9cde47a5.jpg?s=120&d=mm&r=g)
there now is https://github.com/wrohdewald/twisted.git branch spread-py3-7598 Not all is finished, especially utf-8 wherever Python 3 allows it, pbfailure, and handling of wired exceptions. what should I do next to start getting this into trunk? Maybe - open a ticket for porting banana - create a branch with all commits for that - do a pull request I do not want to create one big patch because I think it is helpful to have more smaller commits. But I probably should wait with all that until ticket 7664 is merged. -- Wolfgang
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Yep, this is right. Creating a patch and attaching it to the ticket would also be useful.
- do a pull request
Opening a pull request is not a particularly useful part of this workflow :-). The ticket is sufficient. See <https://github.com/twisted/twisted/pull/46> or any pull request closed recently for more information.
I do not want to create one big patch because I think it is helpful to have more smaller commits.
But I probably should wait with all that until ticket 7664 is merged.
I should note that this is already in review. Anyone want to have a look? -g
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 3 Oct, 11:09 pm, wolfgang.kde@rohdewald.de wrote:
I now have a local git branch with about 70 commits, always rebased onto current trunk.
It makes me sad to learn you're carrying so many patches to Twisted. It might be useful to the project if you could share why this development was easier to do outside the tree rather than contributing it to the project as you developed it.
For testing? Probably not random - instead, comprehensive.
The coding standard only requires you to add tests for code you're changing. Python 3 mixes things up a bit though - by changing everything. Calling a module "Python 3 compatible" without changing it at all still changes lots of things how about it works. I think that before the Twisted project wants to call a particular module ported, we want it to have test coverage that can run on Python 3.
I'll let other people volunteer projects they know of - but I strongly suspect there are few or no such applications.
If twisted.spread were completely supported on Python 3 in only another year I'll be very impressed. :)
That's too bad.
Again, it seems unfortunate that you have all of this work ... somewhere. Somewhere only you (as far as I can tell) can see it. Somewhere only you can test it. Somewhere only you can work on it and get it contributed to Twisted. Jean-Paul
![](https://secure.gravatar.com/avatar/d57bac257873fa3c0993069a9cde47a5.jpg?s=120&d=mm&r=g)
Am Sonntag, 5. Oktober 2014, 13:48:12 schrieb exarkun@twistedmatrix.com:
This is just my personal style of development. A bit chaotic. Sometimes trial and error. And really a lot of git rebase -i. Whenever I find some necessary or helpful change that could be done before porting, put that at the top of the patch list and readjust everything. The methodic part comes last: Look at what I changed, rethink why that is really needed, and look for similar places I overlooked. Then generate tickets for things that seem ready. BTW what about Ticket 7628, news extension "port"? I could soon start feeding porting tickets but if this extension is useful, I guess it should be applied first.
I was afraid you'd say that. https://twistedmatrix.com/trac/wiki/Plan/Python3 does not explicitly say so, maybe this should be changed. I can do that, but then I have no write permission for the wiki.
I'll let other people volunteer projects they know of - but I strongly suspect there are few or no such applications.
That makes me wonder if I really should have used PB at all. But now I do and have no plans to change that.
I believe I saw some mailing list posts where many years ago somebody said he has spread running with PY3 but as it seems nothing of that got into trunk. Be assured that I want to avoid that. What is the preferred place for Twisted public repositories? No svn please, only git. But first I want to do some more cleaning and reshuffling, I cannot really do that anymore with commits already pushed to a public repository. Maybe 2 or 3 weeks. -- Wolfgang
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 04:23 pm, wolfgang.kde@rohdewald.de wrote:
Looks like Kevin is working on it. I'll review it when it comes up.
Ah. I didn't think anyone was paying any attention to the plan anymore (and I've run out of energy to try to get people to stick to it). For example, following the plan, I would have expected to see perhaps two patches to port the banana module: one to add the missing test coverage and one to make any and all changes necessary to get the test suite to pass on Python 3. (Note, I'm just pointing this out for the sake of clarity. I'm not complaining about the tickets/patches you recently contributed. As I said, I'm out of energy for that). As far as wiki access goes, I'm not clear on how those permissions work anymore. Perhaps someone else can chime in. If not, consider asking on #twisted-dev.
Sorry about this. For anyone who's reading and thinking about starting a project, I personally recommend you not use PB. It's almost certainly not the right tool to solve the problem you have (if you think your problem is so unique and challenging it needs a tool like PB, maybe it is! Feel free to start a discussion about the specifics on this list. Maybe you've got one of the rare problems where PB makes sense.)
Anywhere public. Collaboration can't take place when things are private.
Pushing something isn't a promise not to abandoning, screw it up, rebase it, delete it later, whatever. I think that the notion that before something is shared it has to be carefully arranged and re-arranged to make it look like it was developed by an all-knowing, all-programming super being. Real software doesn't get built one perfect, self-contained step at a time. The construction of real software is messy. Jean-Paul
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 08:28 pm, wolfgang.kde@rohdewald.de wrote:
That doesn't mean it's impossible. It just means you may also have to shuffle some files around.
But I can of course reduce the number of tickets related to porting and put more than one patchfile into them.
Note that my point isn't that we need to reduce the number of tickets related to porting or that there should be more than one patch to resolve any one ticket (there shouldn't be). The point is that the documented porting plan is to port module by module, not Python 3 incompatible change by Python 3 incompatible change. Jean-Paul
![](https://secure.gravatar.com/avatar/d57bac257873fa3c0993069a9cde47a5.jpg?s=120&d=mm&r=g)
there now is https://github.com/wrohdewald/twisted.git branch spread-py3-7598 Not all is finished, especially utf-8 wherever Python 3 allows it, pbfailure, and handling of wired exceptions. what should I do next to start getting this into trunk? Maybe - open a ticket for porting banana - create a branch with all commits for that - do a pull request I do not want to create one big patch because I think it is helpful to have more smaller commits. But I probably should wait with all that until ticket 7664 is merged. -- Wolfgang
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Yep, this is right. Creating a patch and attaching it to the ticket would also be useful.
- do a pull request
Opening a pull request is not a particularly useful part of this workflow :-). The ticket is sufficient. See <https://github.com/twisted/twisted/pull/46> or any pull request closed recently for more information.
I do not want to create one big patch because I think it is helpful to have more smaller commits.
But I probably should wait with all that until ticket 7664 is merged.
I should note that this is already in review. Anyone want to have a look? -g
participants (4)
-
exarkun@twistedmatrix.com
-
Glyph
-
Kevin Horn
-
Wolfgang Rohdewald