[Twisted-Python] Twisted for Python 3
Hello,
I'm announcing an experimental, work-in-progress port of Twisted to Python 3. The homepage, where you can find more detailed information, is at https://bitbucket.org/pitrou/t3k
First step is to have core functionality (basic infrastructure, the various reactors, transports, etc.) working. This is test-driven, since Twisted has mostly excellent coverage. We are pretty much there. The current status is:
The following tests pass:
All of twisted/internet/test/* All of twisted/trial/test/* All of twisted/python/test/* Some of twisted/test: including test_amp, test_application, test_banana, test_defer, test_defgen, test_internet, test_jelly, test_log, test_logfile, test_paths, test_pb, test_persisted, test_protocols, test_ssl, test_sslverify, test_tcp, test_threads, test_threadpool, test_twistd, test_udp, test_unix...
The following scripts are known to work:
trial twistd
Feedback and contributions welcome. If you are interested in helping, please go and read the link above, it will give you suggestions.
Regards
Antoine.
Woo! Exciting :)
I'm surprised trial works. Wasn't there still a problem left with trial related to unbound methods going away (and the replacing functions not having a reference to the class they belong to...)? Something in t.python that gets things based on their name, I thought... Oh well, if it works it works.
Thank you very much for your efforts, lvh
On Wed, 19 Oct 2011 14:10:03 +0200 Laurens Van Houtven _@lvh.cc wrote:
Woo! Exciting :)
I'm surprised trial works. Wasn't there still a problem left with trial related to unbound methods going away (and the replacing functions not having a reference to the class they belong to...)? Something in t.python that gets things based on their name, I thought...
Yes, there was. I added a small hack so that it works (and so that I can run individual test cases from the command line :-)).
Regards
Antoine.
On Wed, 2011-10-19 at 13:50 +0200, Antoine Pitrou wrote:
Hello,
I'm announcing an experimental, work-in-progress port of Twisted to Python 3. The homepage, where you can find more detailed information, is at https://bitbucket.org/pitrou/t3k
Cool! Any patches you can submit to us that would make the code more Python 3 compatible, while not breaking Python 2.5 and later, would make your life easier going forward.
I notice you have banana getting extra op code for bytes... I think a better solution is to use existing 'string' op code for bytes, and rejecting unicode strings. What banana calls 'string' *is* bytes, it doesn't support unicode:
http://twistedmatrix.com/documents/current/core/specifications/banana.html
In general, anywhere existing Twisted code uses a Python 2 normal string, you should have the Python 3 version only accept bytes.
On Wed, 2011-10-19 at 09:25 -0400, Itamar Turner-Trauring wrote:
I notice you have banana getting extra op code for bytes... I think a better solution is to use existing 'string' op code for bytes, and rejecting unicode strings.
Just to emphasize this: the current solution of having a new opcode for bytes makes Perspective Broker on Python 3 incompatible with PB on Python 2.
On Wed, 19 Oct 2011 09:41:36 -0400 Itamar Turner-Trauring itamar@itamarst.org wrote:
On Wed, 2011-10-19 at 09:25 -0400, Itamar Turner-Trauring wrote:
I notice you have banana getting extra op code for bytes... I think a better solution is to use existing 'string' op code for bytes, and rejecting unicode strings.
Just to emphasize this: the current solution of having a new opcode for bytes makes Perspective Broker on Python 3 incompatible with PB on Python 2.
Or rather, they're compatible until you use Python 3 bytes objects.
(and I think any RPC system will have the same issues between 2.x and 3.x, regardless of whether it's based on pickle, json or even protocol buffers)
This probably deserves more thinking. I took the most natural approach for Python 3. The solution you are proposing (not supporting the 3.x "str" type) would be, IMHO, worse, because the str type is fundamental and basically no idiomatic Python 3 code could work.
Regards
Antoine.
On Wed, 19 Oct 2011 09:25:14 -0400 Itamar Turner-Trauring itamar@itamarst.org wrote:
Cool! Any patches you can submit to us that would make the code more Python 3 compatible, while not breaking Python 2.5 and later, would make your life easier going forward.
Yes, I've started doing that. However, the extent of these patches is quite limited. Most changes I have to make have to do with introducing bytes literals, or fixing specific issues under Python 3. For example, I could post the following patch for t.p.failure, but would you accept it?
@@ -464,6 +468,14 @@ class Failure: # added 2003-06-23. See comment above in __init__ c['tb'] = None
+ try: + # Clear other references held by exception objects + c['value'].__cause__ = None + c['value'].__context__ = None + c['value'].__traceback__ = None + except AttributeError: + pass + if self.stack is not None: # XXX: This is a band-aid. I can't figure out where these # (failure.stack is None) instances are coming from.
I notice you have banana getting extra op code for bytes... I think a better solution is to use existing 'string' op code for bytes, and rejecting unicode strings. What banana calls 'string' *is* bytes, it doesn't support unicode:
Indeed, this is debatable. But generally, marshalling libraries have to evolve a bit when ported to Python 3. There's no way the semantic gap between 2.x and 3.x can be ignored.
I think banana in 3.x *has* to support both types. str is a fundamental type in 3.x (think that class names, function names, filenames, etc. all are str by default) and not supporting it in a marshalling library would be pretty much awkward. bytes is fundamental in a networking library and not supporting it would be similarly awkward :)
Moreover, 3.x has a natural preference for unicode strings while 2.x has a natural preference for 8-bit strings. Where high-level data is handled, Python 3 users will expect to use str object, not bytes objects, for text data (if you marshal e.g. a class name, you don't expect it to come out as bytes... and neither would Perspective Broker, I guess). Str objects also have more features, which make them more amenable to parsing and formatting.
That's why I think translating the "string" type to 3.x str objects is what users would expect in practice. This is also the decision taken in the standard pickle module: if you unpickle a 2.x pickle containing str objects, you get 3.x str objects (not bytes).
In general, anywhere existing Twisted code uses a Python 2 normal string, you should have the Python 3 version only accept bytes.
Due to the aforementioned differences, I don't think that's as simple. For example, filenames are pretty much str (unicode) by default in Python 3, even though you can also use bytes (but it's discouraged under Windows). Command-line arguments are also str, as are many other things. And I'll think you'll agree that it's more natural to ask for the HTTP "GET" method than for the b"GET" method.
(no, I haven't started on twisted.web yet)
Regards
Antoine.
On 01:49 pm, solipsis@pitrou.net wrote:
On Wed, 19 Oct 2011 09:25:14 -0400 Itamar Turner-Trauring itamar@itamarst.org wrote:
[snip]
Indeed, this is debatable. But generally, marshalling libraries have to evolve a bit when ported to Python 3. There's no way the semantic gap between 2.x and 3.x can be ignored.
I think banana in 3.x *has* to support both types. str is a fundamental type in 3.x (think that class names, function names, filenames, etc. all are str by default) and not supporting it in a marshalling library would be pretty much awkward. bytes is fundamental in a networking library and not supporting it would be similarly awkward :)
Unicode is handled by Jelly. The only thing Banana does is turn objects into bytes to put into a socket and vice versa. Text in Python 3 will be handled the same way as text in Python 2: jelly will UTF-8 encode it before handing it to banana.
Jean-Paul
On Wed, 19 Oct 2011 14:29:17 -0000 exarkun@twistedmatrix.com wrote:
On 01:49 pm, solipsis@pitrou.net wrote:
On Wed, 19 Oct 2011 09:25:14 -0400 Itamar Turner-Trauring itamar@itamarst.org wrote:
[snip]
Indeed, this is debatable. But generally, marshalling libraries have to evolve a bit when ported to Python 3. There's no way the semantic gap between 2.x and 3.x can be ignored.
I think banana in 3.x *has* to support both types. str is a fundamental type in 3.x (think that class names, function names, filenames, etc. all are str by default) and not supporting it in a marshalling library would be pretty much awkward. bytes is fundamental in a networking library and not supporting it would be similarly awkward :)
Unicode is handled by Jelly. The only thing Banana does is turn objects into bytes to put into a socket and vice versa. Text in Python 3 will be handled the same way as text in Python 2: jelly will UTF-8 encode it before handing it to banana.
Hmm, ok, I will take another look then.
Regards
Antoine.
More generally, any place Twisted currently uses strings (i.e. byte strings, as they all are in Python 2), the default should be bytes in Python 3.x. When we switch to a minimum version of Python that has bytes support we should start being more explicit about this and using the b"" syntax to emphasize this.
For example: Protocol.dataReceived, transport.write, protocol constants... all should be bytes.
There are some places that probably should be unicode like FilePath, although perhaps that's easier if we merge the unicode support branch. There might be other places where a high-level API might choose to accept unicode to make things easier (e.g. "GET" for web requests, as you said). But in general, bytes should be the default because bytes is what network programming is about.
On Wed, 19 Oct 2011 11:36:59 -0400 "Itamar Turner-Trauring" itamar@itamarst.org wrote:
More generally, any place Twisted currently uses strings (i.e. byte strings, as they all are in Python 2), the default should be bytes in Python 3.x.
As I already answered, it is not as simple. Idiomatic use of bytes/unicode is not the same in 2.x as it is in 3.x.
For example: Protocol.dataReceived, transport.write, protocol constants...
Agreed. But we are talking about data decoded by a protocol here.
Regards
Antoine.
Agreed. But we are talking about data decoded by a protocol here.
Banana is very low-level in the PB stack, and is explicitly designed to send bytes, not unicode. It's Jelly that does unicode, and even there you shouldn't need to change anything.
If the protocol decodes to bytes in Python 2, you should keep it as bytes. E.g. twisted.web.client.getPage result should fire with bytes, not unicode (downloading a JPEG and then decoding it to unicode isn't that useful). For many old protocol it's not even possible to know what encoding will work, and potentially it might use multiple different encodings in different situations, and if you just pick UTF-8 you'll make the library unusable for some people.
On Wed, 2011-10-19 at 15:49 +0200, Antoine Pitrou wrote:
I think banana in 3.x *has* to support both types. str is a fundamental type in 3.x (think that class names, function names, filenames, etc. all are str by default) and not supporting it in a marshalling library would be pretty much awkward. bytes is fundamental in a networking library and not supporting it would be similarly awkward :)
I'm pretty sure we did unicode support on the Jelly level, *above* Banana. If we didn't, then we should. But that's the natural place to do Unicode, not in the low-level protocol. So I don't think banana should change.
In any case, I'd rather developers have to do a little bit of extra work being compatible across Python versions (being careful about bytes vs. strings), as opposed to the current solution, which makes it impossible to be compatible because it's essentially a different protocol.
On Wed, 2011-10-19 at 10:37 -0400, Itamar Turner-Trauring wrote:
In any case, I'd rather developers have to do a little bit of extra work being compatible across Python versions (being careful about bytes vs. strings), as opposed to the current solution, which makes it impossible to be compatible because it's essentially a different protocol.
This is what I had to do for the Java version of Banana, BTW (use bytearrays, even though Strings are easier) in order to be compatible with the Python 2 version.
On Wed, 2011-10-19 at 15:49 +0200, Antoine Pitrou wrote:
This is also the decision taken in the standard pickle module: if you unpickle a 2.x pickle containing str objects, you get 3.x str objects (not bytes).
That decision is wrong. It's corrupting users' data, though arguably it's your own fault if you actually rely on pickle.
On Wed, 19 Oct 2011 10:43:21 -0400 Itamar Turner-Trauring itamar@itamarst.org wrote:
On Wed, 2011-10-19 at 15:49 +0200, Antoine Pitrou wrote:
This is also the decision taken in the standard pickle module: if you unpickle a 2.x pickle containing str objects, you get 3.x str objects (not bytes).
That decision is wrong. It's corrupting users' data, though arguably it's your own fault if you actually rely on pickle.
No matter how you decide to handle the transition, there will be incompatibilities when transitioning between different data models. Relying on a pickle is no different than relying on a JSON tree, an AMP box or anything else.
Mapping the natural 2.x string type to the natural 3.x string type was a reasonable decision in that context.
Regards
Antoine.
For example, I could post the following patch for t.p.failure, but would you accept it?
@@ -464,6 +468,14 @@ class Failure: # added 2003-06-23. See comment above in __init__ c['tb'] = None
try:# Clear other references held by exception objectsc['value'].__cause__ = Nonec['value'].__context__ = Nonec['value'].__traceback__ = Noneexcept AttributeError:passif self.stack is not None: # XXX: This is a band-aid. I can't figure out where these # (failure.stack is None) instances are coming from.
That sort of patch would probably be fine, though I'd much rather that be inside an if statement that checked the Python version rather than inside a try/except. Faster, and much clearer what's going on.
On Wed, 19 Oct 2011 13:10:22 -0400 "Itamar Turner-Trauring" itamar@itamarst.org wrote:
For example, I could post the following patch for t.p.failure, but would you accept it?
@@ -464,6 +468,14 @@ class Failure: # added 2003-06-23. See comment above in __init__ c['tb'] = None
try:# Clear other references held by exception objectsc['value'].__cause__ = Nonec['value'].__context__ = Nonec['value'].__traceback__ = Noneexcept AttributeError:passif self.stack is not None: # XXX: This is a band-aid. I can't figure out where these # (failure.stack is None) instances are coming from.That sort of patch would probably be fine, though I'd much rather that be inside an if statement that checked the Python version rather than inside a try/except. Faster, and much clearer what's going on.
The try/except is necessary because Failures sometimes wrap something else than an exception, and where setting an arbitrary attribute is not allowed.
Regards
Antoine.
On Oct 19, 2011, at 1:22 PM, Antoine Pitrou wrote:
On Wed, 19 Oct 2011 13:10:22 -0400 "Itamar Turner-Trauring" itamar@itamarst.org wrote:
For example, I could post the following patch for t.p.failure, but would you accept it?
@@ -464,6 +468,14 @@ class Failure: # added 2003-06-23. See comment above in __init__ c['tb'] = None
try:# Clear other references held by exception objectsc['value'].__cause__ = Nonec['value'].__context__ = Nonec['value'].__traceback__ = Noneexcept AttributeError:passif self.stack is not None: # XXX: This is a band-aid. I can't figure out where these # (failure.stack is None) instances are coming from.That sort of patch would probably be fine, though I'd much rather that be inside an if statement that checked the Python version rather than inside a try/except. Faster, and much clearer what's going on.
The try/except is necessary because Failures sometimes wrap something else than an exception, and where setting an arbitrary attribute is not allowed.
This patch would definitely need to be accepted if you expect Twisted to eventually support Python 3. Why do you believe it wouldn't be accepted?
Of course it would need to have some test coverage - but a first step there is to have a buildbot actually running the tests on py3k.
Also: if Failures wrap something other than an Exception, that's a bug. in Failure.__doc__, it says: "@ivar value: The exception instance responsible for this failure." If there are cases where non-Exceptions are passed here, then perhaps that behavior should be deprecated before it raises an exception, but something needs to be adjusted so that the documentation is consistent with the implementation.
-glyph
On Oct 19, 2011, at 7:50 AM, Antoine Pitrou wrote:
Feedback and contributions welcome. If you are interested in helping, please go and read the link above, it will give you suggestions.
Antoine, I'm very happy that you've decided to help Twisted get going on Py3. I see that you've already submitted several of your fixes to the Twisted tracker. Thanks for that. We'll try (as always) to get those reviewed soon. We do eventually want to have 3.x support in trunk, so anything you're doing to support that goal should be an acceptable patch - provided it meets our other quality criteria, of course, but there's no patch for py3k support which shouldn't be able to meet that criteria :).
I would suggest that you submit all patches directly to the Twisted tracker, where they can be properly reviewed before landing, not to this (or any other) fork. As several other messages in this thread have indicated, it looks like you may be misunderstanding a few key aspects of Twisted (such as how the distinction between bytes and unicode needs to be treated in the Banana protocol), so this fork is going to get some things wrong, potentially with wire-level implications and incompatibilities. Review discussions of targeted patches will allow us to address those areas individually, while getting the other parts merged into the mainline.
(This goes double for other people who are not Antoine wanting to help out: please look first at this list of tickets: http://twistedmatrix.com/trac/query?status=assigned&status=new&status=reopened&group=status&milestone=Python-3.x, and help them get through the review process, rather than submitting patches to a fork.)
Most importantly, it will not be reasonable to merge in a humongous branch with hundreds of unrelated changes later; doing code review on changes that big just isn't feasible and chances are good that they will linger forever. If your fork _is_ made to work correctly, folding that correct behavior back in with all the necessary test coverage and fixes will be a much larger job than doing it the right way in the first place, one bit at a time, in Twisted trunk. We have had several bad experiences with this kind of development before and I would rather not repeat them.
Finally, having the code in mainline will allow it to be subjected to our correctness testing http://buildbot.twistedmatrix.com/ and also our performance testing http://speed.twistedmatrix.com/, neither of which is a small effort.
If this is just a short-lived experimental test bed to try things out quickly (i.e. a spike to demonstrate that it's feasible to get Twisted on py3), then that's fine, but I am somewhat concerned that impatient users will actually adopt this fork, and we will have to spend a lot of energy telling people (A) "don't use Antoine's fork, it's broken", and (B) "please stop reporting bugs in this broken fork on our bug tracker".
At any rate, the very first step here should be to add an as-yet-unsupported py3 builder to our build farm. This should be pretty straightforward, as new builders are added all the time, and we have some new hardware that could be put to use for this.
Alternatively, we do already have a py3k-warnings buildbot set up; you can see the current warnings here: http://buildbot.twistedmatrix.com/builders/python-3k-warnings/builds/111/steps/trial/logs/stdio, although possibly this needs to be updated to python 2.7 to get rid of irrelevant noise like warnings about callable() going away, since I believe it's come back to 3.x now.
We might also be able to help with your problem of testing things on Windows, as we do have several Windows test machines set up already. (Is pywin32 supported on 3.x yet? http://pypi.python.org/pypi/pywin32/ doesn't indicate.)
As you can see on http://buildbot.twistedmatrix.com/builders, we already even have some buildbots running python 3 builds for PyOpenSSL, so this isn't even really anything new.
Would anyone like to volunteer to help out with Antoine's efforts on the build-infrastructure side of things?
Hi Glyph,
On Wed, 19 Oct 2011 14:50:41 -0400 Glyph glyph@twistedmatrix.com wrote:
On Oct 19, 2011, at 7:50 AM, Antoine Pitrou wrote:
Feedback and contributions welcome. If you are interested in helping, please go and read the link above, it will give you suggestions.
We do eventually want to have 3.x support in trunk, so anything you're doing to support that goal should be an acceptable patch - provided it meets our other quality criteria, of course, but there's no patch for py3k support which shouldn't be able to meet that criteria :).
To me, 3.x support in trunk is not really workable unless you drop pre-2.6 compatibility. Bytes literals are pretty much required. A possible workaround is to call a factory function that makes an str into a bytes in 3.x (e.g. write B("xyz") everywhere instead of b"xyz"), but that's quite ugly IMO, not to mention suboptimal.
I would suggest that you submit all patches directly to the Twisted tracker, where they can be properly reviewed before landing, not to this (or any other) fork. As several other messages in this thread have indicated, it looks like you may be misunderstanding a few key aspects of Twisted (such as how the distinction between bytes and unicode needs to be treated in the Banana protocol), so this fork is going to get some things wrong, potentially with wire-level implications and incompatibilities.
Incompatibilities are a given when you switch between two different data models. The incompatibilities don't have to be wire-level, but they *will* occur at the higher level anyway.
You may read my answers to Itamar for more explanation on that decision. Of course, it is not set in stone.
Most importantly, it will not be reasonable to merge in a humongous branch with hundreds of unrelated changes later; doing code review on changes that big just isn't feasible and chances are good that they will linger forever. If your fork _is_ made to work correctly, folding that correct behavior back in with all the necessary test coverage and fixes will be a much larger job than doing it the right way in the first place, one bit at a time, in Twisted trunk.
Without wanting to sound unconstructive, the "right way" (the "byte-sized" approach) does not sound workable. Some 3.x-related patches (not mine) have been lingering for years, although they are trivial. And these patches (mine and the others) are really the tip of iceberg.
To clarify, here's my current set of modifications post-2to3 (that is, it does *not* include 2to3's own changes):
.hgignore | 7 + bin/manhole | 2 +- bin/mktap | 2 +- bin/pyhtmlizer | 2 +- bin/tap2deb | 2 +- bin/tap2rpm | 2 +- bin/tapconvert | 2 +- bin/trial | 2 +- bin/twistd | 2 +- doc/core/examples/chatserver.py | 2 +- doc/core/examples/echoclient.py | 6 +- doc/core/examples/echoclient_ssl.py | 6 +- doc/core/examples/echoclient_udp.py | 6 +- doc/core/examples/simpleclient.py | 2 +- setup.py | 6 +- twisted/application/internet.py | 4 +- twisted/enterprise/adbapi.py | 8 + twisted/internet/_sigchld.c | 15 +- twisted/internet/_signals.py | 2 +- twisted/internet/_sslverify.py | 9 +- twisted/internet/abstract.py | 10 +- twisted/internet/address.py | 9 +- twisted/internet/base.py | 14 +- twisted/internet/defer.py | 13 + twisted/internet/error.py | 4 +- twisted/internet/inotify.py | 4 +- twisted/internet/pollreactor.py | 2 +- twisted/internet/posixbase.py | 4 +- twisted/internet/process.py | 8 +- twisted/internet/protocol.py | 2 +- twisted/internet/task.py | 2 +- twisted/internet/tcp.py | 7 +- twisted/internet/test/connectionmixins.py | 10 +- twisted/internet/test/fakeendpoint.py | 5 +- twisted/internet/test/test_fdset.py | 18 +- twisted/internet/test/test_inotify.py | 6 +- twisted/internet/test/test_posixprocess.py | 2 +- twisted/internet/test/test_process.py | 33 +- twisted/internet/test/test_tcp.py | 22 +- twisted/internet/test/test_udp.py | 2 +- twisted/internet/test/test_unix.py | 2 +- twisted/internet/threads.py | 6 +- twisted/internet/unix.py | 5 +- twisted/internet/utils.py | 13 +- twisted/lore/latex.py | 2 +- twisted/lore/lint.py | 4 +- twisted/lore/tree.py | 21 +- twisted/manhole/explorer.py | 25 +- twisted/persisted/aot.py | 146 +++--- twisted/persisted/crefutil.py | 2 +- twisted/persisted/sob.py | 2 +- twisted/persisted/styles.py | 8 +- twisted/plugin.py | 5 +- twisted/protocols/amp.py | 210 +++++----- twisted/protocols/basic.py | 43 +- twisted/protocols/finger.py | 18 +- twisted/protocols/ftp.py | 4 +- twisted/protocols/ident.py | 41 +- twisted/protocols/loopback.py | 10 +- twisted/protocols/memcache.py | 47 +- twisted/protocols/policies.py | 4 +- twisted/protocols/postfix.py | 10 +- twisted/protocols/socks.py | 22 +- twisted/protocols/stateful.py | 7 +- twisted/protocols/test/test_tls.py | 73 +- twisted/protocols/tls.py | 2 +- twisted/protocols/wire.py | 10 +- twisted/python/_initgroups.c | 19 +- twisted/python/_inotify.py | 3 +- twisted/python/_release.py | 10 +- twisted/python/failure.py | 25 +- twisted/python/filepath.py | 44 ++- twisted/python/hook.py | 4 +- twisted/python/htmlizer.py | 5 +- twisted/python/log.py | 4 - twisted/python/logfile.py | 8 +- twisted/python/randbytes.py | 11 +- twisted/python/reflect.py | 6 +- twisted/python/runtime.py | 2 +- twisted/python/test/modules_helpers.py | 2 +- twisted/python/test/test_deprecate.py | 6 +- twisted/python/test/test_dist.py | 10 +- twisted/python/test/test_hashlib.py | 40 +- twisted/python/test/test_htmlizer.py | 18 +- twisted/python/test/test_release.py | 470 ++++++++++++----------- twisted/python/test/test_util.py | 11 +- twisted/python/test/test_versions.py | 19 +- twisted/python/test/test_zipstream.py | 116 +++-- twisted/python/text.py | 24 +- twisted/python/usage.py | 12 +- twisted/python/util.py | 74 +--- twisted/python/versions.py | 51 +- twisted/python/win32.py | 6 +- twisted/python/zippath.py | 22 +- twisted/python/zipstream.py | 28 +- twisted/runner/portmap.c | 18 +- twisted/scripts/_twistd_unix.py | 2 +- twisted/scripts/trial.py | 4 +- twisted/spread/banana.py | 68 ++- twisted/spread/flavors.py | 38 + twisted/spread/jelly.py | 112 +---- twisted/spread/pb.py | 35 +- twisted/spread/publish.py | 4 +- twisted/test/iosim.py | 4 +- twisted/test/proto_helpers.py | 10 +- twisted/test/test_adbapi.py | 6 +- twisted/test/test_amp.py | 330 ++++++++-------- twisted/test/test_application.py | 10 +- twisted/test/test_banana.py | 64 +- twisted/test/test_defgen.py | 2 +- twisted/test/test_epoll.py | 4 +- twisted/test/test_failure.py | 4 +- twisted/test/test_fdesc.py | 26 +- twisted/test/test_finger.py | 16 +- twisted/test/test_ident.py | 42 +- twisted/test/test_internet.py | 28 +- twisted/test/test_iutils.py | 18 +- twisted/test/test_jelly.py | 44 +- twisted/test/test_log.py | 22 +- twisted/test/test_logfile.py | 6 +- twisted/test/test_loopback.py | 64 +- twisted/test/test_memcache.py | 194 ++++---- twisted/test/test_modules.py | 26 +- twisted/test/test_newcred.py | 24 +- twisted/test/test_paths.py | 70 +- twisted/test/test_pb.py | 18 +- twisted/test/test_pbfailure.py | 4 +- twisted/test/test_persisted.py | 14 +- twisted/test/test_policies.py | 49 +- twisted/test/test_postfix.py | 20 +- twisted/test/test_protocols.py | 290 +++++++------- twisted/test/test_socks.py | 112 ++-- twisted/test/test_ssl.py | 28 +- twisted/test/test_sslverify.py | 2 +- twisted/test/test_stateful.py | 8 +- twisted/test/test_tcp.py | 34 +- twisted/test/test_tcp_internals.py | 5 + twisted/test/test_threadable.py | 2 +- twisted/test/test_threads.py | 6 +- twisted/test/test_tpfile.py | 8 +- twisted/test/test_twistd.py | 14 +- twisted/test/test_udp.py | 40 +- twisted/test/test_unix.py | 8 +- twisted/trial/runner.py | 41 +- twisted/trial/test/packages.py | 2 +- twisted/trial/test/test_assertions.py | 2 - twisted/trial/test/test_loader.py | 21 +- twisted/trial/test/test_reporter.py | 26 +- twisted/trial/test/test_tests.py | 15 +- twisted/trial/test/test_util.py | 1 + twisted/trial/test/test_warning.py | 8 +- twisted/trial/unittest.py | 3 + twisted/trial/util.py | 2 +- twisted/web/microdom.py | 5 +- 154 files changed, 2171 insertions(+), 1970 deletions(-)
*Assuming* we find a clean solution to the bytes literal problem, I could try to slice all of that into "byte-sized" patches into which I inject version-checking boilerplate, but that would mean a large waste of time and energy: - for me, as I post dozens of small patches and have to follow up on them, and wait for them to be checked in - for you, as you have to review these patches without perhaps even being Python 3 users yourselves, and without Python 3 compatibility being on your priority list
It would also mean neither of us would be able to remain motivated (which, in the end, is the fundamental problem).
Therefore, my work assumes a different approach. I expect 3.x support for Twisted to need maintenance of a dedicated branch for a non-ephemeral lapse of time. One goal of my experiment is to find out how easy or tedious such maintenance is.
Finally, having the code in mainline will allow it to be subjected to our correctness testing http://buildbot.twistedmatrix.com/ and also our performance testing http://speed.twistedmatrix.com/, neither of which is a small effort.
Setting up buildbots would be very nice indeed. But isn't it independent from having the code live in "trunk"? As for performance testing, it's probably prematurate right now :-)
At any rate, the very first step here should be to add an as-yet-unsupported py3 builder to our build farm. This should be pretty straightforward, as new builders are added all the time, and we have some new hardware that could be put to use for this.
Alternatively, we do already have a py3k-warnings buildbot set up; you can see the current warnings here: http://buildbot.twistedmatrix.com/builders/python-3k-warnings/builds/111/steps/trial/logs/stdio, although possibly this needs to be updated to python 2.7 to get rid of irrelevant noise like warnings about callable() going away, since I believe it's come back to 3.x now.
As I said, getting rid of py3k warnings is really the tip of the iceberg. It's a nice first step, but it's neither necessary (if you plan to maintain a 3.x branch), and it doesn't take you very far away.
(yes, callable() has come back in 3.2)
Regards
Antoine.
To me, 3.x support in trunk is not really workable unless you drop pre-2.6 compatibility. Bytes literals are pretty much required. A possible workaround is to call a factory function that makes an str into a bytes in 3.x (e.g. write B("xyz") everywhere instead of b"xyz"), but that's quite ugly IMO, not to mention suboptimal.
We're getting closer; we're going to drop Python 2.4 as soon as the next release is out. Still not up to 2.6, though.
Incompatibilities are a given when you switch between two different data models. The incompatibilities don't have to be wire-level, but they *will* occur at the higher level anyway.
The thing that worries is me is unnecessary or harmful incompatibilities that are a result of misunderstanding (e.g. the banana thing, which was an unnecessary wire protocol change). Submitting patches would have the benefit of letting someone else help you make these decisions; it's unreasonable to expect you to become an expert on every single API in Twisted.
Without wanting to sound unconstructive, the "right way" (the "byte-sized" approach) does not sound workable. Some 3.x-related patches (not mine) have been lingering for years, although they are trivial. And these patches (mine and the others) are really the tip of iceberg.
There aren't any python 3 patches in the review queue at the moment; do you have any specific tickets in mind? If they're ready to go they should have the "review" keyword set (if they haven't addressed review comments then that is why they aren't merged).
*Assuming* we find a clean solution to the bytes literal problem, I could try to slice all of that into "byte-sized" patches into which I inject version-checking boilerplate, but that would mean a large waste of time and energy:
- for me, as I post dozens of small patches and have to follow up on them, and wait for them to be checked in
- for you, as you have to review these patches without perhaps even being Python 3 users yourselves, and without Python 3 compatibility being on your priority list
Most of us aren't Windows users either, but we still try to support Windows :)
On Oct 19, 2011, at 4:28 PM, Itamar Turner-Trauring wrote:
To me, 3.x support in trunk is not really workable unless you drop pre-2.6 compatibility. Bytes literals are pretty much required. A possible workaround is to call a factory function that makes an str into a bytes in 3.x (e.g. write B("xyz") everywhere instead of b"xyz"), but that's quite ugly IMO, not to mention suboptimal.
We're getting closer; we're going to drop Python 2.4 as soon as the next release is out. Still not up to 2.6, though.
For one thing:
What's the big problem with B("")? It does not strike me as particularly problematic. We call plenty of functions at import time, I'm sure that it's not going to impact performance much. Plus, it's very easy to fix with a trivial regex when we do drop 2.5 support eventually: the expression evaluates to the same thing. If anything, concern about the function call overhead will simply encourage us to make more constant bytestrings module-level constants instead of inline expressions, which is a plus if you ask me :).
Incompatibilities are a given when you switch between two different data models. The incompatibilities don't have to be wire-level, but they *will* occur at the higher level anyway.
The thing that worries is me is unnecessary or harmful incompatibilities that are a result of misunderstanding (e.g. the banana thing, which was an unnecessary wire protocol change). Submitting patches would have the benefit of letting someone else help you make these decisions; it's unreasonable to expect you to become an expert on every single API in Twisted.
Nothing to say to this besides a big "+1". This is the whole point of the review process: to make sure that appropriate knowledge is shared before committing to a change.
Without wanting to sound unconstructive, the "right way" (the "byte-sized" approach) does not sound workable. Some 3.x-related patches (not mine) have been lingering for years, although they are trivial. And these patches (mine and the others) are really the tip of iceberg.
There aren't any python 3 patches in the review queue at the moment; do you have any specific tickets in mind? If they're ready to go they should have the "review" keyword set (if they haven't addressed review comments then that is why they aren't merged).
I should note also that our review queue has gotten to 0 tickets several times since those tickets have last been in review, so things are getting through.
The problem here is that nobody is putting in the consistent work necessary to get these patches through review and merged to trunk. Almost all the py3k patches have been dropped off and then abandoned. The solution is not to give up and abandon all the patches together, but rather to find some people to get involved and continue participating in that work.
(That's not to say that I don't appreciate the effort involved in dropping those patches off in the first place. I do. It also doesn't mean that people shouldn't drop off patches if they're not going to finish them; they definitely should. The original author doesn't necessarily have to be the one to do the responding to feedback.)
It may seem easier right now to put them into a big pile and forget about that process, but what happens once trunk has moved on? We don't make a lot of incompatible changes, but that doesn't mean we don't change the implementation of things quite a bit. Lots of work has gone on in core areas of Twisted recently - including a near-total rewrite of TLS support - and many more such changes are coming. The relative stability of Twisted as an API to depend on says nothing about the stability of the code in terms of diffs continuing to apply; and the larger the diff, the more likely it's going to run into conflicts.
It's worth the extra effort to get the code into the mainline because when the work is done, it's actually done: we continue to maintain that code moving forward, and assuming that there is an appropriate buildslave (possibly one with the 'rachet'-style reporting that exarkun just set up for pydoctor?), we make sure that future changes won't undo the progress that has been made and introduce more py3k warnings or test failures.
If the code isn't in the mainline, then instead of supporting your efforts, all future maintenance undermines them. You'll have to put in lots of extra work to keep your fork applying, and you'll have to do a bunch of work because any changes will not take the py3 compatibility tests into account, so every new module will be a new thing you have to clean up. Plus you don't get the benefit of having each change tested extensively to make sure it doesn't break anything unexpected, on some weird platform or configuration.
This work on the build infrastructure and release management and QA appears to be invisible from the perspective of an individual change, but in reality it's the plurality, if not the majority, of the work that goes into the project.
*Assuming* we find a clean solution to the bytes literal problem,
We've already got one, except perhaps for an unreasonably strict definition of "clean" :-).
I could try to slice all of that into "byte-sized" patches into which I inject version-checking boilerplate, but that would mean a large waste of time and energy:
- for me, as I post dozens of small patches and have to follow up on
them, and wait for them to be checked in
You can choose the appropriate size of patches; you don't have to work on a per-file basis; it would make more sense to choose a topic area than to just submit individual one-liners as patches because that's what happens to be in a single file.
- for you, as you have to review these patches without perhaps even
being Python 3 users yourselves, and without Python 3 compatibility being on your priority list
Most of us aren't Windows users either, but we still try to support Windows :)
Also, the issue is not the priority of python 3 support, but the priority of reviewing incoming patches, which is always at the absolute top of our priority list. As you can see from the scoring system on http://twistedmatrix.com/highscores/, we value it more highly than anything else.
Patches to Twisted trunk are also good for things that'll make 2to3 output work with less manual fixes (e.g. renaming variables named "callable" to something else).
On Wed, 19 Oct 2011 17:46:41 -0400 "Itamar Turner-Trauring" itamar@itamarst.org wrote:
Patches to Twisted trunk are also good for things that'll make 2to3 output work with less manual fixes (e.g. renaming variables named "callable" to something else).
Ah, good example indeed. I take it you had already tried a 2to3 run? (I was quite baffled by this one)
I think I'm going to follow a two-pronged approach right now: - contribute some patches to trunk for 2to3/3.x-friendliness - continue work on the t3k branch, because a working result is more interesting even if some of it has to be amended later
I'm still split about the B("") thing. Do note that a simple regex is not enough, because of string continuations.
Regards
Antoine.
On 05:23 pm, solipsis@pitrou.net wrote:
On Wed, 19 Oct 2011 17:46:41 -0400 "Itamar Turner-Trauring" itamar@itamarst.org wrote:
Patches to Twisted trunk are also good for things that'll make 2to3 output work with less manual fixes (e.g. renaming variables named "callable" to something else).
Ah, good example indeed. I take it you had already tried a 2to3 run? (I was quite baffled by this one)
I think I'm going to follow a two-pronged approach right now:
- contribute some patches to trunk for 2to3/3.x-friendliness
Thanks. :)
Speaking of this, it'd be great to have some help from some other developers who are interested in Python 3 support. There are now four of Antoine's tickets up for review (see http://twistedmatrix.com/trac/report/15) - reviewing one of them would be a great way to help with the Python 3 effort.
If you're interested and you've never done a review before, take a look at http://twistedmatrix.com/trac/wiki/ReviewProcess and then get started! You'll also get yourself onto http://twistedmatrix.com/highscores. :)
Jean-Paul
- continue work on the t3k branch, because a working result is
more interesting even if some of it has to be amended later
I'm still split about the B("") thing. Do note that a simple regex is not enough, because of string continuations.
Regards
Antoine.
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, Oct 19, 2011 at 8:50 AM, Antoine Pitrou solipsis@pitrou.net wrote:
I'm announcing an experimental, work-in-progress port of Twisted to Python 3. The homepage, where you can find more detailed information, is at https://bitbucket.org/pitrou/t3k
Here's something similar from my side:
lp:~facundo/+junk/twisted-py3k/
All testing infrastructure works, running of course the 2to3 inside the project.
A lot of tests fail, of course... I found that there're a lot of code that you *need* to know the protocols and such to be able to convert them...
Thanks for your work!
participants (6)
-
Antoine Pitrou -
exarkun＠twistedmatrix.com -
Facundo Batista -
Glyph -
Itamar Turner-Trauring -
Laurens Van Houtven