[Twisted-Python] Twisted in Python STDLIB?
I'd like to start a discussion on whether Twisted ought to be a part of Python's standard library. I understand some of the developers think it should not, but I personally don't like seeing far inferior networking code sitting there in stdlib. Inclusion in the stdlib carries an implied endorsement, and that endorsement should be going to Twisted, in my view. Has Guido et al. ever expressed any viewpoint on Twisted? It is certainly very Pythonic, and though I have only limited Python and OOP experience to express this viewpoint, it is the best-written Python code I've ever seen. (Now, if only I could say the same for the commenting...) Ed Suominen Registered Patent Agent Open-Source Software Author (yes, both...) Web Site: http://www.eepatents.com
On Mon, 11 Oct 2004 11:34:34 -0700, Ed Suominen <general@eepatents.com> wrote:
Guido (and others, I think) have expressed dislike for framework-structured modules in the Python standard library. Since the majority of Twisted falls into this category, this is a point on which Guido would probably require some convincing. Aside from this, inclusion in the standard library imposes even stricter backwards compatibility requirements (rightly so). Twisted is changing less rapidly than it was at one point, but it is still making backwards incompatible changes and deprecating a lot of old APIs. There are very fundamental parts of Twisted that just aren't ready to be called stable and set in stone. Additionally, the relatively slow Python release process (one major release every 12 - 18 months) would be deadly to Twisted at the current time. Independent releases could still be undertaken, but I would dread this scenario - look at the email package and how many mistakes are made because software assumes a particular version will be installed. So, uh, yea. I'd love to see a mature, stable Twisted (or more likely Twisted subset) in the standard library. I think a few parts of Twisted are even almost ready for this to be considered, but aside from some of the things in twisted/python/, much of it still requires work. Jp
On Mon, 2004-10-11 at 15:31, exarkun@divmod.com wrote:
I had almost finished composing an identical email when this one arrived. So, I'll play devil's advocate: I think that part of the question Ed is asking is, "when can we have some of Twisted in the stdlib rather than asyncore". Considering how ancient and creaky asyncore is looking right now, I think it might be valid to consider some small subset of Twisted that subsumes asyncore's functionality for inclusion in the standard library. Of course this should be post-split. I don't think it makes much sense to try to do something like this now. Were this to happen, I would suggest it be in a different module name, "twistedcore" or something, to allow a stable interface to remain in the standard library, which would have a hope of being easily plug-compatible with future Twisted versions, but would not hamstring the ability of the Twisted team to put out new versions. This might also assuage Guido's concerns about framework-structured code; we could elide utilities such as twistd, and instead provide convenience functions for users to write their own daemonization, et. al. This would at least provide a minimal platform for running Twisted-compatible event handlers, to even out the curve between "install Twisted as a dependency" and "just run this stand-alone script".
On Mon, 11 Oct 2004 15:51:11 -0400, Glyph Lefkowitz <glyph@divmod.com> wrote:
Definitely.
Obviously the package should be named "internet" :)
So what would actually need to be fixed for this to happen? 1) Readable abstract reactor implementation 2) Producer/Consumer API repair 3) Non-recursive Deferred implementation (or "internet" could be Deferred-less, but that would seem to suck) 4) Less terrifying Failure implementation (or, again, Failure-less in the stdlib) 5) Better test coverage (for example, more than one unit test for writeSequence) 6) Latexified documentation (probably automatable) 7) Something with Interfaces Anything else? Jp
Glyph Lefkowitz wrote:
This seems like a good thing to aim for, say, inclusion in 2.5. I don't think there's any chance of a significant amount of twisted being included in the stdlib - for instance, the Failure code absolutely wouldn't be accepted (it's way nasty). asyncore is ugly, brittle, and a pain in the arse. A replacement for it would be most excellent. Key things to remember about stdlib inclusion (some of these have already been mentioned, but it doesn't hurt to emphasise them) - Backwards compatibility is _much_ stronger for the stdlib. At a minimum, you're looking at one release with a DeprecationWarning, and a second to rip the code out. At 12-18 months per major release, that's 2.5 to 3 years. - No-one's going to accept something for the stdlib that requires an external package to be useful. - No-one's going to accept a new significant lump of code unless there's someone (or, even better, more than one person) who's going to be happy to maintain it on an ongoing basis. And finally, there's absolutely no chance of any of this happening for 2.4. I'm cutting beta1 after I get some sleep, so the window there is completely closed. On the positive side, there's plenty of time to think about something for 2.5. <wink> Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Thu, 2004-10-14 at 13:28, Anthony Baxter wrote:
We should have a conversation about that at some point - I don't disagree that the code is nasty, but there are pieces of very core Twisted code that depend upon it. It only exists because Python's native representation of tracebacks is way nastier. ;-) Other than that, I completely agree with your post. That, and I am definitely NOT the person to ask to maintain the stdlib variant...
On Oct 14, 2004, at 2:28 PM, Glyph Lefkowitz wrote:
Well from what I remember from the last time I rewrote Failure, the gnarlyness is used only to "persist" a traceback longer than it should really exist. Perhaps there should be an understandable and high performance version of Failure that is used by default, with a separate debugerriffic version can be monkeypatched or adapted in when it's actually wanted? -bob
Ed Suominen wrote:
Exarkun pointed out some really good reasons why Twisted should not be in stdlib. I pretty much agree with him. However, there is something that could be done to stdlib: refactor the various networking things there so, that they are built out of modules that only implement protocol logic, and blocking "runners" for the protocol logic. That is, separate the socket calls from the protocol. The interface between them would probably look a lot like a twisted Protocol, and hopefully would enable code reuse (in either direction).
On Mon, 11 Oct 2004 22:54:13 +0300, Tommi Virtanen <tv@twistedmatrix.com> wrote:
I've always operated under the assumption that this is already almost trivally possible. Let me explore this for a moment... class CrummyTCPTransport: disconnecting = False # HAHAHA def __init__(self, socket, protocol): self.socket = socket self.protocol = protocol # Skip boring stuff like getHost etc def write(self, bytes): self.socket.sendall(bytes) def writeSequence(self, iovec): self.write("".join(iovec)) def run(self): while True: b = self.socket.recv(1024) if not b: break self.protocol.dataReceived(b) s = socket.socket() s.connect(...) p = imap4.IMAP4Client() t = CrummyTCPTransport(s, p) p.makeConnection(t) t.run() Reasonable? Crappy? Anyway, that's how I've always seen a simplistic, reactorless program using Twisted protocols. It's ugly, but I think it should work (and compare it to trying to make a stdlib protocol implementation work with Twisted!) Which of Twisted's protocol implementations are mature enough for the standard library though? I can think of one, maybe (not counting all the useless ones like ident), and I wrote it so I'm probably biased. Jp
On Mon, 11 Oct 2004 11:34:34 -0700, Ed Suominen <general@eepatents.com> wrote:
Guido (and others, I think) have expressed dislike for framework-structured modules in the Python standard library. Since the majority of Twisted falls into this category, this is a point on which Guido would probably require some convincing. Aside from this, inclusion in the standard library imposes even stricter backwards compatibility requirements (rightly so). Twisted is changing less rapidly than it was at one point, but it is still making backwards incompatible changes and deprecating a lot of old APIs. There are very fundamental parts of Twisted that just aren't ready to be called stable and set in stone. Additionally, the relatively slow Python release process (one major release every 12 - 18 months) would be deadly to Twisted at the current time. Independent releases could still be undertaken, but I would dread this scenario - look at the email package and how many mistakes are made because software assumes a particular version will be installed. So, uh, yea. I'd love to see a mature, stable Twisted (or more likely Twisted subset) in the standard library. I think a few parts of Twisted are even almost ready for this to be considered, but aside from some of the things in twisted/python/, much of it still requires work. Jp
On Mon, 2004-10-11 at 15:31, exarkun@divmod.com wrote:
I had almost finished composing an identical email when this one arrived. So, I'll play devil's advocate: I think that part of the question Ed is asking is, "when can we have some of Twisted in the stdlib rather than asyncore". Considering how ancient and creaky asyncore is looking right now, I think it might be valid to consider some small subset of Twisted that subsumes asyncore's functionality for inclusion in the standard library. Of course this should be post-split. I don't think it makes much sense to try to do something like this now. Were this to happen, I would suggest it be in a different module name, "twistedcore" or something, to allow a stable interface to remain in the standard library, which would have a hope of being easily plug-compatible with future Twisted versions, but would not hamstring the ability of the Twisted team to put out new versions. This might also assuage Guido's concerns about framework-structured code; we could elide utilities such as twistd, and instead provide convenience functions for users to write their own daemonization, et. al. This would at least provide a minimal platform for running Twisted-compatible event handlers, to even out the curve between "install Twisted as a dependency" and "just run this stand-alone script".
On Mon, 11 Oct 2004 15:51:11 -0400, Glyph Lefkowitz <glyph@divmod.com> wrote:
Definitely.
Obviously the package should be named "internet" :)
So what would actually need to be fixed for this to happen? 1) Readable abstract reactor implementation 2) Producer/Consumer API repair 3) Non-recursive Deferred implementation (or "internet" could be Deferred-less, but that would seem to suck) 4) Less terrifying Failure implementation (or, again, Failure-less in the stdlib) 5) Better test coverage (for example, more than one unit test for writeSequence) 6) Latexified documentation (probably automatable) 7) Something with Interfaces Anything else? Jp
Glyph Lefkowitz wrote:
This seems like a good thing to aim for, say, inclusion in 2.5. I don't think there's any chance of a significant amount of twisted being included in the stdlib - for instance, the Failure code absolutely wouldn't be accepted (it's way nasty). asyncore is ugly, brittle, and a pain in the arse. A replacement for it would be most excellent. Key things to remember about stdlib inclusion (some of these have already been mentioned, but it doesn't hurt to emphasise them) - Backwards compatibility is _much_ stronger for the stdlib. At a minimum, you're looking at one release with a DeprecationWarning, and a second to rip the code out. At 12-18 months per major release, that's 2.5 to 3 years. - No-one's going to accept something for the stdlib that requires an external package to be useful. - No-one's going to accept a new significant lump of code unless there's someone (or, even better, more than one person) who's going to be happy to maintain it on an ongoing basis. And finally, there's absolutely no chance of any of this happening for 2.4. I'm cutting beta1 after I get some sleep, so the window there is completely closed. On the positive side, there's plenty of time to think about something for 2.5. <wink> Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Thu, 2004-10-14 at 13:28, Anthony Baxter wrote:
We should have a conversation about that at some point - I don't disagree that the code is nasty, but there are pieces of very core Twisted code that depend upon it. It only exists because Python's native representation of tracebacks is way nastier. ;-) Other than that, I completely agree with your post. That, and I am definitely NOT the person to ask to maintain the stdlib variant...
On Oct 14, 2004, at 2:28 PM, Glyph Lefkowitz wrote:
Well from what I remember from the last time I rewrote Failure, the gnarlyness is used only to "persist" a traceback longer than it should really exist. Perhaps there should be an understandable and high performance version of Failure that is used by default, with a separate debugerriffic version can be monkeypatched or adapted in when it's actually wanted? -bob
Ed Suominen wrote:
Exarkun pointed out some really good reasons why Twisted should not be in stdlib. I pretty much agree with him. However, there is something that could be done to stdlib: refactor the various networking things there so, that they are built out of modules that only implement protocol logic, and blocking "runners" for the protocol logic. That is, separate the socket calls from the protocol. The interface between them would probably look a lot like a twisted Protocol, and hopefully would enable code reuse (in either direction).
On Mon, 11 Oct 2004 22:54:13 +0300, Tommi Virtanen <tv@twistedmatrix.com> wrote:
I've always operated under the assumption that this is already almost trivally possible. Let me explore this for a moment... class CrummyTCPTransport: disconnecting = False # HAHAHA def __init__(self, socket, protocol): self.socket = socket self.protocol = protocol # Skip boring stuff like getHost etc def write(self, bytes): self.socket.sendall(bytes) def writeSequence(self, iovec): self.write("".join(iovec)) def run(self): while True: b = self.socket.recv(1024) if not b: break self.protocol.dataReceived(b) s = socket.socket() s.connect(...) p = imap4.IMAP4Client() t = CrummyTCPTransport(s, p) p.makeConnection(t) t.run() Reasonable? Crappy? Anyway, that's how I've always seen a simplistic, reactorless program using Twisted protocols. It's ugly, but I think it should work (and compare it to trying to make a stdlib protocol implementation work with Twisted!) Which of Twisted's protocol implementations are mature enough for the standard library though? I can think of one, maybe (not counting all the useless ones like ident), and I wrote it so I'm probably biased. Jp
participants (7)
-
Andrew Bennetts -
Anthony Baxter -
Bob Ippolito -
Ed Suominen -
exarkun＠divmod.com -
Glyph Lefkowitz -
Tommi Virtanen