[Twisted-Python] Re: [Twisted-commits] r20094 - Add a spewing decorator which stores all of the function and method calls
On Sat, 28 Apr 2007 23:37:57 -0600, Jonathan Lange <jml@wolfwood.twistedmatrix.com> wrote:
Author: jml Date: Sat Apr 28 23:37:56 2007 New Revision: 20094
Modified: branches/spew-decorator/twisted/python/util.py branches/spew-decorator/twisted/test/test_util.py
Log: Add a spewing decorator which stores all of the function and method calls within a function in a structured data object.
What ticket is this associated with? Also, there is already an improved spew function in Epsilon which does nice things like track call depth and exception propagation and call arguments. Jean-Paul
On 4/29/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Sat, 28 Apr 2007 23:37:57 -0600, Jonathan Lange <jml@wolfwood.twistedmatrix.com> wrote:
Author: jml Date: Sat Apr 28 23:37:56 2007 New Revision: 20094
Modified: branches/spew-decorator/twisted/python/util.py branches/spew-decorator/twisted/test/test_util.py
Log: Add a spewing decorator which stores all of the function and method calls within a function in a structured data object.
What ticket is this associated with?
None, as yet.
Also, there is already an improved spew function in Epsilon which does nice things like track call depth and exception propagation and call arguments.
Should it go into Twisted? cheers, jml
On Mon, 30 Apr 2007 09:04:24 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/29/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Sat, 28 Apr 2007 23:37:57 -0600, Jonathan Lange <jml@wolfwood.twistedmatrix.com> wrote:
Author: jml Date: Sat Apr 28 23:37:56 2007 New Revision: 20094
Modified: branches/spew-decorator/twisted/python/util.py branches/spew-decorator/twisted/test/test_util.py
Log: Add a spewing decorator which stores all of the function and method calls within a function in a structured data object.
What ticket is this associated with?
None, as yet.
I guess we should avoid having branches without tickets. There's still the sandbox to develop stuff where the direction is uncertain.
Also, there is already an improved spew function in Epsilon which does nice things like track call depth and exception propagation and call arguments.
Should it go into Twisted?
I don't have any problem with it going into Twisted, and if you're looking for more informative spewage, it's definitely handy. I would suggest it unequivocally, but it has no unit tests (it's pretty old). It's probably salvageable, but even if it not, it should at least suggest some useful things. Jean-Paul
On 4/30/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Mon, 30 Apr 2007 09:04:24 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/29/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Sat, 28 Apr 2007 23:37:57 -0600, Jonathan Lange <jml@wolfwood.twistedmatrix.com> wrote:
Author: jml Date: Sat Apr 28 23:37:56 2007 New Revision: 20094
Modified: branches/spew-decorator/twisted/python/util.py branches/spew-decorator/twisted/test/test_util.py
Log: Add a spewing decorator which stores all of the function and method calls within a function in a structured data object.
What ticket is this associated with?
None, as yet.
I guess we should avoid having branches without tickets. There's still the sandbox to develop stuff where the direction is uncertain.
What's wrong with renaming the branch to include a ticket number after a suitable ticket has been filed? I thought the sandbox was for random bits of junk code[1] and Foolscap, not branches of Twisted. cheers, jml [1] FWIW, these days I just push branches up to launchpad.net/~jml/+junk/foo to achieve the same effect as a commit to sandbox.
On Mon, 30 Apr 2007 12:45:59 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/30/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Mon, 30 Apr 2007 09:04:24 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/29/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Sat, 28 Apr 2007 23:37:57 -0600, Jonathan Lange <jml@wolfwood.twistedmatrix.com> wrote:
Author: jml Date: Sat Apr 28 23:37:56 2007 New Revision: 20094
Modified: branches/spew-decorator/twisted/python/util.py branches/spew-decorator/twisted/test/test_util.py
Log: Add a spewing decorator which stores all of the function and method calls within a function in a structured data object.
What ticket is this associated with?
None, as yet.
I guess we should avoid having branches without tickets. There's still the sandbox to develop stuff where the direction is uncertain.
What's wrong with renaming the branch to include a ticket number after a suitable ticket has been filed?
That operation is unsupported by the tool-chain. But even so, for the reasons below, it's better to start things off with a ticket, rather than eventually introduce one.
I thought the sandbox was for random bits of junk code[1] and Foolscap, not branches of Twisted.
More or less, yes. Anything goes in the sandbox. Sticking whole Twisted branches there probably isn't the best idea on the world, but developing functionality that actually needs to go into the Twisted tree probably also shouldn't be undertaken without some minimal amount of planning. That's supposed to be part of what tickets are for (although I recognize that sometimes they do not end up filling that role). When I suggested the sandbox, I really had in mind the development of some spewer-related functionality in an independent module with little or no Twisted integration (since from a quick look at the branch, that seemed to be what the code looked like so far). In case that's a little too muddled and fuzzy to make sense, here's another angle. A ticket serves as a point where the relevance, utility, correctness, whatever, of a change can be discussed. If someone is interested in some development which is taking place, they should be able to find that point easily and without interactive assistance from anyone. If not, their input may be lost or delayed until it is useless or integrating it costs more than the ultimate payoff. The kind of "just-in-time" ticket creation that already happens so frequently in Twisted development is something it would be nice to move away from. It's completely fine to find a bug, create a ticket, and develop a fix in a brief period of time and I don't want to discourage that. However, and this is really close to the heart of UQDS, for refactoring and feature enhancements, the end result is much improved by input from other people. Leaving a little time between ticket creation and actual development doesn't guarantee that any useful input will be offered by other developers, but it at least presents the possibility. If development starts _before_ anyone else even knows what it's about, obviously that doesn't present any window at all. This really also applies to bug fixes, but I think it's at least /possible/ in some of those cases for the necessary change to be sufficiently straightforward such that the process works well enough with only two developers involved. If any of this seems unreasonable, please say so and let's discuss it. The goal here is to make Twisted the best piece of software we can make it, nothing else. Jean-Paul
On 4/30/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Mon, 30 Apr 2007 12:45:59 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/30/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Mon, 30 Apr 2007 09:04:24 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/29/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Sat, 28 Apr 2007 23:37:57 -0600, Jonathan Lange <jml@wolfwood.twistedmatrix.com> wrote:
Author: jml Date: Sat Apr 28 23:37:56 2007 New Revision: 20094
Modified: branches/spew-decorator/twisted/python/util.py branches/spew-decorator/twisted/test/test_util.py
Log: Add a spewing decorator which stores all of the function and method calls within a function in a structured data object.
What ticket is this associated with?
None, as yet.
I guess we should avoid having branches without tickets. There's still the sandbox to develop stuff where the direction is uncertain.
What's wrong with renaming the branch to include a ticket number after a suitable ticket has been filed?
That operation is unsupported by the tool-chain. But even so, for the reasons below, it's better to start things off with a ticket, rather than eventually introduce one.
I thought the sandbox was for random bits of junk code[1] and Foolscap, not branches of Twisted.
More or less, yes. Anything goes in the sandbox. Sticking whole Twisted branches there probably isn't the best idea on the world, but developing functionality that actually needs to go into the Twisted tree probably also shouldn't be undertaken without some minimal amount of planning. That's supposed to be part of what tickets are for (although I recognize that sometimes they do not end up filling that role). When I suggested the sandbox, I really had in mind the development of some spewer-related functionality in an independent module with little or no Twisted integration (since from a quick look at the branch, that seemed to be what the code looked like so far).
In case that's a little too muddled and fuzzy to make sense, here's another angle. A ticket serves as a point where the relevance, utility, correctness, whatever, of a change can be discussed. If someone is interested in some development which is taking place, they should be able to find that point easily and without interactive assistance from anyone. If not, their input may be lost or delayed until it is useless or integrating it costs more than the ultimate payoff.
The kind of "just-in-time" ticket creation that already happens so frequently in Twisted development is something it would be nice to move away from. It's completely fine to find a bug, create a ticket, and develop a fix in a brief period of time and I don't want to discourage that. However, and this is really close to the heart of UQDS, for refactoring and feature enhancements, the end result is much improved by input from other people. Leaving a little time between ticket creation and actual development doesn't guarantee that any useful input will be offered by other developers, but it at least presents the possibility. If development starts _before_ anyone else even knows what it's about, obviously that doesn't present any window at all. This really also applies to bug fixes, but I think it's at least /possible/ in some of those cases for the necessary change to be sufficiently straightforward such that the process works well enough with only two developers involved.
If any of this seems unreasonable, please say so and let's discuss it. The goal here is to make Twisted the best piece of software we can make it, nothing else.
Thanks for the thoughtful reply.
From my perspective, the problem here is that strict adherence to UQDS as described in your email is that there is no real room for exploratory coding.
At other times, I have wanted to muck around doing some Trial refactoring, I filed a ticket which stated my intentions with all the clarity available, something along the lines of "muck around doing trial refactoring". I was chastised for filing such a ticket (fair enough, it's a lousy ticket), so I stopped working along those lines. This time, I decided to not file a ticket, because I did not want to prematurely problem of concisely defining my work. I am being chastised (albeit gently) for such work, and I will probably stop working on this code.
I sent this email by accident during the process of drafting. Please ignore until I write my email in full. On 4/30/07, Jonathan Lange <jml@mumak.net> wrote:
On 4/30/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Mon, 30 Apr 2007 12:45:59 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/30/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Mon, 30 Apr 2007 09:04:24 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/29/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Sat, 28 Apr 2007 23:37:57 -0600, Jonathan Lange <jml@wolfwood.twistedmatrix.com> wrote: >Author: jml >Date: Sat Apr 28 23:37:56 2007 >New Revision: 20094 > >Modified: > branches/spew-decorator/twisted/python/util.py > branches/spew-decorator/twisted/test/test_util.py > >Log: >Add a spewing decorator which stores all of the function and method calls >within a function in a structured data object. >
What ticket is this associated with?
None, as yet.
I guess we should avoid having branches without tickets. There's still the sandbox to develop stuff where the direction is uncertain.
What's wrong with renaming the branch to include a ticket number after a suitable ticket has been filed?
That operation is unsupported by the tool-chain. But even so, for the reasons below, it's better to start things off with a ticket, rather than eventually introduce one.
I thought the sandbox was for random bits of junk code[1] and Foolscap, not branches of Twisted.
More or less, yes. Anything goes in the sandbox. Sticking whole Twisted branches there probably isn't the best idea on the world, but developing functionality that actually needs to go into the Twisted tree probably also shouldn't be undertaken without some minimal amount of planning. That's supposed to be part of what tickets are for (although I recognize that sometimes they do not end up filling that role). When I suggested the sandbox, I really had in mind the development of some spewer-related functionality in an independent module with little or no Twisted integration (since from a quick look at the branch, that seemed to be what the code looked like so far).
In case that's a little too muddled and fuzzy to make sense, here's another angle. A ticket serves as a point where the relevance, utility, correctness, whatever, of a change can be discussed. If someone is interested in some development which is taking place, they should be able to find that point easily and without interactive assistance from anyone. If not, their input may be lost or delayed until it is useless or integrating it costs more than the ultimate payoff.
The kind of "just-in-time" ticket creation that already happens so frequently in Twisted development is something it would be nice to move away from. It's completely fine to find a bug, create a ticket, and develop a fix in a brief period of time and I don't want to discourage that. However, and this is really close to the heart of UQDS, for refactoring and feature enhancements, the end result is much improved by input from other people. Leaving a little time between ticket creation and actual development doesn't guarantee that any useful input will be offered by other developers, but it at least presents the possibility. If development starts _before_ anyone else even knows what it's about, obviously that doesn't present any window at all. This really also applies to bug fixes, but I think it's at least /possible/ in some of those cases for the necessary change to be sufficiently straightforward such that the process works well enough with only two developers involved.
If any of this seems unreasonable, please say so and let's discuss it. The goal here is to make Twisted the best piece of software we can make it, nothing else.
Thanks for the thoughtful reply.
From my perspective, the problem here is that strict adherence to UQDS as described in your email is that there is no real room for exploratory coding.
At other times, I have wanted to muck around doing some Trial refactoring, I filed a ticket which stated my intentions with all the clarity available, something along the lines of "muck around doing trial refactoring". I was chastised for filing such a ticket (fair enough, it's a lousy ticket), so I stopped working along those lines.
This time, I decided to not file a ticket, because I did not want to prematurely problem of concisely defining my work. I am being chastised (albeit gently) for such work, and I will probably stop working on this code.
On 4/30/07, Jonathan Lange <jml@mumak.net> wrote:
On 4/30/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Mon, 30 Apr 2007 12:45:59 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/30/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Mon, 30 Apr 2007 09:04:24 +1000, Jonathan Lange <jml@mumak.net> wrote:
On 4/29/07, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Sat, 28 Apr 2007 23:37:57 -0600, Jonathan Lange <jml@wolfwood.twistedmatrix.com> wrote: >Author: jml >Date: Sat Apr 28 23:37:56 2007 >New Revision: 20094 > >Modified: > branches/spew-decorator/twisted/python/util.py > branches/spew-decorator/twisted/test/test_util.py > >Log: >Add a spewing decorator which stores all of the function and method calls >within a function in a structured data object. >
What ticket is this associated with?
None, as yet.
I guess we should avoid having branches without tickets. There's still the sandbox to develop stuff where the direction is uncertain.
What's wrong with renaming the branch to include a ticket number after a suitable ticket has been filed?
That operation is unsupported by the tool-chain. But even so, for the reasons below, it's better to start things off with a ticket, rather than eventually introduce one.
I thought the sandbox was for random bits of junk code[1] and Foolscap, not branches of Twisted.
More or less, yes. Anything goes in the sandbox. Sticking whole Twisted branches there probably isn't the best idea on the world, but developing functionality that actually needs to go into the Twisted tree probably also shouldn't be undertaken without some minimal amount of planning. That's supposed to be part of what tickets are for (although I recognize that sometimes they do not end up filling that role). When I suggested the sandbox, I really had in mind the development of some spewer-related functionality in an independent module with little or no Twisted integration (since from a quick look at the branch, that seemed to be what the code looked like so far).
In case that's a little too muddled and fuzzy to make sense, here's another angle. A ticket serves as a point where the relevance, utility, correctness, whatever, of a change can be discussed. If someone is interested in some development which is taking place, they should be able to find that point easily and without interactive assistance from anyone. If not, their input may be lost or delayed until it is useless or integrating it costs more than the ultimate payoff.
The kind of "just-in-time" ticket creation that already happens so frequently in Twisted development is something it would be nice to move away from. It's completely fine to find a bug, create a ticket, and develop a fix in a brief period of time and I don't want to discourage that. However, and this is really close to the heart of UQDS, for refactoring and feature enhancements, the end result is much improved by input from other people. Leaving a little time between ticket creation and actual development doesn't guarantee that any useful input will be offered by other developers, but it at least presents the possibility. If development starts _before_ anyone else even knows what it's about, obviously that doesn't present any window at all. This really also applies to bug fixes, but I think it's at least /possible/ in some of those cases for the necessary change to be sufficiently straightforward such that the process works well enough with only two developers involved.
If any of this seems unreasonable, please say so and let's discuss it. The goal here is to make Twisted the best piece of software we can make it, nothing else.
Thanks for the thoughtful reply.
From my perspective, the problem here is that strict adherence to UQDS as described in your email is that there is no real room for exploratory coding.
At other times, I have wanted to muck around doing some Trial refactoring, I filed a ticket which stated my intentions with all the clarity available, something along the lines of "muck around doing trial refactoring". I was chastised for filing such a ticket (fair enough, it's a lousy ticket), so I stopped working along those lines.
This time, I decided to not file a ticket, because I did not want to prematurely problem of concisely defining my work. I am being chastised (albeit gently) for such work, and I will probably stop working on this code.
Actually, on reflection, I'll leave that email as is. It's blunter than I would have made it otherwise, but it accurately represents my opinions. However, I will note: - I could have chosen to work on this branch offline. In which case, the code wouldn't have been available for review at all. - The very fact of this thread indicates that there is a window for discussion. - Every branch has chance to be reviewed, and thus discussed. - The time penalty of discussion is significant to me because of my timezone. - I really do think discussion is good. However, I think that planning as a group is not always worth the cost. My goal is more complex than making Twisted the best software possible. My goal is to be able to sit down and work on Twisted even if I don't have a fully specified, fully approved goal in mind. cheers, jml
I'll respond to a few specific things JML said here, but I want to get across my personal view of the purpose and benefits of the process we're using. I specifically told JML to make a branch and continue with these changes before he got started, and I still think that was the right thing to do. UQDS was instituted on Twisted because tracking changes to Twisted was becoming too difficult. Some changes were critical bugfixes, some were random whitespace twiddling, and some introduced regressions. It was designed in the first place (for Divmod) because we were having similar issues in a different context. In other words, not all change is progress. The goal of UQDS was to provide a way to encourage and ensure that all changes to *trunk* are, in fact, progress, and Twisted is always improving, so that we don't need to burn our energy tracking down bugs or gratuitously bad design decisions after they've been merged to trunk and people have begun depending on them. I think it has been very successful at meeting this goal. It's true that the UQDS document on the Divmod wiki does specify that the lifetime of a task in UQDS begins with the creation of a ticket. For the context that I originally wrote "If it's not in the tracker, you shouldn't be working on it.", i.e. the full-time developers at Divmod, that makes perfect sense. In a commercial environment, management needs to have a clear idea of where all the resources of the company are being deployed. I would go so far as to say that it makes sense for Twisted as well, insofar as UQDS manages our workflow. However, Twisted SVN does things other than UQDS (the trunk/doc/fun/Twisted.Quotes exception and the sandbox, to name two notable things) and I don't think that every branch needs to be backed by a ticket. I'd rather have non-ticketed branches that we can delete as prototypes than a profusion of vague tickets which only their author understands. (In other words, SVN branches are not always necessarily part of the UQDS workflow, but tickets are.) It's fine by me if there is tons of bad, broken code that gets checked in to branches (as long as it is then deleted - see below). Sometimes, the only way to learn how to implement a feature correctly is to implement it incorrectly. Sometimes the only way to figure out what feature you're implementing is to mess around for a while and see what kind of code you write. This is especially true of refactoring. The policy on having tickets applies to anything wanting a review, and therefore, anything that hopes to go to trunk. If you have done some work in a branch which you want reviewed, you still need to explain, for the reviewer's benefit, what it is that the change is supposed to accomplish in a ticket. Again, this is especially true of refactoring - it's fine to noodle around for a while trying to discover a better shape for the code, but once you've found one, it's important to clearly express *why* the new shape is better rather than simply different. Just because the description eventually needs to be written doesn't mean it needs to be the first thing that happens, though. There is one caveat to creating branches for experimentation. I think it's rude to leave dead branches around. This can be mitigated by using ticket numbers, because someone else can come along later, notice that the branch refers to a closed ticket, and delete it. Open branches have a cost: "svn ls svn://svn.twistedmatrix.com/svn/Twisted/branches" should, at a glance, give an indication as to what's in progress in Twisted right now, and dead branches obscure that view and make it difficult to get a feel for what's going on. Right now there are 133 entries there, and it's probably time for some pruning. So I think it's reasonable to say that if you leave a ticket-number-free branch around for more than a month, don't be surprised if someone else deletes it. (But please delete it yourself before it comes to that.) However, deletion or rejection should not necessarily be seen as a failure, either of the developer or of the process. The whole *point* of using branches as temporary development lines is that, sometimes, those lines end without reaching trunk. Each rejected branch should be a learning experience. Another important thing to keep in mind, as long as I'm talking about how to use and not use branches: don't *ever* deploy Twisted from a branch, and this goes triple for a branch created for experimentation. Such branches are still at the mercy of a reviewer, and may be substantially changed or deleted. Eventually, *all* branches will be deleted, as they are merged to trunk and become obsolete. On 08:42 am, jml@mumak.net wrote:
This time, I decided to not file a ticket, because I did not want to prematurely problem of concisely defining my work. I am being chastised (albeit gently) for such work, and I will probably stop working on this code.
Please don't. This kind of clean-up can be very important. I don't know if the particular changes in this branch are, but certainly the changes which can come out of exploratory coding are useful.
However, I will note: - The time penalty of discussion is significant to me because of my timezone.
An important feature of UQDS is its asynchrony. Although branches have to be reviewed before they're merged, to the extent that changes do not directly depend upon one another, it's critical that someone be able to sit down and work on feature X, then feature Y, then feature Z, without stopping each time to wait 24 hours for a review and then again for the second review. With 24 hours (or more!) of context-switch latency, the inability to pipeline requests could completely kill development speed. This asynchrony should apply to all aspects of the process, including discussion of requirements and design goals. The key feature of a heavyweight (read: broken) process is that a developer who is itching to do some work has to sit around and wait for permission from a committee to do so.
- I really do think discussion is good. However, I think that planning as a group is not always worth the cost.
Enforcement by consensus is great. Design by consensus (i.e. committee) is, unfortunately, terrible. I never intended UQDS to require that. The best sort of group design is when one person has an idea and champions it and the group offers criticism and analysis. Having the idea in the first place can require writing some code.
My goal is more complex than making Twisted the best software possible. My goal is to be able to sit down and work on Twisted even if I don't have a fully specified, fully approved goal in mind.
As I see it, irrespective of their priorities, the latter is a prerequisite for the former. If, as an open source project, we can't harness the power of people's idle time, I think we're doomed.
Jean-Paul Calderone wrote: [...]
In case that's a little too muddled and fuzzy to make sense, here's another angle. A ticket serves as a point where the relevance, utility, correctness, whatever, of a change can be discussed. If someone is interested in some development which is taking place, they should be able to find that point easily and without interactive assistance from anyone. If not, their input may be lost or delayed until it is useless or integrating it costs more than the ultimate payoff.
Some quick observations: * the reasons for using tickets you give here (a place to discuss a change, a central place to look for development in progress, facilitating early discussion of a change) are not given on the official UQDS page (http://www.divmod.org/trac/wiki/UltimateQualityDevelopmentSystem). * the official UQDS page gives reasons for using tickets that are irrelevant to volunteer contributions: "a way to manage distractions and continuously re-focus on what's really important." Open source volunteers are scratching itches. Whatever they are working on *is* what's important. Other things may be important to other people, and they are certainly welcome to file tickets asking for things to be done, but that's outside of stated role of UQDS: that's just ordinary bug (and feature request) tracking. I think it is useful to have a place to discuss a branch *once it is ready for public consumption*. A developer should *not* be discouraged from using version control tools just because they haven't decided exactly what direction they are going in yet, or even if their experiment is worthwhile. I think that developers *should* be encouraged to share their in development work early and often, because as you say input from others is often helpful. But forcing them to jump through the hoops of filing tickets (thus requiring them to have a clear statement exactly what they are working when they might not know yet) seems like a bad idea. Have you never started hacking on something without a clear idea of if this was something worthwhile or not, or even exactly what it might be good for? This is a useful and legimitate form of development. Or are you saying that a ticket titled "Do stuff!" is fine? :) A final thought... we *do* have a place to discuss branches without tickets, as you have demonstrated: this mailing list :) -Andrew.
participants (4)
-
Andrew Bennetts -
glyph＠divmod.com -
Jean-Paul Calderone -
Jonathan Lange