Re: [Python-Dev] trace.py and the obscurity of Tools/scripts/ (was: Unittest list)
Zooko <zooko@zooko.com> writes:
Since even python-dev'ers find ancient copies of trace.py on ftp sites before they find the one in the Tools/scripts/ directory, and since the debian package of Python 2.2 comes without a Tools/scripts/ directory at all, I conclude that the Tools/scripts/ directory it isn't doing its job very well. I suggest it either be killed or fixed. I'm not sure how to do the latter -- link to it from the doc pages?
Kill it and I'll be after you with an axe. Admittedly there's a lot of cruft that should be removed, rewritten or reorganised, but there's some mighty useful stuff there too (eg. logmerge.py). I don't see why I should suffer just because other people don't know where to look... Cheers, M. -- Darned confusing, unless you have that magic ingredient coffee, of which I can pay you Tuesday for a couple pounds of extra-special grind today. -- John Mitchell, 11 Jan 1999
[On Tools/scripts]
Kill it and I'll be after you with an axe. Admittedly there's a lot of cruft that should be removed, rewritten or reorganised, but there's some mighty useful stuff there too (eg. logmerge.py). I don't see why I should suffer just because other people don't know where to look...
Maybe more tools should follow the evolution of ndiff and migrate to the library (where ndiff lives on as difflib). --Guido van Rossum (home page: http://www.python.org/~guido/)
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> Maybe more tools should follow the evolution of ndiff and GvR> migrate to the library (where ndiff lives on as difflib). Probably so. I wonder if it makes sense to put them in a package (e.g. a `scripts' package)? -Barry
GvR> Maybe more tools should follow the evolution of ndiff and GvR> migrate to the library (where ndiff lives on as difflib).
Probably so. I wonder if it makes sense to put them in a package (e.g. a `scripts' package)?
-Barry
No, the point is that most of them ought to be converted into a usable library module, whose __main__ is just a simple way to invoke it. Many existing library modules already have this: e.g. pdb, profile, mimify, base64, etc. If you're going to keep them *just* as scripts, then they should remain in Tools. --Guido van Rossum (home page: http://www.python.org/~guido/)
Barry A. Warsaw writes:
Probably so. I wonder if it makes sense to put them in a package (e.g. a `scripts' package)?
Not sure about the package, but I like having the bulk of each script be in some importable module. Giving it a main() that takes an argv, using sys.argv if None is passed, and returning the return code for the shell is very attractive. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
[Guido]
Maybe more tools should follow the evolution of ndiff and migrate to the library (where ndiff lives on as difflib).
[Barry]
Probably so. I wonder if it makes sense to put them in a package (e.g. a `scripts' package)?
I don't see how that would be an improvement over leaving them in the scripts directory (which, btw, doesn't bother me a bit). Migration to the library is likely counterproductive unless the code is rewritten to *be* a library. Like ndiff was extensively refactored by two people (me and David Goodger), over two release cycles, to restructure it as a collection of reusable classes and functions. Even so, a much smaller ndiff.py *still* lives in the scripts directory, because the specific application of these algorithms-- and cmdline interface --it supplies don't make sense in a general library module.
"TP" == Tim Peters <tim.one@comcast.net> writes:
TP> I don't see how that would be an improvement over leaving them TP> in the scripts directory (which, btw, doesn't bother me a TP> bit). TP> Migration to the library is likely counterproductive unless TP> the code is rewritten to *be* a library. Like ndiff was TP> extensively refactored by two people (me and David Goodger), TP> over two release cycles, to restructure it as a collection of TP> reusable classes and functions. Even so, a much smaller TP> ndiff.py *still* lives in the scripts directory, because the TP> specific application of these algorithms-- and cmdline TP> interface --it supplies don't make sense in a general library TP> module. That's what I think I meant,... yeah! IOW, convert the script to a library module, with a main() and leave a little driver in the Tools/scripts directory. I was just thinking that a scripts package might be a convenient place to dump these little thingies. But maybe not. -Barry
Barry A. Warsaw <barry@zope.com> wrote:
That's what I think I meant,... yeah! IOW, convert the script to a library module, with a main() and leave a little driver in the Tools/scripts directory.
So in terms of `trace.py', it is a widely useful tool and already has a programmatic interface. Being added to the hallowed Python Standard Library would be a major step up in publicity and hence usage. It would require better docs regarding the programmatic usage. Anything else that would be required for trace to make this transition? Regards, Zooko Zooko.Com -- Security and Distributed Systems Engineering
So in terms of `trace.py', it is a widely useful tool and already has a programmatic interface. Being added to the hallowed Python Standard Library would be a major step up in publicity and hence usage. It would require better docs regarding the programmatic usage.
Anything else that would be required for trace to make this transition?
I see a copyright notice in the comments; I'd appreciate a transfer to the PSF so we can remove the copyrights. (The CWI copyright can be removed.) It could use a style upgrade. Change log comments typically don't go into comments (we have CVS for that). There are many lines longer than 80 characters (the preferred limit is 72 or 78). It uses strange @param markup in docstrings. There are lots of signed inline comments that sounds more like CVS checkin narrative than help on understanding the source. Other than that, I don't see why it couldn't be given more prominence. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Zooko]
... It would require better docs regarding the programmatic usage.
I believe it needs much better docs period. I've tried to steer people toward it, but they just can't figure out how to use it.
Anything else that would be required for trace to make this transition?
It would help if the bugs were fixed, or identified as common gotchas and explained in the hypothesized better docs <wink>. For example, def f(n): sum = 0 for i in range(n): sum for j in range(n): sum += 1 return sum print f(10) produces a .cover file containing 2: def f(n): 2: sum = 0 12: for i in range(n): 10: sum 120: for j in range(n): 100: sum += 1 1: return sum . 1: print f(10) Half the counts reported are "a surprise" to the uninitiated in one way or another; most people seem to expect 1: def f(n): 1: sum = 0 1: for i in range(n): 10: sum 10: for j in range(n): 100: sum += 1 1: return sum . 1: print f(10) while a smaller but still significant number seem to expect 1: def f(n): 1: sum = 0 11: for i in range(n): 10: sum 110: for j in range(n): 100: sum += 1 1: return sum . 1: print f(10) I realize "intuitive" counts are hard to achieve, but if it's in the standard library I'm likely to become the one who has to try to explain "non-intuitive" counts away year after year after year. So, to save myself a couple years of possibly otherwise happy life <wink>, I'm -1 unless the docs do a great job of cutting off questions before they get asked.
On Thu, 11 Apr 2002, Tim Peters wrote:
[Zooko]
Anything else that would be required for trace to make this transition?
It would help if the bugs were fixed [...]
And there are a ton of problems/bugs in trace.py from the current Python CVS. I've fixed about a dozen so far, and still going. I'll submit diffs as soon as I see the light at the end of the tunnel. Maybe I'm just running trace in a very different way than most and exercising some of the weirder code paths? In spite of the many esteemed contributors and the upcoming fixes from me, the design and the code are very raw and are (IMHO) a long way from being production quality. Of course, if everyone pitches in and we have a code frag-fest, it could conceivably be ready quite soon. Regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
[Kevin Jacobs]
And there are a ton of problems/bugs in trace.py from the current Python CVS.
Doesn't surprise me. It's a hairy subject, and judging from the comments embedded in the code, none of the contributors so far have shown a complete grasp of the issues. :-)
I've fixed about a dozen so far, and still going. I'll submit diffs as soon as I see the light at the end of the tunnel. Maybe I'm just running trace in a very different way than most and exercising some of the weirder code paths?
Everybody who uses this code uses it in a completely ideosyncratic way. That's what makes it so frustrating to try to unify the changes.
In spite of the many esteemed contributors and the upcoming fixes from me, the design and the code are very raw and are (IMHO) a long way from being production quality. Of course, if everyone pitches in and we have a code frag-fest, it could conceivably be ready quite soon.
Go for it! --Guido van Rossum (home page: http://www.python.org/~guido/)
zooko> So in terms of `trace.py', it is a widely useful tool and already zooko> has a programmatic interface. Being added to the hallowed Python zooko> Standard Library would be a major step up in publicity and hence zooko> usage. It would require better docs regarding the programmatic zooko> usage. It's speed cries out for a rewrite of some sort. I haven't thought about it, but I wonder if it could be layered on top of hotshot. Skip
"SM" == Skip Montanaro <skip@pobox.com> writes:
zooko> So in terms of `trace.py', it is a widely useful tool and zooko> already has a programmatic interface. Being added to the zooko> hallowed Python Standard Library would be a major step up in zooko> publicity and hence usage. It would require better docs zooko> regarding the programmatic usage. SM> It's speed cries out for a rewrite of some sort. I haven't SM> thought about it, but I wonder if it could be layered on top of SM> hotshot. Can any of the handler methods be re-coded in C? The hotshot changes allow you to install a C function as a trace hook, but doesn't the trace function in trace.py do a fair amount of work? Jeremy
SM> It's speed cries out for a rewrite of some sort. I haven't thought SM> about it, but I wonder if it could be layered on top of hotshot. Jeremy> Can any of the handler methods be re-coded in C? The hotshot Jeremy> changes allow you to install a C function as a trace hook, but Jeremy> doesn't the trace function in trace.py do a fair amount of work? Dunno. As you might guess I haven't looked at the trace code in quite awhile... :-) Skip
On Fri, 12 Apr 2002, Skip Montanaro wrote:
SM> It's speed cries out for a rewrite of some sort. I haven't thought SM> about it, but I wonder if it could be layered on top of hotshot.
Jeremy> Can any of the handler methods be re-coded in C? The hotshot Jeremy> changes allow you to install a C function as a trace hook, but Jeremy> doesn't the trace function in trace.py do a fair amount of work?
Dunno. As you might guess I haven't looked at the trace code in quite awhile... :-)
One way to make the function tracking code faster is to use the profiler hooks instead of the tracer hooks. I've already made this and many othe rcleanups nad bug fixes on my local version. I'm also looking at translating bits into C, though I've been side-tracked by an import related problem that trashes module paths. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
[cleaning out some old mail...] Zooko wrote:
So in terms of `trace.py', it is a widely useful tool and already has a programmatic interface. Being added to the hallowed Python Standard Library would be a major step up in publicity and hence usage. It would require better docs regarding the programmatic usage.
Skip wrote:
It's speed cries out for a rewrite of some sort. I haven't thought about it, but I wonder if it could be layered on top of hotshot.
Jeremy Hylton writes:
Can any of the handler methods be re-coded in C? The hotshot changes allow you to install a C function as a trace hook, but doesn't the trace function in trace.py do a fair amount of work?
Another possibility is to use _hotshot.coverage() instead of the "normal" profiler in HotShot. This records the enter/exit events and SET_LINENO instructions, but doesn't take any timing measurements, so operates much quicker than profiling. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
participants (9)
-
barry@zope.com
-
Fred L. Drake, Jr.
-
Guido van Rossum
-
Jeremy Hylton
-
Kevin Jacobs
-
Michael Hudson
-
Skip Montanaro
-
Tim Peters
-
Zooko