Guido van Rossum :

Eric didn't merge it with fileinput, but instead checked in a separate module "compilerlike". I have several comments on the code, but my main complaint is about process. This is a random bit of code that got checked in without any kind of discussion about whether it was worth checking into the standard library, and whether this particular implementation was right. There was some discussion afterwards, to which Eric did not respond. Given that Eric apparently doesn't care enough about his code to defend it, I propose to delete it from CVS.

I'll do this as part of the 2.2a3 release which (given the encouraging feedback so far) I'll try to do around Sept. 5.

Now, wait a second! There *was* in fact discussion of this thing beforehand. Several listmembers said they thought it was a good idea. And I have seen *no* discussion *at all* since it was checked in. What is going on here? Is it possible that you are mistaken about the timing of the checkin, and that what you thought was discussion afterwards was discussion before? Or am I somehow missing listmail? As for process issues...I agree that we need better procedures and criteria for what goes into the library. As you know I've made a start on developing same, but my understanding has been that *you* don't think you'll have the bandwidth for it until 2.2 is out. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> As war and government prove, insanity is the most contagious of diseases. -- Edward Abbey

Guido van Rossum :

I have now re-read that discussion; it's in the archives starting this message:

http://mail.python.org/pipermail/python-dev/2001-August/016629.html

As have I. All the stuff in this thread was before the checkin; you were in fact mistaken about the timing of most of the discussion.

There were several suggestions to merge it with fileinput and some suggestions to restructure it. You seem to have ignored these except the criticism on the name "ccframe" (by choosing an even worse name :-).

I did not ignore these suggestions (one that I took was Greg Ward's suggestion that, after all, just throwing an exception was the right thing). And I was in fact planning to merge this thing with fileinput. Then I looked as what would have to be done to the documentation of fileinput -- in fact, I edited together a combined fileinput documentation page. The result was a mess that convinced me that this does indeed need to be a separate module. There wasn't enough coherence between the old fileinput stuff and my entry points to even make the *documentation* look like a logical unit, let alone the code.

...

What is going on here? Is it possible that you are mistaken about the timing of the checkin, and that what you thought was discussion afterwards was discussion before? Or am I somehow missing listmail?

Your mail was probably broken -- it wouldn't be the first time :-(.

In the event, my mail was not broken.

There are two posts in the archives that start with a quote from the checkin mail:

http://mail.python.org/pipermail/python-dev/2001-August/017131.html http://mail.python.org/pipermail/python-dev/2001-August/017132.html

Right...one of which completely misses the point by suggesting that this is a filter framework, and the other one of which is a "me too" basically addressing the naming issue. Guido, you are yourself *notorious* for dismissing naming issues with "that's unimportant" and "we can fix it later". How can you criticize me for doing likewise?

...

As for process issues...I agree that we need better procedures and criteria for what goes into the library. As you know I've made a start on developing same, but my understanding has been that *you* don't think you'll have the bandwidth for it until 2.2 is out.

That's not an excuse for you to check in random bits of code.

So what, exactly, makes this 'random'? That, Guido, is not a rhetorical question. We don't have any procedures. We don't have any guidelines. We don't have any history of anything but discussing submissions on python-dev before somebody with commit access checks them in. If no -1 votes and the judgment of somebody with commit privileges who has already got a lot of stuff in the library is not sufficient, *what is*? I'm not trying to be difficult here, but this points at a weakness in our way of doing things. I want to play nice, but I can't if I don't know your actual rules. I don't know what *would* have been sufficient if what I did was not. I don't think anyone else does, either.

Some comments on the code:

This is the sort of critique I was looking for two weeks ago, not a bunch of bikeshedding about how the thing should be named.

- A framework like this should be structured as a class or set of related classes, not a bunch of functions with function arguments. This would make the documentation easier to read as well; instead of having a bunch of functions you pass in, you customize the framework byu overriding methods.

Yes, I thought of this. There's a reason I didn't do it that way. Method override would work just fine as a way to pass in the filename transformer, but not the data transformer. The problem is this: the driver or "go do it" method of your hypothetical class (the one you'd pass sys.argv[1:]) can't know which overriden method to call in advance, because which one is right would depend on the argument signature of the hook function -- does it take filelike objects, does it take two strings, etc. Actually it's worse than that; two of the cases (the sponge and the line-by-line filtering) aren't even distinguishable by type signature. So, what the driver function could do is step through three method names looking to see which if any is overridden in the user-created subclass. But would that really be a gain in clarity over having three functions in the module? I'm willing to listen if you think the answer is "yes" and want to tell me why, but it didn't seem so to me. There's something else I could have done. I could have required that the hook function use specific unique formal argument names in each of the three cases and then had the driver code use inspect to dispatch among them -- but that seemed even more klugey. Maybe there is a really elegant and low-overhead method of wrapping these functions in a class, and I have just not found it yet. But if so, it is not (as you appear to believe) for lack of looking. If you have an insight that I have missed, I will cheerfully accept instruction on this issue.

- The name "compilerlike" is a really poor choice (there's nothing compiler-like in the code).

No, there isn't. It's called "compilerlike" because it's a framework for making compilerlike interfaces out of functions. But I'm not attached to that name; CompilerFramework or something of that sort would be fine.

- I would like to see failure to open the file handled differently (so the caller can issue decent error message for inaccessible input files without having to catch all IOError exceptions), but again this is a policy issue that should be customizable.

Originally the code originally fielded file I/O errors by complaining to stderr and then exiting. At least two respondents argued that it should simply throw an exception and let the caller do policy, and upon reflection I came to agree with this (this is one of those suggestions you thought I was ignoring). I realize it's tempting to try and embed a range of policy options in the module to save time, but unless we can have reasonable confidence that they will cover all important cases I don't judge the complexity overhead to be worth it. Again, I am open to instruction on this.

- The policy of not writing the output if it's identical to the input should be optional. There are contexts (like when the tool is invoked by a Makefile) where not writing the output could be harmful: if you touch the input without changing it, Make would invoke the tool over and over again because the output doesn't get touched by the tool.

Interesting point. A better rule, perhaps, would be to suppress writing of output only if both the content *and* the transformed filename are identical -- that would avoid doing a spurious touch on a no-op modification in pace, without confusing Make.

Moreover, there seems to be some bugs: if the output is the same as the input, the output file is not written even if a filename transformation was requested (making Make even less happy); when a transformation is specified by a string, an undefined variable 'stem' is used. Hasty work, Eric. :-(

I'll take the hit for this; my test framework should have covered that case and didn't, because I was in a hurry to get in before the freeze. However; I know the other cases work because I'm *using* them. OK, so here's how I see it: 1. I made a minor implementation error with one case; this can be fixed. 2. You were mistaken in believing that (a) there was no discussion or endorsement of the idea before hand, and that (b) I did not defend or justify the design. 3. Some of the respondents simply missed the point; this thing is *not* a framework for creating filters, and shouldn't be named like one or put in the wrong library bin because of it. 4. There is room for technical debate about the interface design, but no choice I'm aware of that is *obviously* better than three functions -- the class-wrapper approach would have unobvious problems doing the hook function dispatch properly. 5. I was trying to do the right thing, but we sorely lack a useful set of norms for what constitutes `good' vs. `bad' librsary checkins. I am actively interested in helping solve problem. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> Non-cooperation with evil is as much a duty as cooperation with good. -- Mohandas Gandhi

Gordon McMillan :

True. It's a tad more general than just a filter. How about UnixishTextfileTransformFramework since that captures it's most distinguishing feature (the fact that it follows a popular unix convention). Also hints that it would chew the hell out of a binary file on Windows.

Eh? What's specific to text files in thec design? -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> A wise and frugal government, which shall restrain men from injuring one another, which shall leave them otherwise free to regulate their own pursuits of industry and improvement, and shall not take from the mouth of labor the bread it has earned. This is the sum of good government, and all that is necessary to close the circle of our felicities. -- Thomas Jefferson, in his 1801 inaugural address

Guido van Rossum :

...

I have now re-read that discussion; it's in the archives starting this message:

http://mail.python.org/pipermail/python-dev/2001-August/016629.html

As have I. All the stuff in this thread was before the checkin; you were in fact mistaken about the timing of most of the discussion.

No, I was not mistaken about the timing; you must have misunderstood what I said about the timing. When I posted the URL for this thread this afternoon, I knew that it took place before your checkin. I did not see evidence either in the mailing list or in the code that you took any of the advice though.

...

There were several suggestions to merge it with fileinput and some suggestions to restructure it. You seem to have ignored these except the criticism on the name "ccframe" (by choosing an even worse name :-).

I did not ignore these suggestions (one that I took was Greg Ward's suggestion that, after all, just throwing an exception was the right thing). And I was in fact planning to merge this thing with fileinput.

Then I looked as what would have to be done to the documentation of fileinput -- in fact, I edited together a combined fileinput documentation page. The result was a mess that convinced me that this does indeed need to be a separate module. There wasn't enough coherence between the old fileinput stuff and my entry points to even make the *documentation* look like a logical unit, let alone the code.

So, as a matter of process, you should not have checked it in without coming back to the list with your experience.

...

...

What is going on here? Is it possible that you are mistaken about the timing of the checkin, and that what you thought was discussion afterwards was discussion before? Or am I somehow missing listmail?

Your mail was probably broken -- it wouldn't be the first time :-(.

In the event, my mail was not broken.

...

There are two posts in the archives that start with a quote from the checkin mail:

http://mail.python.org/pipermail/python-dev/2001-August/017131.html http://mail.python.org/pipermail/python-dev/2001-August/017132.html

Right...one of which completely misses the point by suggesting that this is a filter framework, and the other one of which is a "me too" basically addressing the naming issue. Guido, you are yourself *notorious* for dismissing naming issues with "that's unimportant" and "we can fix it later". How can you criticize me for doing likewise?

I am criticizing you for not responding at all to the feedback -- whether it was mistaken or not. That's another violation of process. Why is suggesting it is a filtering framework completely missing the point? If this is not a filter framework, WHAT IS IT?

...

...

As for process issues...I agree that we need better procedures and criteria for what goes into the library. As you know I've made a start on developing same, but my understanding has been that *you* don't think you'll have the bandwidth for it until 2.2 is out.

That's not an excuse for you to check in random bits of code.

So what, exactly, makes this 'random'?

That, Guido, is not a rhetorical question. We don't have any procedures. We don't have any guidelines. We don't have any history of anything but discussing submissions on python-dev before somebody with commit access checks them in. If no -1 votes and the judgment of somebody with commit privileges who has already got a lot of stuff in the library is not sufficient, *what is*?

Absence of -1 votes is not enough. I didn't see any +1 votes -- just suggestions to try a different tack. I happened to be too busy at the time you checked this in to weigh in, but I had a big -1 in my head which I thought was reflected by other comments. Eric, I respect you as a person, but as a Python developer, I don't trust your judgement enough to let you check stuff in without a clear green light from me.

I'm not trying to be difficult here, but this points at a weakness in our way of doing things. I want to play nice, but I can't if I don't know your actual rules. I don't know what *would* have been sufficient if what I did was not. I don't think anyone else does, either.

Everybody else who doesn't know the rules for sure starts a discussion, either here or on the patch manager. You are the only one of the 30+ committers who *repeatedly* commits controversial stuff. I'm not saying that the rules are clear enough (they clearly aren't if even you don't get them), but I think there's a better way to get clarity than by acting like a bull in a china cabinet.

...

Some comments on the code:

This is the sort of critique I was looking for two weeks ago, not a bunch of bikeshedding about how the thing should be named.

I'll respond to this later. First I want you to be clear on the process: commit privileges are not to be used to force an issue. (Admin privileges will be used to force an issue if necessary. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

Eric, are you interested in pursuing this discussion further? You stopped replying, but that could be due to the holiday weekend. Anyway, the conclusion seems to be: - You violated the process as commonly understood by checking in a new module without sufficient consensus. - What you checked in is no good; it needs to be redesigned and renamed. - If you absolutely do not want to use the patch manager for this, you can check it in under the nondist/sandbox part of the tree. You can move your code to the sandbox, or I can delete it for you. --Guido van Rossum (home page: http://www.python.org/~guido/)

...

...

...

...

"AM" == Aahz Maruch writes:

AM> I'd add one more thing: there seems to be enough confusion and AM> disagreement that this probably should be a PEP. Volunteers?

As for process issues...I agree that we need better procedures and criteria for what goes into the library.

For a start, my current understanding is: - Adding a module to the standard library is a big change. - Modifying a core type or the interpreter loop is a big change. - Breaking compatibility is a big change. - Any big change requires confirmation specifically from Guido. I was pretty surprised to see compilerlike.py go in. Even if i had seen +1s from everyone else on python-dev, i still wouldn't have been comfortable checking it in without hearing from the BDFL. -- ?!ng

[Barry A. Warsaw]

Yeah, usually the BDFL has to berate you for weeks to get some minor addition, like oh, mimelib, into the standard library <wink>.

Yeah, I usually check something into Tools/scripts/ after 50 people ask for it. Guido never says anything, but I can sense him scowling. Then after a few years he demands to move it into the library. So I do. Then he complains that the exposed API is more suitable for a tool than a library, but that we have more important things to do than twiddle the API. It's a lot of fun to play within the system <wink>! speaking-of-which-i-should-check-combgen.py-into-Tools/scripts-ly y'rs - tim

Guido wrote:

I propose to delete it from CVS.

+1 (delete or move to the non-dist sandbox)

I'll do this as part of the 2.2a3 release which (given the encouraging feedback so far) I'll try to do around Sept. 5.

</F>

I have tried this with 2.2a1: 1. I have not checked the related C code 2. I have not checked the bug report list

...

...

...

class C(object): pass ... c=C() c.a=3 object.__getattr__(c,'a') 3 object.__setattr__(c,'a',6) Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: function takes exactly 2 arguments (3 given)

Is that meaningful or a problem? I'm trying to understand type/class unification to consider possible jython issues.

regards, Samuele Pedroni

This was a 2.2a1 specific problem: in that version, the object class didn't define a setattr dispatch slot, and object.__setattr__ ended up returning a *bound* method of the object *class*. In 2.2a2 this won't be a problem -- the object now defines __setattr__. --Guido van Rossum (home page: http://www.python.org/~guido/)