GvR> BTW, I think that Misc/unicode.txt should be converted to a GvR> PEP, for the historic record. It was very much a PEP before GvR> the PEP process was invented. Barry, how much work would GvR> this be? No editing needed, just formatting, and assignment GvR> of a PEP number (the lower the better).

Not much work at all, so I'll do this (and replace Misc/unicode.txt with a pointer to the PEP). Let's go with PEP 7, but stick it under the "Other Informational PEPs" category.

-Barry

Rather than informational, how about "Standard Track - Accepted (or Final)" ? That really matches the history best. I'd propose PEP number 100 -- the below-100 series is more for meta-PEPs. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:

...

Looking at the recent burst of checkins for the Unicode implementation completely bypassing the standard SF procedure and possible comments I might have on the different approaches, I guess I've been ruled out as maintainer and designer of the Unicode implementation.

Well, I guess that's how things go. Was nice working for you guys, but no longer is... I'm tired of having to defend myself against meta-comments about the design, uncontrolled checkins and no true backup about my standing in all this from Guido.

Perhaps I am misunderstanding the role of a maintainer and implementation designer, but as it is all respect for the work I've put into all this seems faded. That's the conclusion I draw from recent postings by Martin and Fredrik and their nightly "takeover".

Thanks, -- Marc-Andre Lemburg

[For those of us to whom Marc-Andre's complaint comes as a total surprise: there was a thread on i18n-sig about whether we should support Unicode surrogates, followed by a conclusion to skip surrogates and jump directly to optional support for UCS-4, followed by some checkins that enabled a configuration choice between UCS-2 and UCS-4, and code to make it work. As a side effect, surrogate support in the UCS-2 version actually improved slightly.]

Now, now, Marc-Andre.

The only comments I recall from you on my "surrogates: just say no" post seemed favorable, except that you proposed to to all the way and make UCS-4 mandatory. I explained why I didn't want to go that far, and why I didn't believe your arguments against giving users a choice. I didn't hear back from you then, and I didn't think you could have much of a problem with my position.

Our process requires the use of the SF patch manager only for controversial changes. Based on your feedback, I didn't think there was anything controversial about the changes that Fredrik and Martin have made! (If there was, IMO it was temporarily breaking the Windows build and the test suite -- but that's all fixed now.)

I don't understand where you get the idea that we lost respect for your work! In fact, the fact that it was so easy to make the changes suggested to me that the original design was well suited to this particular change (as opposed to the surrugate support proposals, which all sounded like they would require a *lot* of changes).

I don't think that we have very strict roles in this community anyway. (My role as BDFL excluded -- that's why I get to write this response. :-) I'd say that Fredrik owns SRE, because he has asserted that ownership at various times: he's undone changes by others that broke the 1.5.2 support, for example.

But the Unicode support in Python isn't owned by one person: many folks have contributed to that, including Fredrik, who designed and wrote the original Unicode string object implementation.

If you have specific comments about the changes made, please be specific. If you feel slighted by meta-comments, please also be specific. I don't think I've said anything derogatory about you or your design.

You didn't get my point. I feel responsable for the Unicode implementation design and would like to see it become a continued success. In that sense and taking into account that I am the maintainer of all this stuff, I think it is very reasonable to ask me before making any significant changes to the implementation and also respect any comments I put forward. Currently, I have to watch the checkins list very closely to find out who changed what in the implementation and then to take actions only after the fact. Since I'm not supporting Unicode as my full-time job this is simply impossible. We have the SF manager and there is really no need to rush anything around here. If I am offline or too busy with other things for a day or two, then I want to see patches on SF and not find new versions of the implementation already checked in. This has worked just fine during the last year, so I can only explain the latest actions in this direction with an urge to bypass my comments and any discussion this might cause. Needless to say that quality control is not possible anymore. Conclusion: I am not going to continue this work if this does not change. Another other problem for me is the continued hostility I feel on i18n against parts of the design and some of my decisions. I am not talking about your feedback and the feedback from many other people on the list which was excellent and to high standards. But reading the postings of the last few months you will find notices of what I am referring to here (no, I don't want to be specific). If people don't respect my comments or decision, then how can I defend the design and how can I stop endless discussions which simply don't lead anywhere ? So either I am missing something or there is a need for a clear statement from you about my status in all this. If I don't have the right to comment on proposals and patches, possibly even rejecting them, then I simply don't see any ground for keeping the implementation in a state which I can maintain. And last but not least: The fun-factor has faded which was the main motor driving my into working on Unicode in the first place. Nothing much you can do about this, though :-/

Paul Prescod offered to write a PEP on this issue. My cynical half believes that we'll never hear from him again, but my optimistic half hopes that he'll actually write one, so that we'll be able to discuss the various issues for the users with the users. I encourage you to co-author the PEP, since you have a lot of background knowledge about the issues.

I guess your optimistic half won :-) I think Paul already did all the work, so I'll simply comment on what he wrote.

BTW, I think that Misc/unicode.txt should be converted to a PEP, for the historic record. It was very much a PEP before the PEP process was invented. Barry, how much work would this be? No editing needed, just formatting, and assignment of a PEP number (the lower the better).

Thanks for converting the text to PEP format, Barry. Thanks for reading this far, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/

Guido van Rossum wrote:

Hi Marc-Andre,

I'm dropping the i18n-sig from the distribution list.

I hear you:

...

You didn't get my point. I feel responsable for the Unicode implementation design and would like to see it become a continued success.

I'm sure we all share this goal!

...

In that sense and taking into account that I am the maintainer of all this stuff, I think it is very reasonable to ask me before making any significant changes to the implementation and also respect any comments I put forward.

I understand you feel that we've rushed this in without waiting for your comments.

Given how close your implementation was, I still feel that the changes weren't that significant, but I understand that you get nervous. If Christian were to check in his speed hack changes to the guts of ceval.c I would be nervous too! (Heck, I got nervous when Eric checked in his library-wide string method changes without asking.)

Next time I'll try to be more sensitive to situations that require your review before going forward.

Good.

...

Currently, I have to watch the checkins list very closely to find out who changed what in the implementation and then to take actions only after the fact. Since I'm not supporting Unicode as my full-time job this is simply impossible. We have the SF manager and there is really no need to rush anything around here.

Hm, apart from the fact that you ought to be left in charge, I think that in this case the live checkins were a big win over the usual SF process. At least two people were making changes, sometimes to each other's code, and many others on at least three continents were checking out the changes on many different platforms and immediately reporting problems. We would definitely not have a patch as solid as the code that's now checked in, after two days of using SF! (We could've used a branch, but I've found that getting people to actually check out the branch is not easy.)

True, but I was thinking of the concept and design questions which should be resolved *before* taking the direct checkin approach.

So I think that the net result was favorable. Sometimes you just have to let people work in the spur of the moment to get the results of their best thinking, otherwise they lose interest or their train of thought.

Understood, but then I'd like to at least receive a summary of the changes in some way, so that I continue to understand how the implementation works after the checkins and which corners to keep in mind for future additions, changes, etc.

...

If I am offline or too busy with other things for a day or two, then I want to see patches on SF and not find new versions of the implementation already checked in.

That's still the general rule, but in our enthousiasm (and mine was definitely part of this!) we didn't want to wait. Also, I have to admit that I mistook your silence for consent -- I didn't think the main proposed changes (making the size of Py_UNICODE a config choice) were controversial at all, so I didn't realize you would have a problem with it.

I don't have a problem with it; I was just seeing things slip my fingers and getting worried about this.

...

This has worked just fine during the last year, so I can only explain the latest actions in this direction with an urge to bypass my comments and any discussion this might cause.

I think you're projecting your own stuff here.

Not really. I have processed many patches on SF, gave comments etc. and did the final checkin. This has worked great over the last months and I intend to keep working this way since it is by far the best way to both manage and document the issues and questions which arise during the process. E.g. I'm currently processing a patch by Walter Dörwald which adds support for callback error handlers. He has done some great work there which was the result of many lively discussions. Working like this is fun while staying manageable at the same time... and again, there's really no need to rush things !

I honestly didn't think there was much disagreement on your part and thought we were doing you a favor by implementing the consensus. IMO, Martin and and Fredrik are familiar enough with both the code and the issues to do a good job.

Well, the above was my interpretation of how things went. I may have been wrong (and honestly do hope that I am wrong), but my gutt feeling simply said: hey, what are these guys doing there... is this some kind of

...

Needless to say that quality control is not possible anymore.

Unclear. Lots of other people looked over the changes in your absence. And CVS makes code review after it's checked in easy enough. (Hey, in many other open source projects that's the normal procedure once the rough characteristics of a feature have been agreed upon: check in first and review later!)

That was not my point: quality control also includes checking the design approach. This is something which should normally be done in design/implementation/design/... phases -- just like I worked with you on the Unicode implementation late in 1999.

...

Conclusion: I am not going to continue this work if this does not change.

That would be sad, and I hope you will stay with us. We certainly don't plan to ignore your comments!

...

Another other problem for me is the continued hostility I feel on i18n against parts of the design and some of my decisions. I am not talking about your feedback and the feedback from many other people on the list which was excellent and to high standards. But reading the postings of the last few months you will find notices of what I am referring to here (no, I don't want to be specific).

I don't know what to say about this, and obviously nobody has the time to go back and read the archives. I'm sure it's not you as a person that was attacked. If the design isn't perfect -- and hey, since Python is the 80 percent language, few things in it are quite perfect! -- then (positive) criticism is an attempt to help, to move it closer to perfection.

If people have at times said "the Unicode support sucks", well, that may hurt. You can't always stay friends with everybody. I get flames occasionally for features in Python that folks don't like. I get used to them, and it doesn't affect my confidence any more. Be the same!

I'll try.

But sometimes, after saying "it sucks", people make specific suggestions for improvements, and it's important to be open for those even from sources that use offending language. (Within reason, of course. I don't ask you to listen to somebody who is persistently hostile to you as a person.)

Ok.

...

If people don't respect my comments or decision, then how can I defend the design and how can I stop endless discussions which simply don't lead anywhere ? So either I am missing something or there is a need for a clear statement from you about my status in all this.

Do you really *want* to be the Unicode BDFL? Being something's BDFL a full-time job, and you've indicated you're too busy. (Or is that temporary?)

I am currently doing a lot of consulting work, so things sometimes tighten up and are less work intense at other times. Given this setup, I think that I will be able to play the BD (without the FL) for Unicode for some time. I will certainly pass on the flag to someone else if I find myself not spending enough time on it. The only thing I'm asking for, is some more professional work mentality at times. If people make it hard for me to follow the development, then I cannot manage this task in a satisfying way.

I see you as the original coder, which means that you know that section of the code better than anyone, and whenever there's a question that others can't answer about its design, implementation, or restrictions, I refer to you. But given that you've said you wouldn't be able to work much on it, I welcome contributions by others as long as they seem knowledgeable.

Same here.

...

If I don't have the right to comment on proposals and patches, possibly even rejecting them, then I simply don't see any ground for keeping the implementation in a state which I can maintain.

Nobody said you couldn't comment, and you know that.

If I don't get a chance to comment on a summary of changes (be it before or after a batch of checkings), how am I supposed to follow up on them ? Keeping a close eye on the checkin mailing list doesn't help: it simply doesn't always give you the big picture. We are all professional quality programmers and I respect Fredrik and Martin for their coding quality and ideas. What I am asking for is some more teamwork.

When it comes to rejecting or accepting, I feel that I am still the final arbiter, even for Unicode, until I get hit by a bus. Since I don't always understand the implementation or the issues, I'll of course defer to you in cases where I think I can't make the decision, but I do reserve the right to be convinced by others to override your judgement, occasionally, if there's a good reason. And when you're not responsive, I may try to channel you. (I'll try to be more explicit about that.)

That's perfectly OK (and indeed can be very useful at times).

...

And last but not least: The fun-factor has faded which was the main motor driving my into working on Unicode in the first place. Nothing much you can do about this, though :-/

Yes, that happens to all of us at times. The fun factor goes up and down, and sometimes we must look for fun elsewhere for a while. Then the fun may come back where it appeared lost. Go on vacation, read a book, tackle a new project in a totally different area! Then come back and see if you can find some fun in the old stuff again.

I'll visit the Bordeaux Python conference later week. That should give me some time to breathe (and hopefully to write some more PEPs :=).

...

...

Paul Prescod offered to write a PEP on this issue. My cynical half believes that we'll never hear from him again, but my optimistic half hopes that he'll actually write one, so that we'll be able to discuss the various issues for the users with the users. I encourage you to co-author the PEP, since you have a lot of background knowledge about the issues.

I guess your optimistic half won :-) I think Paul already did all the work, so I'll simply comment on what he wrote.

Your suggestions were very valuable. My opinion of Paul also went up a notch!

...

...

BTW, I think that Misc/unicode.txt should be converted to a PEP, for the historic record. It was very much a PEP before the PEP process was invented. Barry, how much work would this be? No editing needed, just formatting, and assignment of a PEP number (the lower the better).

Thanks for converting the text to PEP format, Barry.

Thanks for reading this far,

You're welcome, and likewise.

Just one more thing, Marc-Andre. Please know that I respect your work very much even if we don't always agree. We would get by without you, but Python would be hurt if you turned your back on us.

Thanks. Be assured that I'll stay around for quite some time -- you won't get by that easily ;-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/

Thanks for your response, Marc-Andre. I'd like to close this topic now. I'm not sure how to get you a "summary of changes", but I think you can ask Fredrik directly (Martin annonced he's away on vacation). One thing you can do is pipe the output of "cvs log" through tools/scripts/logmerge.py -- this gives you the checkin messages in (reverse?) chronological order. --Guido van Rossum (home page: http://www.python.org/~guido/)

Fredrik Lundh wrote:

guido wrote:

...

I'm not sure how to get you a "summary of changes", but I think you can ask Fredrik directly (Martin annonced he's away on vacation).

summary:

- portability: made unicode object behave properly also if sizeof(Py_UNICODE) > 2 and >= sizeof(long) (FL) - same for unicode codecs and the unicode database (MvL) - base unicode feature selection on unicode defines, not platform (FL) - wrap surrogate handling in #ifdef Py_UNICODE_WIDE (MvL, FL) - tweaked unit tests to work with wide unicode, by replacing explicit surrogates with \U escapes (MvL) - configure options for narrow/wide unicode (MvL) - removed bogus const and register from some scalars (GvR, FL) - default unicode configuration for PC (Tim, FL) - default unicode configuration for Mac (Jack) - added sys.maxunicode (MvL)

Thank you for the summary. Please let me suggest that for the next coding party you prepare a patch which spans all party checkins and upload that patch with a summary like the above to SF. That way we can keep the documentation of the overall changes in one place and make the process more transparent for everybody. Now let's get on with business... Thanks, -- Marc-Andre Lemburg ________________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Fredrik Lundh wrote:

...

Please let me suggest that for the next coding party you prepare a patch which spans all party checkins and upload that patch with a summary like the above to SF. That way we can keep the documentation of the overall changes in one place and make the process more transparent for everybody.

Sorry, but as long as Guido wants an open development approach based on collective code ownership (aka "egoless programming"), that's what he gets.

The current environment provides several tools to track changes to the code base. The python-checkins list provides instant info on every single change to the code base; the investment to track tha list is a few minutes per day. The CVS history is also easy to access; you can reach it via the viewcvs interface, or from the command line.

I think you misunderstood my suggestion: I didn't say you can't have a coding party with lots of small checkins, I just suggested that *after* the party someone does a diff before-and-after-the-party.diff and uploads this diff to SF with a description of the overall changes. You simply don't get the big picture from looking at various small checkin messages which are sometimes spread across mutliple files/checkins.

Using both CVS and SF's patch manager to track development history is a waste of time. A development project manned by volunteers doesn't need bureaucrats; the version control system provides all the accountability we'll ever need.

(commercial development projects doesn't need bureaucrats either, and usually don't have them, but that's another story).

Wasn't talking about bureaucrats...

I'd also argue that using many incremental checkins improves quality -- the smaller a change is, the easier it is to understand, and the more likely it is that also non-experts will notice simple mistakes or portability issues. (I regularily comment on checkin messages that look suspicious codewise, even if I don't know anything about the problem area. I'm even right, sometimes). Reviewing big patches on SF is really hard, even for experts.

It's just for keeping a combined record of changes. Following up on dozens of checkins spanning another dozen files using CVS is harder, IMHO, than looking at one single before/after diff.

And every hour a patch sits on sourceforge instead of in the code repository is ten hours less burn-in in a heterogenous testing en- vironment. That's worth a lot.

Agreed.

Finally, my experience from this and other projects is that the "visible heartbeat" you get from a continuous flow of checkin messages improves team productivity and team morale. No- thing is more inspiring than seeing others working for a common goal. It's the final product that matters, not who's in charge of what part of it. The end user couldn't care less.

I'd prefer if you didn't feel the need to play miniboss on the Python project (I'm sure you have plenty of 'mx' projects that you can use that approach, if you have to).

I have no intention of playing "miniboss" (I have enough of that being the boss of a small company), I'm just trying to keep the task of a code maintainer manageable; that's all. 'nuff said.

And I'd rather see you at the next party than out there whining over how you missed the last one.

Perhaps you can send around invitations first, before starting the party next time ?! BTW, do you have plans to update the Unicode database to the 3.1 version ? If not, I'll look into this next week. -- Marc-Andre Lemburg ________________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/