On Wed, Jan 29, 2014 at 8:11 PM, anatoly techtonik wrote:

Normalized Python - a set of default, standard behaviors that backup common user expectations about cross-platform and system-independent behavior regardless of backward compatibility and code compatibility concerns.

Having a separate "Normalized Python" concept is needed to set the context for developing and engineering ideas, instead of concentrating on the sad reality of backward compatibility curse.

You can achieve the first two simply by opening files with parameters. There is NOTHING Windows-specific or Linux-specific in that. As of Python 3, opening in text mode is the default... but you can override that so easily. Why change the default (which breaks back compat) when you can just change your code? And I believe you can reopen stdin/stdout as binary, if you really want to, but that is a little harder. It's still not going to have any platform-specific code in it. (As I've never written a filter for binary files in Python, I've never had the need to read/write standard streams in binary. But I've no doubt that someone who has can show you how easy it is - I'd guess it's less than five lines of code, knowing Python.)

This is needed, for example, to collect these two features:

(Among our features are such diverse elements as... oh, wrong Pythons.)

1. open files in binary mode by default why? because "text file" is a human abstraction, for operating system it is just another format of binary data, so default operation is to read this data without any preprocessing

A reasonably plausible argument. C++ follows that sort of model (you shouldn't pay for anything you're not using). SQL mostly follows that model (it generally takes more keywords to get the database to do more work - compare "SELECT x FROM y" and "SELECT x FROM y ORDER BY z", where the latter adds a sort phase; there are exceptions to this, like UNION ALL vs UNION, but they're notable _because_ they're exceptions). But it's nothing like a strong enough argument for changing. Creating two subtly different languages is a major problem, especially when the exact same syntax means different things. Imagine if I create a fork of Python that's absolutely identical except that you create a set with [1,2,3] and a list with {1,2,3}. All your code will be syntactically correct, but suddenly it does something quite different. That is a BAD idea. It would have to be *immensely* better to justify the breakage; and this is only "arguably better". (The most obvious contrary argument is that the default should do the thing most people want most often, which is working with text files. This same argument justifies the use of arbitrary-precision integers by default, instead of requiring an explicit "long" type; I'm sure you'll agree that the Py3 unification of these types was an advantage.)

2. open text files in utf-8 encoding why? because users can not know the encoding of operating system, their programs can not choose right encoding, therefore a best guess is to expect the most widely used standard

Yes, this one is an issue. Python lets the OS recommend a default encoding, on the expectation that a Python script should fit into its host platform, rather than that all platforms should conform to what Python wants. A judgment call, and I'm sure there can be endless debates about what Python should do, but since it can be overridden with a single parameter on the open call, not a big deal IMO.

3. threat stdout/stdin streams as binary why? because you don't want you data to be corrupt when you pass it in and out of Python via standard streams

Most definitely NOT. The standard streams should, by default, be text streams, and should have their encodings set according to what the other side wants. If there's a way for the OS and Python to communicate an encoding, that's absolutely perfect. Yes, there'll be a few edge cases involving redirection, but that's pretty much unsolvable anyway. The normal usage of Python MUST include Unicode; and that means the most obvious way to produce output (the print function) needs to write Unicode. So if stdout is a binary stream, what's print going to do with a str? Encode it? If so, you just move the issue - and print can send to multiple streams, so it'd need to know which are text and which are binary, etc, etc. Or should it throw an error, and force the programmer to do stuff like this: CONSOLE_ENCODING = "utf-8" # add some logic for guessing this s = "Hello, world!" print(s.encode(CONSOLE_ENCODING)) just to ensure that every programmer has to battle with the encodings manually, in lots of places, instead of configuring it once (or, more likely, having the default be right) and then having clean code everywhere? The only way that opening stdin/out as binary will prevent the corruption of your data is if your data is fundamentally bytes. Most programs, in any language, work with data that's fundamentally text; granted, a lot of languages don't distinguish, but if you look at what the programmer's doing, it's still text. Anything that prints "Hello, world!" is printing text, not bytes, and if the console's encoding is UTF-16, that should emit 26 bytes (plus any newline that's appropriate). Forcing the programmer to think about this is completely unnecessary. How many times do you actually come across these issues in porting? How much effort would you really save if these measures were implemented? If it's that important to you, fork CPython and create this "Normalized Python" that does everything you want (and then, linking this with the other thread, continue development of Normalized Python according to an Agile model and see if people join you rather than CPython). Good luck. ChrisA