On 01/11/2014 10:32 AM, Antoine Pitrou wrote:

On Sat, 11 Jan 2014 18:41:49 +0100 Victor Stinner <victor.stinner@gmail.com> wrote:

...

If you agree, I will modify the PEP. If Antoine disagree, I will fork the PEP 460 ;-)

Please fork it.

You've already stated you don't care that much and are willing to let the PEP as-is be rejected. Why not remove your name and let Victor have it back? Is he not the original author? (If this is protocol just say so -- remember I'm still new to the ways of PyDev. :). -- ~Ethan~

On Sat, 11 Jan 2014 10:38:01 -0800 Ethan Furman <ethan@stoneleaf.us> wrote:

On 01/11/2014 10:32 AM, Antoine Pitrou wrote:

...

On Sat, 11 Jan 2014 18:41:49 +0100 Victor Stinner <victor.stinner@gmail.com> wrote:

...

If you agree, I will modify the PEP. If Antoine disagree, I will fork the PEP 460 ;-)

Please fork it.

You've already stated you don't care that much and are willing to let the PEP as-is be rejected. Why not remove your name and let Victor have it back? Is he not the original author? (If this is protocol just say so -- remember I'm still new to the ways of PyDev. :).

Because the PEP is IMO a much saner compromise than what you're trying to do (and would also stand a better chance of being accepted, if it weren't for your stupid maximalist opposition). Regards Antoine.

Am 11.01.2014 20:22, schrieb Antoine Pitrou:

On Sat, 11 Jan 2014 10:38:01 -0800 Ethan Furman <ethan@stoneleaf.us> wrote:

...

On 01/11/2014 10:32 AM, Antoine Pitrou wrote:

...

On Sat, 11 Jan 2014 18:41:49 +0100 Victor Stinner <victor.stinner@gmail.com> wrote:

...

If you agree, I will modify the PEP. If Antoine disagree, I will fork the PEP 460 ;-)

Please fork it.

You've already stated you don't care that much and are willing to let the PEP as-is be rejected. Why not remove your name and let Victor have it back? Is he not the original author? (If this is protocol just say so -- remember I'm still new to the ways of PyDev. :).

Because the PEP is IMO a much saner compromise than what you're trying to do (and would also stand a better chance of being accepted, if it weren't for your stupid maximalist opposition).

Can you please stop throwing personal insults around? You don't have to resort to that level. Georg

On 01/12/2014 12:02 PM, Stephen J. Turnbull wrote:

Georg Brandl writes:

...

Antoine writes:

...

. . . if it weren't for your stupid maximalist opposition. . .

Can you please stop throwing personal insults around? You don't have to resort to that level.

Ethan's posts (as an example of one general trend in this thread) are pretty frustrating, you have to admit.

Two points: 1) Are you saying it's okay to be insulting when frustrated? I also find this mega-thread frustrating, but I'm trying very hard not to be insulting. 2) If you are going to use my name, please be certain of the facts [1]. More below.

MAL posted straight out the Python 2 model of text makes it easier for him to write some programs, so he's all for reintroducing it. And that is the whole truth of the matter. Although I disagree with him, I appreciate his honesty.

If you have an example of me lying (even if it's just a possibility), please refer to it directly so I can either try to explain the misunderstanding or apologize.

But people keep posting "we don't want Python 2's confounding of text and binary, we just want bytes with (nearly) all the functionality of strings [because they are (partially|really) encoded text]". Some of them actually use the literal word "text" in their justification!

In only one case did I use the word "text" loosely, and that was when I claimed that Py2 had three text types, and Py3 had two. I was wrong, I apologize. Py3 has one definite text type, str, and, I claim, one half text type in bytes, because bytes itself provides ASCII text processing methods. If you have a better term for the notion of b'ethan'.title() --> b'Ethan' than ASCII-text processing, I'll use that instead. If there are good reasons to not allow further concessions to the ASCII-ness of bytes (and you provide a good one below) then that makes living with the handicap easier. But don't lie to me (as Nick tried to) and say that "In particular, the bytes type is, and always will be, designed for pure binary manipulation" when it has methods like .center(). If I am wrong, and that was not a lie, please explain it to me.

That's, well, what would you call it? Either they know what they're saying, in which case it's disingenuous at best, or they don't know what they're saying, in which case it's a proposal based on a clear misunderstanding of the situation.

I think some of the misunderstanding (which you also seem to suffer from) is that we (or at least I) /ever/ want a unicode string back from bytes interpolation. I don't! If I start with bytes, I want bytes back! And I have a very clear grasp on the difference between str and bytes and what ACSII encoding means, it was a hard and painful lesson for me and I'm not likely to forget it. To summarize, I used the term text when referring to unicode text (str), ASCII or ASCII-encoded text to refer to bytes that are to be used in a place that requires ASCII bytes for communication (such as content length or field type). I do /not/ use ASCII to refer to any ol' collection of bytes that happens to look like it might be ASCII-encoded text.

The problem is not going to go away just because they *say* they don't want to reintroduce Python 2 text processing. That is precisely what this proposal is *intended* to do, whether in the limited form proposed by Antoine or in the much more extensive form that folks like Ethan want.

What "maximalists" mean is that they promise not to abuse Python 2 text processing when writing Python 3 programs. This promise is highly unlikely to be kept for two reasons. First, they can't make that promise on behalf of third parties, who for various reasons certainly will abuse these features to avoid the encoded-text-to- Unicode-text and vice-versa conversions.

I concede that this is a good reason to not allow % interpolation. Kinda like not allowing sum on strings. And I don't make promises for other people, and abusing this feature would be a bug.

Second, I doubt they themselves will keep the promise to my satisfaction because their definition of "text" is ambiguous.

*My* definition is not ambiguous at all. If this particular part of the byte stream is defined to contain ASCII-encoded text, then I can use the bytes text methods to work with it. The only time I would return a bytes object is if it was supposed to be bytes (an image, for example); otherwise I return a bool, an int, a float, a date, or, even, a str.

When it's convenient for them to use text-processing operations on bytes, they'll say "oh, yes, these are conventionally considered text-processing features, but that's just an accident of the particular configuration of bytes -- yup, bytes -- I'm processing."

If that particular configuration of bytes is because it's ASCII-encoded text, then sure. To use, for example, bytes.__upper__ on data that wasn't ASCII-encoded text (even if it happened to look like it was) would be the height of stupidity. Please don't include me in such accusations.

But Nick's important example of web frameworks demonstrates the problem: unless they convert to text where appropriate, they're just pushing the problem off on application writers. Sometimes passing on data as bytes is appropriate, of course, but the framework authors are likely to be biased in favor of doing that, and it's not hard to imagine frameworks ported from Python 2 passing on the problem wholesale on the grounds that "we returned str in Python 2 which is bytes in Python 3, and since we were processing bytes the whole time, we see no reason to change the 'ABI'." Of course the application writers thought they were receiving text "in an inconvenient and ambiguous form". IMO, with the proposed changes, that is likely to continue indefinitely, negating some of the gains I expected to receive from Python 3. :-(

This would be a good reason to reject PEP 460, if that danger was deemed more likely than the good it would bring.

Note: there are a lot of high-level frameworks like Django that even in Python 2 basically went to Unicode everywhere internally. I don't deny that. I think that Python 3 as currently constituted makes it a lot easier to make an appropriate decision of where to convert, and should take some of the burden off the high-level frameworks. Approving this PEP, especially in a maximalist form, will blur the lines.

I understand your point, but I disagree. When I open a file (in binary mode, obviously, as otherwise I'd get massive corruption) I get back a bunch of bytes. When working with tcp, I get back a bunch of bytes. bytes are /already/ the boundary type. If we have to make a third type for proper boundary processing it's an admission that bytes failed in its role. -- ~Ethan~ [1] I double-checked all my posts on this topic both here and on Python Ideas to make sure.

On 01/12/2014 02:52 PM, Antoine Pitrou wrote:

You are right, it is not ok. The wording wasn't constructive or controlled at all. I'd like to apologize for that.

Thank you. Apology accepted.

At the same point, I was expressing a fair amount of frustration. I think the last discussion rounds have largely failed to produce any new meaningful insight (to the point that I've stopped reading several subthreads). IMO the best thing *for now* would be to "agree to disagree", let things bake in everyone's mind for some time, and revisit the subject in some weeks.

For the most part I agree. I did, though, finally figure out what Nick thought I wanted, so there was at least a little progress. But yes, I think tabling the discussion for now, and working on Brett's ideas, is entirely appropriate. -- ~Ethan~ P.S. Direct reply so you don't miss my response. :)

On 01/12/2014 07:02 PM, Stephen J. Turnbull wrote: [snip most of very eloquent reply] Thank you, Stephen, for remaining calm despite my somewhat heated response. A few comments in-line. I now better understand your viewpoint about text always being unicode strings; I just happen to disagree. Hopefully as some consolation I will be very vocal about using str unless bytes is necessary. Any application that uses text should be using str for it, and only using bytes, if necessary, on the back-end.

Ethan Furman writes:

...

In only one case did I use the word "text" loosely,

[...] Bytes are *never* Python 3 text in my terminology [...] "ASCII-encoded text" as you call it [...] and want to manipulate using str-like methods on bytes

The part that you don't seem to acknowledge (sorry if I missed it) is that there are str-like methods already on bytes. While the actual implementation of isupper (your example from below) may be done using integer methods, it only makes semantic sense if interpreted as ASCII-encoded text.

is *exactly* the Python 2 model of text. But you deny that the effect of your proposals (eg, b"%d" % (12,)) is to reintroduce Python 2's bytes/character confusion, don't you?

Given that the default (and only) text type in Py3 is str, which is unicode, I don't think any confusion will be as severe, but I acknowledge that there could be some.

I hardly think Nick is *lying*, any more than you are. AFAICT, you're *both* wrong.

LOL, well, at least I'm in good company, then! :)

...

I think some of the misunderstanding (which you also seem to suffer from) is that we (or at least I) /ever/ want a unicode string back from bytes interpolation. I don't!

Please tell me why you think I suffer from that misunderstanding.

I no longer recall, but whatever misapprehension I was suffering from you have alleviated. (That sentence would make my daughter pround! English major. ;)

But did you get that I'm worried that programmers in Omaha will use that same functionality to communicate American English (for which it is basically sufficient, and which also requires ASCII when bytes are used for communication)?

Yes, I get that. Hopefully their friends and neighbors will slap them with fishes if they do.

...

*My* definition is not ambiguous at all. If this particular part of the byte stream is defined to contain ASCII-encoded text, then I can use the bytes text methods to work with it.

But how is Python supposed to know that?

Python doesn't need to. bytes is a low-level object -- it could contain music, movies, dbf data, pdf data, or my mothers cheesecake recipe (properly encoded, of course). Python can't protect me from treating a music file as if it were a movie file, or even just writing proper music info at the wrong place in the music file; all that is up to me, as the programmer, to get right, and to understand what is needed.

But under your definition, you need to make the decision, or explicitly code the decision, on the basis of context.

Exactly so. I even have to do that in Py2.

...

If that particular configuration of bytes is because it's ASCII-encoded text, then sure.

Once again, you are advocate precisely the Python 2 model of text.

Not exactly, because what I get back is bytes, which cannot directly be mixed with unicode (str) as it was in Py2. I think this is a key difference.

...

To use, for example, bytes.__upper__ on data that wasn't ASCII-encoded text (even if it happened to look like it was) would be the height of stupidity. Please don't include me in such accusations.

I have no idea why you think I think anybody would be that stupid. That never occured to me. It's precisely "magic numbers" that happen to look like English words when interpreted as ASCII coded characters that I don't want manipulated by str-like methods that interpret text (such as full-featured format or %).

This confuses me somewhat. It's okay to use b'ethan'.upper(), which only makes semantic sense as ASCII-encoded text, but b'age: %d' % 43 isn't? (Aside, I'm perfectly comfortable with "ASCII-encoded text" because if you took u'ethan'.encode('ascii') you would get b'ethan'. If it was some other encoding, such as cp1251, I would call that particular byte stream "cp1251-encoded text". And if there were methods that worked directly on a cp1251-encoded byte stream I would not have any problem using them on cp1251-encoded text.)

What Nick means by a "boundary type" is a type that works seamlessly with the types on each side of the boundary as a helper in the conversion. So when you use a struct to pack a bool, an int, and a date into a bytes, the struct is the boundary type. And if there's a helper type to work with bytes and/or str simultaneously, that's a boundary type, eg, asciistr. But bytes itself is not a boundary type, it's just a type with no internal structure, not even characters.

Hmmm. I'll have to think about this. Okay, I've thought somewhat. Under the definition above would it be fair to say that Db3Table (a class in my dbf module) is a boundary type? It sits between the actual file and the program, and transforms bytes into actual Python types. -- ~Ethan~

Ethan Furman writes:

The part that you don't seem to acknowledge (sorry if I missed it) is that there are str-like methods already on bytes.

I haven't expressed myself well, but I don't much care about that. It's what Knuth would classify as a seminumerical method. What I do care about is that the methods that convert other types to text (including format) not work for bytes. That's where I consider text to "start".

...

is *exactly* the Python 2 model of text. But you deny that the effect of your proposals (eg, b"%d" % (12,)) is to reintroduce Python 2's bytes/character confusion, don't you?

Given that the default (and only) text type in Py3 is str, which is unicode, I don't think any confusion will be as severe, but I acknowledge that there could be some.

I fear it will be quite severe where I live, in Shift JIS/GB18030 land. (The two most obnoxious encodings known to man, except perhaps the syntax of Brainf!ck.)

...

...

*My* definition is not ambiguous at all. If this particular part of the byte stream is defined to contain ASCII-encoded text, then I can use the bytes text methods to work with it.

But how is Python supposed to know that?

Python doesn't need to.

... because you know it. But the ideal of object-oriented programming (and duck-typing) is that you shouldn't need to; the object should know how to produce appropriate behavior itself.

...

But under your definition, you need to make the decision, or explicitly code the decision, on the basis of context.

Exactly so. I even have to do that in Py2.

"Even." This is exactly where PBP and EIBTI part company, I think. EIBTI thinks its a bad idea to pass around bytes that are implicitly some other type, and Python 3 *should be good enough to make that unnecessary*. I'm convinced, and Nick is convinced, that we can make that true for 90% of the cases that it isn't now, if we could just figure out what's hard about the use cases where Python 3 isn't up to snuff yet (and figure out which use cases we need to handle to get us up to 90%!) PBP doesn't think it's a great idea to pass around bytes that are implicitly some other type, but didn't mind it (or got used to it) in Python 2, and so they're not looking at that as a problem that Python 3 can solve. They're looking at Python 3 as the problem that prevents them from doing what worked fine in Python 2. I understand that point of view, I just think we should be able to do better in Python 3, and should give it a serious try before giving in. Remember, "Special cases aren't special enough to break the rules" comes *before* "Although practicality beats purity". Not to forget that "Explicit is better than implicit" is second[1] on the list. ;-) After looking at this thread, I feel that (due to misunderstandings on both sides) purity hasn't really been tried yet.

...

...

If that particular configuration of bytes is because it's ASCII-encoded text, then sure.

Once again, you are advocate precisely the Python 2 model of text.

Not exactly, because what I get back is bytes, which cannot directly be mixed with unicode (str) as it was in Py2. I think this is a key difference.

You're in good company there; that was Guido's rationale for not worrying, too. I agree it's "key" (and I'm sure Nick will, on reflection if not already). But I worry (a lot) that it's not enough.

This confuses me somewhat. It's okay to use b'ethan'.upper(), which only makes semantic sense as ASCII-encoded text,

Not really OK. In theory, because it doesn't require serialization/ encoding of a primitive type, it doesn't matter. In practice, without powerful formatting, it isn't even a major attraction. In practice, with powerful formatting, it adds to the attraction. Note that regex doesn't require type conversions (matches have methods to return positions in the target or subsequences of the target, not values of other types), which is why I (and I suspect Nick for the same reason) am comfortable with polymorphic regex but not with bytes formatting.

(Aside, I'm perfectly comfortable with "ASCII-encoded text" because if you took u'ethan'.encode('ascii') you would get b'ethan'. If it was some other encoding, such as cp1251, I would call that particular byte stream "cp1251-encoded text".

Even though "ethan" is perfectly good ASCII-encoded text (as well as the integer 435,744,694,638 on a bigendian machine with 5-byte words, and you have no way of knowing whether it was user data (CP1251) or a metadata keyword (ASCII) or be the US national debt in 1967 dollars (integer) when b'ethan' shows up in a trace?

And if there were methods that worked directly on a cp1251-encoded byte stream I would not have any problem using them on cp1251-encoded text.)

I was afraid of that: all of those methods (except the case methods[2]) will work fine on a cp1251-encoded text. And because they only know that the string is bytes, the case methods will silently corrupt your "text" as soon as they get a chance. That bothers me, even if it doesn't bother you. Purity again, if you like. (But you'd take a safe .upper if you got it for free, no?)

Okay, I've thought somewhat. Under the definition above would it be fair to say that Db3Table (a class in my dbf module) is a boundary type? It sits between the actual file and the program, and transforms bytes into actual Python types.

Yes, I'd call that a boundary type. Footnotes: [1] Yes, I know what's number 1, but I'm not going to mention it out loud! [2] Arguably those too, since bytes don't have a locale. They're in C locale and the bytes >127 don't have semantics like case.

Stephen J. Turnbull wrote:

PBP doesn't think it's a great idea to pass around bytes that are implicitly some other type, but didn't mind it (or got used to it) in Python 2, and so they're not looking at that as a problem that Python 3 can solve. They're looking at Python 3 as the problem that prevents them from doing what worked fine in Python 2.

While some people may think that way, I don't think it's fair to characterise *all* proponents of bytes formatting as luddites that refuse to get with the Python 3 way. Some of us *do* understand the principles of text/ bytes separation in Python 3 and agree that they're a good idea. We just don't agree that the proposed formatting operations violate those principles to any degree worth worrying about. I don't think of my viewpoint as being PBP. That term assumes there is purity there to be beaten. To my mind, any notion of purity with respect to bytes objects went out the window as soon as it was given a pile of text methods -- together with a text-like literal syntax and default repr(), even though at least half the time they're completely inappropriate! -- Greg

Greg Ewing writes:

I don't think of my viewpoint as being PBP. That term assumes there is purity there to be beaten. To my mind, any notion of purity with respect to bytes objects went out the window as soon as it was given a pile of text methods -- together with a text-like literal syntax and default repr(), even though at least half the time they're completely inappropriate!

Isn't an analogous statement true of every programming language taken as a whole? Does that mean that, because Python 1 text handling was unavoidably "practical", adding the "purist" unicode type in Python 2 was a mistake? Python 3's sacrifice of Python 2 compatibility seems positively degenerate by your standard! To be less contentious, surely the concept of "purity" includes "purification" (even if that doesn't apply to some subdivisions of purity)? In any case, taking your statement at face value, I consider adding the methods to have been a mistake and the literal syntax and repr to be compact abbreviations that are frequently convenient. Byte sequences that can be considered to serializations of objects including ASCII text, so that some subsequences of bytes in the range 0-127 can be usefully considered as a text representation are very common. But I think that they're important enough that their representation in Python deserves a type (maybe more than one) that tries to enforce what regularities there are in such streams. The purity position is probably going to lose in the end, since Guido is clearly in the PBP camp at this point, and that's a strong indicator (especially since Nick has given up on convincing python-dev). But that does not mean it's entirely invalid.

On Mon, Jan 13, 2014 at 9:34 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 14 January 2014 15:15, Stephen J. Turnbull <stephen@xemacs.org> wrote:

...

The purity position is probably going to lose in the end, since Guido is clearly in the PBP camp at this point, and that's a strong indicator (especially since Nick has given up on convincing python-dev). But that does not mean it's entirely invalid.

I didn't give up regarding PEP 460 - Guido pointed out an error in my assumptions that made my position invalid, and his correct. "Give up" makes it sound like I got tired of arguing without being convinced rather than admitting I was just plain wrong.

Thanks for that. (I was worried when I saw your first huge post in the reboot thread.)

While I'll still work on the asciistr proposal, that's unrelated to PEP 460 - it's about making hybrid APIs less painful to write in Python 3 when you're willing to place the burden of ensuring ASCII compatibility of binary data on the calling code. That kind of thing is likely to be a reasonable approach in specific domains (when writing a web development framework, for example), even though I think it's an *in*appropriate design for the standard library.

I've now looked at asciistr. (Thanks Glenn and Ethan for the link.) Now that I (hopefully) understand it, I'm worried that a text processing algorithm that uses asciistr might under hard-to-predict circumstances (such as when the arguments contain nothing of interest to the algorithm) might return an asciistr instance instead of a str or bytes instance, and this might confuse a caller (e.g. isinstance() checks might fail, dict lookups, or whatever -- it feels like the problem is similar to creating the perfect proxy type).

PEP 460 should actually make asciistr easier in the long run, as I now expect we'll run into some "interesting" issues getting formatting to produce anything other than text (contrary to what I said elsewhere in these threads - I hadn't thought through the full implications at the time).

For example? -- --Guido van Rossum (python.org/~guido)

[Other readers: asciistr is at https://github.com/jeamland/asciicompat] On Mon, Jan 13, 2014 at 11:44 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

Right, asciistr is designed for a specific kind of hybrid API where you want to accept binary input (and produce binary output) *and* you want to accept text input (and produce text output). Porting those from Python 2 to Python 3 is painful not because of any limitations of the str or bytes API but because it's the only use case I have found where I actually *missed* the implicit interoperability offered by the Python 2 str type.

Yes, the use case is clear.

It's not an implementation style I would consider appropriate for the standard library - we need to code very defensively in order to aid debugging in arbitrary contexts, so I consider having an API like urllib.parse demand 7-bit ASCII in the binary version, and require text to handle impure input to be a better design choice.

This surprises me. I think asciistr should strive to be useful for the stdlib as well.

However, in an environment where you can place greater preconditions on your inputs (such as "ensure all input data is ASCII compatible")

That gives me the Python 2 willies. :-(

and you're willing to tolerate the occasional obscure traceback for particular kinds of errors,

Really? Can you give an example where the traceback using asciistr() would be more obscure than using the technique you used in urllib.parse?

then it should be a convenient way to use common constants (like separators or URL scheme names) in an algorithm that can manipulate either binary or text, but not a combination of the two (the latter is still a nice improvement in correctness over Python 2, which allowed them to be mixed freely rather than requiring consistency across the inputs).

Unfortunately I suspect there are still examples where asciistr's "submissive" behavior can produce surprises. E.g. consider a function of two arguments that must either be both bytes or both str. It's easily conceivable that for certain combinations of incorrect arguments (i.e. one bytes and one str) the function doesn't raise an error but returns something of one or the other type. (And this is exactly the Python 2 outcome we're trying to avoid.)

It's still slightly different from Python 2, though. In Python 2, the interaction model was:

str & str -> str str & unicode -> unicode

(with the one exception being str.format: that consistently produces str rather than promoting to Unicode)

Or raises good old UnicodeError. :-(

My goal for asciistr is that it should exhibit the following behaviour:

str & asciistr -> str asciistr & asciistr -> str (making it asciistr would be a pain and I don't have a use case for that)

I almost had one in the example code I sent in response to Greg.

bytes & asciistr -> bytes

I understand that '&' here stands for "any arbitrary combination", but what about searches? Given that asciistr's base class is str, won't it still blow up if you try to use it as an argument to e.g. bytes.startswith()? Equality tests also sound problematic; is b'x' == asciistr('x') == 'x' ???

So in code like that in urllib.parse (but in a more constrained context), you could just switch all your constants to asciistr, change your indexing operations to length 1 slices and then in theory essentially the same code that worked in Python 2 should also work in Python 3.

The more I think about this, the less I believe it's that easy. I suspect you had the right idea when you mentioned singledispatch. It might be easier to write the bytes version in terms of the string versions wrapped in decode/encode, or vice versa, rather than trying to reason out all the different combinations of str, bytes, asciistr.

However, Benno is finding that my warning about possible interoperability issues was accurate - we have various places where we do PyUnicode_Check() rather than PyUnicode_CheckExact(), which means we don't always notice a PEP 3118 buffer interface if it is provided by a str subclass.

Not sure I understand this, but I believe him when he says this won't be easy.

We'll look at those as we find them, and either work around them (if we can), decide not to support that behaviour in asciistr, or else I'll create a patch to resolve the interoperability issue.

It's not necessarily a type I'd recommend using in production code, as there *will* always be a more explicit alternative that doesn't rely on a tricksy C extension type that only works in CPython. However, it's a type I think is worth having implemented and available on PyPI, even if it's just to disprove the claim that you *can't* write that kind of code in Python 3.

Hm. It is beginning to sound more and more flawed. I also worry that it will bring back the nightmare of data-dependent UnicodeError back. E.g. this (from tests/basic.py): def test_asciistr_will_not_accept_codepoints_above_127(self): self.assertRaises(ValueError, asciistr, 'Schrödinger') looks reasonable enough when you assume asciistr() is always used with a literal as argument -- but I suspect that plenty of people would misunderstand its purpose and write asciistr(s) as a "clever" way to turn a string into something that's compatible with both bytes and strings... :-( -- --Guido van Rossum (python.org/~guido)

On Tue, Jan 14, 2014 at 1:37 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

Yep - that's why I consider asciistr to be firmly in the "power tool" category. If you know what you're doing, it should let you write hybrid API code that is just as concise as Python 2, but it's also far more error prone than the core Python 3 text model.

Hm. It sounds like the kind of power tool that only candidates for the Darwin award would use. The more I hear you defend it, the less I think it's a good idea for *anything*. And limiting it to PyPy doesn't make it less dangerous. -- --Guido van Rossum (python.org/~guido)

On Tue, Jan 14, 2014 at 10:16:17AM -0800, Guido van Rossum wrote:

Hm. It is beginning to sound more and more flawed. I also worry that it will bring back the nightmare of data-dependent UnicodeError back. E.g. this (from tests/basic.py):

def test_asciistr_will_not_accept_codepoints_above_127(self): self.assertRaises(ValueError, asciistr, 'Schrödinger')

looks reasonable enough when you assume asciistr() is always used with a literal as argument -- but I suspect that plenty of people would misunderstand its purpose and write asciistr(s) as a "clever" way to turn a string into something that's compatible with both bytes and strings... :-(

I am one of those people. I've been trying to keep on top of this enormous multiple-thread discussion, and although I haven't read every single post in its entirety, I thought I understand the purpose of asciistr was exactly that, to produce something that was compatible with both bytes and strings. -- Steven

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/15/2014 12:57 AM, Stephen J. Turnbull wrote:

asciistr *canonizes* something as an ASCII string, and therefore compatible with both bytes and str. It can't *create* such a thing ex nihilo.

How many miracles must be attested? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLWJbYACgkQ+gerLs4ltQ7RHACfft2ysdHiE9zJM72ycqi0Uqyl s5EAnR9Z21tgqsFVsPUEPiWgtXNxCWF4 =Thyi -----END PGP SIGNATURE-----

Guido van Rossum wrote:

I've now looked at asciistr. (Thanks Glenn and Ethan for the link.)

Now that I (hopefully) understand it, I'm worried that a text processing algorithm that uses asciistr might under hard-to-predict circumstances (such as when the arguments contain nothing of interest to the algorithm) might return an asciistr instance instead of a str or bytes instance,

It seems to me that any algorithm with that property has a genuine ambiguity as to what it should return in that case. Arguably, returning an asciistr would be the *right* thing to do, because that would allow it to be used as a component of a larger algorithm that was polymorphic with respect to text/bytes. -- Greg

On Tue, Jan 14, 2014 at 7:59 AM, Guido van Rossum <guido@python.org> wrote:

Here's an example of what I mean:

I sent that off without proofreading, and I also got one detail about asciistr() wrong. Here are some corrections.

def spam(a): r = asciistr('(') if a: r += a.strip() r += asciistr(')') return r

The argument must be a string.

Or a bytes object. And the point is that the return type should be the same as the argument type.

If I call spam(''),

or spam(b'')

a's type is never concatenated with r, so the return value is an asciistr.

Actually, Nick explained that asciistr() + asciistr() returns str, so this would be accidentally correct if called with '', but wrong (returning a str instead of a bytes) if called with b''.

To fix this particular case, we could drop the "if a:" part. But it could be more significant, e.g. it could be something like "if a contains any digits". The general fix would be to add

else: r += a[:0]

but that's still an example of the awkwardness that asciistr() is trying to avoid.

This is still valid. -- --Guido van Rossum (python.org/~guido)

On 15 Jan 2014 08:14, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:

Guido van Rossum wrote:

...

Actually, Nick explained that asciistr() + asciistr() returns str,

That part seems wrong to me, because it means that you can't write polymorphic byte/string functions that are composable.

I would be -1 on that, and prefer that asciistr + asciistr --> asciistr.

You have to pretty much reimplement str to do that. I wouldn't say no to a patch that implemented it, but we're unlikely to do that much work ourselves for something which is primarily intended as a proof of concept. Cheers, Nick.

-- Greg

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 15 Jan 2014 08:00, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:

Guido van Rossum wrote:

...

def spam(a): r = asciistr('(') if a: r += a.strip() r += asciistr(')') return r

The general fix would be to add

else: r += a[:0]

The awkwardness might be reducable if asciistr let you write something like

r = asciistr('(', a)

meaning "give me either a string or bytes containing the value '(', depending on the type of a".

But taking a step back, how bad would it really be if an asciistr were returned in this case? Is it just that asciistr doesn't behave exactly like a str in all situations, so it might break something?

If so, would it help if asciistr were a built-in type, so that other things could be made aware of it?

That way lies the Python 2 text model, and we're not going there. It's probably best to think of asciistr as a way of demonstrating a rhetorical point about the superiority of the Python 3 text model rather than something that anyone should actually use in production Python 3 code (although, depending on how rough the edges turn out to be, it *might* eventually find a place in some single source 2/3 code bases, as well as in prototype code and personal scripts). Cheers, Nick.

-- Greg

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On Wed, Jan 15, 2014 at 01:03:13PM +1300, Greg Ewing wrote:

Nick Coghlan wrote:

...

That way lies the Python 2 text model, and we're not going there. It's probably best to think of asciistr as a way of demonstrating a rhetorical point about the superiority of the Python 3 text model

Hmmm... something like "The Python 3 text model is so superior that we have to use this weird hack to write something that makes perfectly good semantic sense but is very awkward to write otherwise" ?-)

I don't think mixing bytes and strings makes good semantic sense. If this discussion has taught me anything, it is that mixing the two is "Here Be Dragons" territory, fraught with danger. It may be that there are applications where mixing them is *unavoidable*, but I think that it's never *sensible*. -- Steven

Nick Coghlan writes:

"Give up" makes it sound like I got tired of arguing without being convinced rather than admitting I was just plain wrong.

I thought it was something in between (you explicitly said "lenient PEP 460" doesn't hurt you, but my understanding was you still believe that there's a safer way, and it's the latter you aren't going to try to convince folks of).

While I'll still work on the asciistr proposal,

Thank you for that. I really wish I had time to, myself, but not for several weeks... :-(

that's unrelated to PEP 460 - it's about making hybrid APIs less

"It" refers to asciistr or to PEP 460?

painful to write in Python 3 when you're willing to place the burden of ensuring ASCII compatibility of binary data on the calling code.

Versus what?

On 01/11/2014 10:32 AM, Antoine Pitrou wrote:

On Sat, 11 Jan 2014 18:41:49 +0100 Victor Stinner <victor.stinner@gmail.com> wrote:

...

b'x=%s' % 10 is well defined, it's pure bytes.

It is well-defined? Then please explain me what the general case of b'%s' % x is supposed to call:

This is the key question, isn't it?

- does it call x.__bytes__? int.__bytes__ doesn't exist

Perhaps that's the problem. According to the docs: ======================================================================== object.__bytes__(self) Called by bytes() to compute a byte-string representation of an object. This should return a bytes object. ======================================================================== Obviously, with the plethora of different binary possibilities for representing a number (how many bytes? endianness? which complement?), we would be well within our rights to decide that the "byte-string representation" of the numeric types is the ASCII equivalent of their __repr__ or __str__, and implement __bytes__ appropriately for them. Any other object that wants to be represented easily in a byte stream would also have to implement __bytes__. If necessary we could add __bytes__ to str for /strict/ ASCII conversion (even latin-1 would have to be explicitly encoded)[1]. -- ~Ethan~ [1] I'm iffy on this point as I'm not at all sure it's needed.

On 1/11/2014 1:50 PM, Ethan Furman wrote:

Perhaps that's the problem. According to the docs: ======================================================================== object.__bytes__(self)

Called by bytes() to compute a byte-string representation of an object. This should return a bytes object. ========================================================================

Obviously, with the plethora of different binary possibilities for representing a number (how many bytes? endianness? which complement?), we would be well within our rights to decide that the "byte-string representation" of the numeric types is the ASCII equivalent of their __repr__ or __str__, and implement __bytes__ appropriately for them. Any other object that wants to be represented easily in a byte stream would also have to implement __bytes__. If necessary we could add __bytes__ to str for /strict/ ASCII conversion (even latin-1 would have to be explicitly encoded)[1].

In spite of Victor's explanation of internals, which I didn't understand, this sounds like a very interesting idea, conceptually, that any object could implement its __bytes__representation. On the other hand, it would probably have to be parameterized in the general case: for binary data values, one protocol or format may wish the data to be big-endian, and another may wish the data to be little-endian; for str, one protocol or format may require one encoding and another may require a different encoding, even (as for email) for different parts of the message. So it could be somewhat complex, yet would be very powerful in allowing complex objects, made up of other objects, some of which might have a variety of potential bytes formats (think TIFF files, for example) to convert themselves into a stream of bytes that fits the standard. On the flip side, one would want to convert the stream of bytes into the set of objects, which is a parsing problem. This is a bit beyond what can be done automatically, just by calling __bytes__ with no parameters, though. What it may be, though, is a meta-operation from which the needed bytes operations can be determined. It may also not be an easy "compatible with existing Python 2 code with minor tweaks" solution, either. It would be more like a pickle protocol, but pickle defines its own formats, and thus is useless for creating standard formats. I guess it would belong on python-ideas.

On 12 January 2014 01:01, Victor Stinner <victor.stinner@gmail.com> wrote:

Supporting formating integers would allow to write b"Content-Length: %s\r\n" % 123, which would work on Python 2 and Python 3.

I'm surprised that no-one is mentioning b"Content-Length: %s\r\n" % str(123) which works on Python 2 and 3, is explicit, and needs no special-casing of int in the format code. Paul

On 12 January 2014 09:23, Georg Brandl <g.brandl@gmx.net> wrote:

...

On 12 January 2014 01:01, Victor Stinner <victor.stinner@gmail.com> wrote:

...

Supporting formating integers would allow to write b"Content-Length: %s\r\n" % 123, which would work on Python 2 and Python 3.

I'm surprised that no-one is mentioning b"Content-Length: %s\r\n" % str(123) which works on Python 2 and 3, is explicit, and needs no special-casing of int in the format code.

Certainly doesn't work on Python 3 right now, and never should :)

Sorry, I meant str(123).encode("ascii"), and I'd probably use a helper function for it. I could easily argue at this point that this is the type of bug that having %-formatting operations on bytes would encourage - %s means "format a string" (from years of C and Python (text) experience) so I automatically supply a string argument when using %s in a bytes formatting context. The reality is that I was probably just being sloppy, though :-) Paul

Paul Moore wrote:

I could easily argue at this point that this is the type of bug that having %-formatting operations on bytes would encourage - %s means "format a string" (from years of C and Python (text) experience) so I automatically supply a string argument when using %s in a bytes formatting context.

So don't call it %s -- call it something else such as %b. -- Greg

On 01/12/2014 01:06 PM, Greg Ewing wrote:

Paul Moore wrote:

...

I could easily argue at this point that this is the type of bug that having %-formatting operations on bytes would encourage - %s means "format a string" (from years of C and Python (text) experience) so I automatically supply a string argument when using %s in a bytes formatting context.

So don't call it %s -- call it something else such as %b.

Which is fine for 3.5+ code, but not at all helpful for a 2/3 code base. -- ~Ethan~

On 13 Jan 2014 01:22, "Kristján Valur Jónsson" <kristjan@ccpgames.com> wrote:

Well, my suggestion would that we _should_ make it work, by having the %s

format specifyer on bytes objects mean: str(arg).encode('ascii', 'strict')

It would be an explicit encoding operator with a known, fixed, and well specified encoder. This would cover most of the use cases seen in this threadnought. Others could be handled with explicit str formatting and encoding.

Imho, this is not equivalent to re-introducing automatic type conversion between binary/unicode, it is adding a specific convenience function for explicitly asking for ASCII encoding.

It is not explicit, it is implicit - whether or not the resulting string assumes ASCII compatibility or not depends on whether you pass a binary value (no assumption) or a string value (assumes ASCII compatibility). This kind of data driven change in assumptions about correctness is utterly unacceptable in the core text and binary types in Python 3. It's also completely unnecessary - asciistr will be a third party extension type that allows those users pining for the halcyon days of the Python 2 str type to stop harassing the core devs with requests to compromise the core Python 3 text model with implicit encoding operations. I'll ensure any interoperability bugs between asciistr and the core types that can't be worked around get fixed. A separate type is genuinely explicit (since the ASCII assumption is no longer hidden from the type system), and allows much simpler interoperability for code that wants (indexing asciistr will eventually produce length 1 asciistr instances instead of str instances, it will avoid the bytes(intval) discrepancy, it will avoid the str(bytesval) problem, etc). I've been suggesting for years that Python 3 might need a third type (not required to be a builtin, since it's so specialised), but folks migrating from Python 2 have been so focused on making the core binary type a hybrid type again, the notion of taking advantage of PEP 393 to create a dedicated extension type specifically for working with ASCII compatible binary protocols has failed to compute. I'm hoping a test suite and preliminary implementation will help more people to finally get the point. Regards, Nick.

K ________________________________________ From: Python-Dev [python-dev-bounces+kristjan=ccpgames.com@python.org] on

behalf of Georg Brandl [g.brandl@gmx.net]

Sent: Sunday, January 12, 2014 09:23 To: python-dev@python.org Subject: Re: [Python-Dev] PEP 460: allowing %d and %f and mojibake

Am 12.01.2014 09:57, schrieb Paul Moore:

...

On 12 January 2014 01:01, Victor Stinner <victor.stinner@gmail.com> wrote:

...

Supporting formating integers would allow to write b"Content-Length: %s\r\n" % 123, which would work on Python 2 and Python 3.

I'm surprised that no-one is mentioning b"Content-Length: %s\r\n" % str(123) which works on Python 2 and 3, is explicit, and needs no special-casing of int in the format code.

Certainly doesn't work on Python 3 right now, and never should :)

Georg _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 01/12/2014 08:21 AM, Ethan Furman wrote:

On 01/12/2014 08:09 AM, Nick Coghlan wrote:

...

On 13 Jan 2014 01:22, "Kristján Valur Jónsson" wrote:

...

Imho, this is not equivalent to re-introducing automatic type conversion between binary/unicode, it is adding a specific convenience function for explicitly asking for ASCII encoding.

It is not explicit, it is implicit - whether or not the resulting string assumes ASCII compatibility or not depends on whether you pass a binary value (no assumption) or a string value (assumes ASCII compatibility).

Nick, I don't understand what you are saying here. Are you saying that the result of b'%s' % var may be either a bytes object or a str object? Because that would be wrong -- it would always be a bytes object.

Okay, I just went and took a closer look at the asciistr type [1]. For what it's worth I don't think this is Antoine's understanding of what we [2] are asking for, nor is it what we are asking for (I'm sure Antoine will correct me if I'm wrong. ;) We know full well the difference between unicode and bytes, and we know full well that numbers and much of the text we need has an ASCII (bytes!) representation. When we do a b'Content Length: %d' % len(binary_data) we are expecting to get back a bytes object, /not/ a unicode object. Your asciistr, which sometimes returns bytes and sometimes returns text, is absolutely *not* what we want. -- ~Ethan~ [1] https://github.com/jeamland/asciicompat [2] the dbf and pdf folks, at least

On 01/12/2014 09:26 AM, Paul Moore wrote:

On 12 January 2014 17:03, Ethan Furman <ethan@stoneleaf.us> wrote:

...

We know full well the difference between unicode and bytes, and we know full well that numbers and much of the text we need has an ASCII (bytes!) representation. When we do a b'Content Length: %d' % len(binary_data) we are expecting to get back a bytes object, /not/ a unicode object.

What I am struggling to understand here is what room for compromise there is. Clearly, for whatever reason,

b'Content Length: ' + str(len(binary_data)).encode('ascii'))

is not acceptable for you. OK, fair enough. Also, apparently, writing a helper

def int_to_bytes(n): return str(n).encode('ascii')

b'Content Length: ' + int_to_bytes(len(binary_data))

is unacceptable. But I'm not clear why it's unacceptable. Maybe I missed the explanation - God knows, the thread is long enough :-)

True enough! ;) It's unacceptable in the sense that the bytes type is /almost/ there, it's /almost/ what is needed to handle the boundary conditions. We have a __bytes__ method (how is it supposed to be used?) that could be made to fit the interpolation bill. It seems to me the core of Nick's refusal is the (and I agree!) rejection of bytes interpolation returning unicode -- but that's not what I'm asking for! I'm asking for it to return bytes, with the interpolated data (in the case if %d, %s, etc) being strictly-ASCII encoded.

On the other hand, Nick has explained why b'Content Length: %d' % len(binary_data) is unacceptable to him (you don't have to agree with his opinion, just concede that he has explained his position in a way that you understand).

Only because he (or Benno) finally wrote some tests and I was able to see what he thought I was wanting. Which does seem to leave a *tiny* bit of wiggle room if bytes interpolation always return bytes, and never a unicode (yeah, I know, snowball's chance and all that).

I'm not trying to argue you're wrong - I don't know your codebase, nor do I know your application area. But surely somewhere between "we must have % formatting including %d for bytes" and the above, there's a middle ground that you *are* willing to accept? Can you give any indications of what that might be? What, specifically, about the helper function is the problem? I don't think it is any less space efficient, it doesn't double-encode, and I don't think it's more difficult to understand (although it is a little longer, it trades that off against being a bit more explicit as to what's going on). Surely you're not arguing that your code must work unchanged (not "there's a way of writing the code so it works on Python 2 and 3", but "the code you currently have for Python 2 must work with no changes at all")?

I'm arguing from three PoVs: 1) 2 & 3 compatible code base 2) having the bytes type /be/ the boundary type 3) readable code

Can you give an example of code that is *nearly* acceptable to you, which works in Python 2 and 3 today, and explain what improvements you would like to see to it in order to use it instead of waiting for a core change?

I'm not trying to be difficult (just naturally good at it, I guess ;) , but I don't see a lot room for compromises -- I would like % interpolation, I'm told I have to use a helper function. I will if I have to, but first I have to try and make myself understood, and I'm not sure that has happened yet. Following Nick's example I'm writing up some tests that clearly show what I would like to see. Then at least we can debate what I'm actually asking for, and now what the (understandably) unicode-what-a-mess-we-had-in-py2k-don't-want-again that some think I am asking for. -- ~Ethan~

On 01/12/2014 11:00 AM, Paul Moore wrote:

And yet I still don't follow what you *want*. Unless it's that b'%d' % (12,) must work and give b'12', and nothing else is acceptable.

Nothing else is ideal. I'll go that route if I have to. I understand that in the real world you go with what works, but in the development stage you fight for the ideal. :)

My reading of Nick's refusal is that %d takes a value which is semantically a number, converts it into a base-10 representation (which is semantically a *string*, not a sequence of bytes[1]) and then *encodes* that string into a series of bytes using the ASCII encoding. That is *two* semantic transformations, and one (the ASCII encoding) is *implicit*. Specifically, it's implicit because (a) the normal reading of %d is "produce the base-10 representation of a number, and a base-10 representation is a *string*, and (b) because nowhere has ASCII been mentioned (why not UTF16? that would be entirely plausible for a wchar-based environment like Windows). And a core principle of the bytes/text separation in Python 3 is that encoding should never happen implicitly.

That could be. And yet the bytes type already has several concessions to ASCII encoding.

By the way, I should point out that I would never have understood *any* of the ideas involved in this thread before Python 3 forced me to think about Unicode and the distinction between text and bytes. And yet, I now find myself, in my (non-Python) work environment, being the local expert whenever applications screw up text encodings. So I, for one, am very grateful for Python 3's clear separation of bytes and text. (And if I sometimes come across as over-dogmatic, I apologise - put it down to the enthusiasm of the recent convert :-))

No worries. I was forced to learn the difference when I wrote my dbf module for 2.5. Took longer than I'd like to admit to realize that ASCII was an encoding. :/

[1] If you cannot see that there's no essential reason why the base-10 representation '123' should correspond to the bytes b'\x31\x32\x33' then you are probably not old enough to have started programming on EBCDIC-based computers :-)

I can see it. :) But bytes already acknowledges an ASCII bias. ;) And even EBCDIC machines speak ASCII when talking telnet. -- ~Ethan~

Glenn Linderman writes:

the proposals to embed binary in Unicode by abusing Latin-1 encoding.

Those aren't "proposals", they are currently feasible techniques in Python 3 for *some* use cases. The question is why infecting Python 3 with the byte/character confoundance virus is preferable to such techniques, especially if their (serious!) deficiencies are removed by creating a new type such as asciistr.

Glenn Linderman writes:

On 1/12/2014 4:08 PM, Stephen J. Turnbull wrote:

...

Glenn Linderman writes:

...

the proposals to embed binary in Unicode by abusing Latin-1 encoding.

...

Those aren't "proposals", they are currently feasible techniques in Python 3 for *some* use cases. The question is why infecting Python 3 with the byte/character confoundance virus is preferable to such techniques, especially if their (serious!) deficiencies are removed by creating a new type such as asciistr.

"smuggled binary" (great term borrowed from a different subthread) muddies the waters of what you are dealing with.

Not really. The "mud" is one or more of the serious deficiencies. It can be removed, I believe (and Nick apparently does, too). "asciistr" is one way to try that.

When the mixture of text and binary is done as encoded text in binary, then it is obvious that only limited text processing can be performed,

Hardly. After all, that's how all text processing was done for decades. Still is, in some programs, especially C programs.

And there are no extra, confusing Latin-1 encode/decode operations required.

The "extra" encode/decode operations are mostly (perhaps all) due to examples that started from bytes and end with bytes. Of course if you assume that API and propose to do the operations using Unicode, you'll get "extra" decode/encode operations.

From a higher-level perspective, I think it would be great to have a module, perhaps called "boundary" (let's call it that for now), that allow some definition syntax (augmented BNF? augmented ABNF?) to explain the format of a binary blob.

We have struct, for one. I'm not sure why you want more than that. I suppose you could go all the way to ASN.1.

Glenn Linderman writes:

On 1/13/2014 6:43 AM, Stephen J. Turnbull wrote:

...

Glenn Linderman writes:

...

...

"smuggled binary" (great term borrowed from a different subthread) muddies the waters of what you are dealing with.

...

Not really. The "mud" is one or more of the serious deficiencies. It can be removed, I believe (and Nick apparently does, too). "asciistr" is one way to try that.

Yes really. Use of smuggled binary means the str containing it can no longer be treated completely as a str. That is "muddier" than having a str that is only a str.

You don't seem to understand what *asciistr* is: it's a *different type* that is simultaneously compatible in operation with bytes and str, by automatically converting to whichever it is used with. If we used asciistr, str would no longer be muddy (except in cases where we would have used surrogateescape anyway). You also don't seem to understand that bytes are conceptually pure mud. Anything that is pushed to bytes because you don't know what type it is (or because at the time the program is written, the type can't be known) is no longer subject to duck-typing. So the question is "how is mud best handled?" Obviously, incorporating it in str with .decode('latin1') is inappropriate. However, if you use .decode('ascii') you have your choice of error handlers. If you use errors='strict' then no mud can get in. Use of any other error handler is obviously a "consenting adults" behavior; it should only be done when you expect that you can keep the muddy str from leaking into places where it might be passed to an I/O function. (Note that the internal processing of an application that never outputs such a str is completely conformant to the Unicode Standard. That's not a goal of Python, since surrogateescape is designed to be used on output too. But if the developer applies that standard to each *program component*, he's going to be in pretty good shape.) If you use asciistr, then you're pretty much in complete control. The exception is operations that munge individual characters (case conversion). If you have a protocol with ASCII keywords but their case is specified, you'll need to define another type to remove the case-munging methods if you want that level of safety. If, as in your proposal, bytes are tagged with descriptions, you are effectively creating types on the fly. But if the program doesn't anticipate that, they're mud. If the program doesn't anticipate all of them those descriptions that are unhandled become mud, too. ITSM that the "syntax descriptor" feature is already present in Python, and it's called "class". So, IMHO, simply converting to an appropriate Python type on input is what should be done, but in any case, I don't see how adding a "syntax descriptor" attribute to bytes is going to improve the situation significantly. Note that such a class can postpone parsing for efficiency or lack of information reasons, and store the object as bytes until needed. But this is not the same as passing around naked bytes, because the class can ensure that bytes can't get out, only parsed objects.

On 01/13/2014 09:06 PM, Guido van Rossum wrote:

Sorry to butt in, but can you post a link to the asciistr code? Google has too many hits for other things to be useful to find it, it seems.

https://github.com/jeamland/asciicompat -- ~Ethan~

On 01/13/2014 09:12 PM, Ethan Furman wrote:

On 01/13/2014 09:06 PM, Guido van Rossum wrote:

...

In contrast, here's the tests I drew up for what I thought bytes should do for us (no code, just tests):

https://bitbucket.org/stoneleaf/bytestring

Ugh. Ignore for now, I need to update them to reflect the recent developments. :/ -- ~Ethan~

On 1/13/2014 8:58 PM, Stephen J. Turnbull wrote:

Glenn Linderman writes:

...

On 1/13/2014 6:43 AM, Stephen J. Turnbull wrote:

...

Glenn Linderman writes:

...

...

...

"smuggled binary" (great term borrowed from a different subthread) muddies the waters of what you are dealing with.

...

...

Not really. The "mud" is one or more of the serious deficiencies. It can be removed, I believe (and Nick apparently does, too). "asciistr" is one way to try that.

...

Yes really. Use of smuggled binary means the str containing it can no longer be treated completely as a str. That is "muddier" than having a str that is only a str.

You don't seem to understand what *asciistr* is: it's a *different type* that is simultaneously compatible in operation with bytes and str, by automatically converting to whichever it is used with. If we used asciistr, str would no longer be muddy (except in cases where we would have used surrogateescape anyway).

No, I haven't fully understood what asciistr is, only Nick's several descriptions of it. I do understand it is a different type, and can interact with both bytes and str. If it automatically converts, then it sounds terribly inefficient with long data, but I didn't hear Nick say that, but maybe I missed it. You mentioned asciistr in the snippet above, but most of what you have been writing about smuggled binary was using str... I hadn't grokked that you were now a full-fledged proponent of asciistr, and were now proposing to put your smuggled binary into asciistr.

You also don't seem to understand that bytes are conceptually pure mud. Anything that is pushed to bytes because you don't know what type it is (or because at the time the program is written, the type can't be known) is no longer subject to duck-typing.

If you are talking str, then bytes are mud. If you are talking bytes, then str is mud. I'm wouldn't think of "pushing something to bytes" (whatever that means) because I don't know what it is... I may manipulate bytes because I know what they are, and that is the most appropriate form for that piece of data for the present manipulations; if something is text, I want to transform the bytes to str if I need to manipulate it, parse it, or present it. If I don't know what something is, it is because it didn't meet my expectations of what it should be, and I want to present an error, which may include some representation (probably hex) of some of the bytes that cannot be understood. But if I'm "pushing to bytes", which I would interpret as creating a byte stream, then I know what I have, and I need to convert it to bytes either to store it in a file, or communicate it to another process. That's far from not knowing what it is.

So the question is "how is mud best handled?" Obviously, incorporating it in str with .decode('latin1') is inappropriate.

Glad to hear you say that; I thought that was what you were promoting, when you said, in an earlier message: On 1/12/2014 4:08 PM, Stephen J. Turnbull wrote:

Glenn Linderman writes:

...

the proposals to embed binary in Unicode by abusing Latin-1 encoding.

Those aren't "proposals", they are currently feasible techniques in Python 3 for*some* use cases.

Back to this one, though.

However, if you use .decode('ascii') you have your choice of error handlers. If you use errors='strict' then no mud can get in. Use of any other error handler is obviously a "consenting adults" behavior; it should only be done when you expect that you can keep the muddy str from leaking into places where it might be passed to an I/O function. (Note that the internal processing of an application that never outputs such a str is completely conformant to the Unicode Standard. That's not a goal of Python, since surrogateescape is designed to be used on output too. But if the developer applies that standard to each *program component*, he's going to be in pretty good shape.)

If you use asciistr, then you're pretty much in complete control. The exception is operations that munge individual characters (case conversion). If you have a protocol with ASCII keywords but their case is specified, you'll need to define another type to remove the case-munging methods if you want that level of safety.

The above doesn't sound like a use case I care about, much. If I get a garbled file without an accurate definition of what it contains, then I probably want to stick it in the trash. The only "processing" that can be done is to pass on the garbage to someone else, and stink up their system, and that can be done purely as bytes.

If, as in your proposal, bytes are tagged with descriptions, you are effectively creating types on the fly. But if the program doesn't anticipate that, they're mud.

Interpreting a file format or wire protocol requires parsing and manipulating an incoming byte stream, and converting it to useful types in the program... if it can't be converted to useful types, then why bother parsing it? So the rest of my discussion was not talking about creating types on the fly, but on a systematic way of converting a well-specified byte stream (file format, or wire protocol) to a collection of useful types, in an organized manner, that might be verifiable, rather than with ad-hoc coding. And similarly in reverse... after manipulating the objects to perform useful transformations, possibly based on user input (that's what a program does), then to write them back out to a byte stream in modified form, in an organized manner, that might be verifiable, rather than with ad-hoc coding.

If the program doesn't anticipate all of them those descriptions that are unhandled become mud, too. ITSM that the "syntax descriptor" feature is already present in Python, and it's called "class". So, IMHO, simply converting to an appropriate Python type on input is what should be done, but in any case, I don't see how adding a "syntax descriptor" attribute to bytes is going to improve the situation significantly.

Syntax descriptors would be a description of the substructures of a file format (think TIFF files) or wire protocol, and might allow parsing of binary files similarly to the way computer languages are parsed, producing errors when encountering mud. What you dismiss as "converting to an appropriate Python type on input" can be quite complex when for complex file formats, but it is the process of converting to such a heirarchy of Python objects that was to be described by the syntax descriptors.

Note that such a class can postpone parsing for efficiency or lack of information reasons, and store the object as bytes until needed. But this is not the same as passing around naked bytes, because the class can ensure that bytes can't get out, only parsed objects.

Sure, it could. My proposal is suggesting that the distribution of bytes to objects in a hierarchy might be automated in the sense of parsing the binary format, so that instead of writing "a class" for the whole, that class would be pre-written, based on the syntax description of the file, and matching that with the syntax descriptions of the component types. It is really a topic for python ideas, to flesh it out further, but it seemed related, as a use case, a class that would live on the bytes processing boundary, producing other objects, some of which may be text strings, in an organized, probably hierarchical, collection of objects.

On 1/12/2014 2:57 PM, Stephen J. Turnbull wrote:

...

But bytes already acknowledges an ASCII bias.

True, but that bias is implemented without use of encoding or decoding. b'%d' % (123,) -> b'123' does require encoding, at the very least in the sense of type change and serialization. b'%d' all by itself, even before using the % operator, does require encoding, at the very list in the sense of type change and serialization.

Ethan Furman writes:

On 01/12/2014 02:57 PM, Stephen J. Turnbull wrote:

...

No, Nick's point is that there's no encoding needed there are all, just a bunch of methods that handle numbers in the range 0-255. You can rationalize the particular choice of numbers by referring to the ASCII coded character set, and that's very useful to users. But knowledge of ASCII isn't necessary to specify these methods; they can be defined in an encoding/decoding-free way.

How can you say that with a straight face? [1]

Because I showed you code that does it. Did you see an .encode or a .decode in there?

Do you really think that .title, .isalnum, and .center (to name only a few) would work the same if the assumed encoding was EBCIDC?

Yes, yes, and yes. The numbers involved would change, and the test for finding letters would be different (and more complicated IIRC). The only one to worry about is .title, but neither ASCII nor EBCDIC has confused or multiple letter titlecase.

Do you think they would do the proper transformations, or return the proper result, if the bytes they were used on were encoded Japanese?

That depends on which Japanese encoding. It would work correctly on UTF-8 and on EUC-JP (packed), and not on any of the others. But you wouldn't consider that "ASCII-encoded text", would you?

...

...

But bytes already acknowledges an ASCII bias.

True, but that bias is implemented without use of encoding or decoding. b'%d' % (123,) -> b'123' does require encoding, at the very least in the sense of type change and serialization.

You mean like changing a number into text does? Really, this is no different.

Precisely. "There should be one- and preferably only one -way to do it." The one way uses text, so preferably bytes shouldn't.

I want to add one more PoV: small performance regression, especially on Python 2. Because programs that needs byte formatting may be low level and used heavily from application. Many programs uses one source approach to support Python 3. And supporting Python 3 should not means large performance regression on Python 2. In Python 2: In [1]: def int_to_bytes(n): ...: return unicode(n).encode('ascii') ...: In [2]: %timeit int_to_bytes(42) 1000000 loops, best of 3: 691 ns per loop In [3]: %timeit b'Content-Type: ' + int int int_to_bytes intern In [3]: %timeit b'Content-Type: ' + int_to_bytes(42) 1000000 loops, best of 3: 737 ns per loop In [4]: %timeit b'Content-Type: %d' % 42 10000000 loops, best of 3: 20.2 ns per loop In [5]: %timeit (u'Content-Type: %d' % 42).encode('ascii') 1000000 loops, best of 3: 381 ns per loop In Python 3: In [1]: def int_to_bytes(n): ...: return str(n).encode('ascii') ...: In [2]: %timeit int_to_bytes(42) 1000000 loops, best of 3: 612 ns per loop In [3]: %timeit b'Content-Type: ' + int_to_bytes(42) 1000000 loops, best of 3: 668 ns per loop In [4]: %timeit ('Content-Type: %d' % 42).encode('ascii') 1000000 loops, best of 3: 326 ns per loop

I'm arguing from three PoVs:

...

1) 2 & 3 compatible code base 2) having the bytes type /be/ the boundary type 3) readable code

The only one of these that I can see being in any way an argument against

def int_to_bytes(n): return str(n).encode('ascii')

b'Content Length: ' + int_to_bytes(len(binary_data))

is (3), and that's largely subjective. Personally, I see very little difference between the above and %d-interpolation in terms of *readability*. Brevity, certainly %d wins. But that's not important on its own, and I'd argue that my version is more clear in terms of describing the intent (and would be even better if I wasn't rubbish at thinking of function names, or if this wasn't in isolation, and more application-focused functions were used).

...

It seems to me the core of Nick's refusal is the (and I agree!) rejection of bytes interpolation returning unicode -- but that's not what I'm asking for! I'm asking for it to return bytes, with the interpolated data (in the case if %d, %s, etc) being strictly-ASCII encoded.

My reading of Nick's refusal is that %d takes a value which is semantically a number, converts it into a base-10 representation (which is semantically a *string*, not a sequence of bytes[1]) and then *encodes* that string into a series of bytes using the ASCII encoding. That is *two* semantic transformations, and one (the ASCII encoding) is *implicit*. Specifically, it's implicit because (a) the normal reading of %d is "produce the base-10 representation of a number, and a base-10 representation is a *string*, and (b) because nowhere has ASCII been mentioned (why not UTF16? that would be entirely plausible for a wchar-based environment like Windows). And a core principle of the bytes/text separation in Python 3 is that encoding should never happen implicitly.

By the way, I should point out that I would never have understood *any* of the ideas involved in this thread before Python 3 forced me to think about Unicode and the distinction between text and bytes. And yet, I now find myself, in my (non-Python) work environment, being the local expert whenever applications screw up text encodings. So I, for one, am very grateful for Python 3's clear separation of bytes and text. (And if I sometimes come across as over-dogmatic, I apologise - put it down to the enthusiasm of the recent convert :-))

Paul

[1] If you cannot see that there's no essential reason why the base-10 representation '123' should correspond to the bytes b'\x31\x32\x33' then you are probably not old enough to have started programming on EBCDIC-based computers :-) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com

-- INADA Naoki <songofacandy@gmail.com>

On 12 January 2014 22:10, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

I think the readability argument becomes a bit sharper when you consider more complex examples, e.g. if I have a tuple of 3 floats that I want to put into a PDF file, then

b"%f %f %f" % my_floats

is considerably clearer than

b" ".join((float_to_bytes(f) for f in my_floats))

Hmm, I'm not sure I'd agree. I'd quote "explicit is better than implicit", but given comments below, that would be a mistake :-) Let's just leave it that I'd probably wrap the whole thing in a float_list(floats) function in my application, and not *care* how it was implemented. One thing that this does bring up, though, is that all the talk is about %-formatting. Do the people who are arguing for numeric formatting have views on what (if any) features will be included in bytes.format()? It seems to me that recasting many of the discussions using format() make it much less "obvious" that adding the features to bytes formatting is a reasonable thing to do. I won't give specific examples, because I would be putting words into people's mouths. But I *would* say that any genuine proposal for numeric formatting in bytes should be cast as a formal PEP and explicitly document both % and format() behaviours.

It's indicated (I won't say "implied", see below) by the fact that we're interpolating it into a bytes object rather than a string.

This is no more or less implicit than the fact that when we write

b"ABC"

then we're saying that those characters are to be encoded in ASCII, and not EBCDIC or UTF-16 or...

That's a fair point, and one I had not taken into consideration.

BTW, there's a problem with bandying around the words "implicit" and "explicit", because they depend on your frame of reference. For example, one person might say that the fact that b"%s" encodes into ASCII is implicit, because ASCII isn't written down in the code anywhere. But another person might say it's explicit, because the manual explicitly says that stuff interpolated into a bytes object is encoded as ASCII.

In my defense, I would say that I was trying to clarify Nick's objections, and it's entirely possible I misrepresented this aspect of them. Personally, I agree that it's not as black and white as simply saying "numeric formatting is wrong", but I think that the fact that %d et al represent a "double transformation" (from number to string representation to encoded bytes) is the differentiating factor here. Proposals that do nothing but interpolation are essentially convenience wrappers for various combinations of concatenation and join. Adding "double transformation" formatting codes is a step change, and needs to be explicitly acknowledged and justified. (If you *do* manage to justify such codes, there's a secondary question of precisely what codes should be supported, but we can start by getting agreement that the *class* of codes is allowed). PEP 460 explicitly excludes anything but pure interpolation.

So arguments of the form "X is bad because it's not explicit" are prone to getting people talking past each other.

Fair point. I hope my above paragraph clarifies my position somewhat better. Paul

On 12 January 2014 19:30, Emile van Sebille <emile@fenx.com> wrote:

len(open('chars','wb').write("".join(map (chr,range(256)))).read())

Python 2:

...

...

len(open('chars','wb').write("".join(map (chr,range(256)))).read()) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'NoneType' object has no attribute 'read'

I could be facetous and say "None.read", but more seriously, what are you trying to say here? How do I write a 256-byte file with one byte for each value? bytes(range(256)) gives you the bytestring you want. I simply don't see your point here.

...

And yet I still don't follow what you *want*. Unless it's that b'%d' % (12,) must work and give b'12', and nothing else is acceptable.

Nothing else is ideal. I'll go that route if I have to. I understand that in the real world you go with what works, but in the development stage you fight for the ideal. :)

OK, but can you fight by giving arguments as to why it's better than the plethora of alternatives that have been suggested? Or counter-arguments to the objections that have been raised to the proposal? Paul

Am 12.01.2014 20:30, schrieb Emile van Sebille:

On 01/12/2014 09:26 AM, Paul Moore wrote:

...

Can you give an example of code that is *nearly* acceptable to you, which works in Python 2 and 3 today, and explain what improvements you would like to see to it in order to use it instead of waiting for a core change?

I'm not a developer, but I'm trying to understand how in v3 I accomplish what in v2 is easy:

len(open('chars','wb').write("".join(map (chr,range(256)))).read())

What's the v3 equivalent?

That's actually very easy and shows a strength of the bytes type, since there's no text involved: open('chars', 'wb').write(bytes(range(256))) Georg

On 12/01/2014 17:03, Ethan Furman wrote:

On 01/12/2014 08:21 AM, Ethan Furman wrote:

...

On 01/12/2014 08:09 AM, Nick Coghlan wrote:

...

On 13 Jan 2014 01:22, "Kristján Valur Jónsson" wrote:

...

Imho, this is not equivalent to re-introducing automatic type conversion between binary/unicode, it is adding a specific convenience function for explicitly asking for ASCII encoding.

It is not explicit, it is implicit - whether or not the resulting string assumes ASCII compatibility or not depends on whether you pass a binary value (no assumption) or a string value (assumes ASCII compatibility).

Nick, I don't understand what you are saying here. Are you saying that the result of b'%s' % var may be either a bytes object or a str object? Because that would be wrong -- it would always be a bytes object.

Okay, I just went and took a closer look at the asciistr type [1]. For what it's worth I don't think this is Antoine's understanding of what we [2] are asking for, nor is it what we are asking for (I'm sure Antoine will correct me if I'm wrong. ;)

We know full well the difference between unicode and bytes, and we know full well that numbers and much of the text we need has an ASCII (bytes!) representation. When we do a b'Content Length: %d' % len(binary_data) we are expecting to get back a bytes object, /not/ a unicode object.

Your asciistr, which sometimes returns bytes and sometimes returns text, is absolutely *not* what we want.

I've just tried asciistr using your test code (having corrected the typo, it's assertIsInstance, not assertIsinstance :) and it looks like a very good starting point. Have you, or anyone else for that matter, actually tried asciistr out?

-- ~Ethan~

[1] https://github.com/jeamland/asciicompat [2] the dbf and pdf folks, at least

-- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

Now you're just splitting hairs, Nick. An explicit operator, %s, _defined_ to be "encode a string object using strict ascii", how is that any less explicit than the .encode('ascii', 'strict') spelt out in full? The language is full of constructs that are shorthands for others, more lengthy but equivalent things. I mean, basically what I am suggesting is that in addition to %b with def helper(o): return str(o).encode('ascii', 'strict') b'foo%bbar'%(helper(myobj), ) you have b'foo%sbar'%(myobj, ) There is no "data driven change in assumptions." Just an interpolation operator with a clearly defined meaning. I don't think anyone is trying to compromise the text model. All people are asking for is that the _boundary_ is made a little easier to deal with. K ________________________________ From: Nick Coghlan [ncoghlan@gmail.com] Sent: Sunday, January 12, 2014 16:09 To: Kristján Valur Jónsson Cc: python-dev@python.org; Georg Brandl Subject: Re: [Python-Dev] PEP 460: allowing %d and %f and mojibake It is not explicit, it is implicit - whether or not the resulting string assumes ASCII compatibility or not depends on whether you pass a binary value (no assumption) or a string value (assumes ASCII compatibility). This kind of data driven change in assumptions about correctness is utterly unacceptable in the core text and binary types in Python 3.

Right. I'm saying, let's support two interpolators only: %b interpolates a bytes object (or one supporting the charbuffer interface) into a bytes object. %s interpolates a str object by first converting to a bytes object using strict ascii conversion. This makes it very explicit what we are trying to do. I think that using %s to interpolate a bytes object like the current PEP does is a bad idea, because %s already means 'str' elsewhere in the language, both in 2.7 and 3.x As for the case you mention: b"abc %s" % (b"def",) -> b"abc def" b"abc %s" % (b"def",) -> b"abc b\"def\"" # because str(bytesobject) == repr(bytesobject) This is perfectly fine, imho. Let's not overload %s to mean "bytes" in format strings if those format strnings are in fact not strings byt bytes. That way madness lies. K ________________________________________ From: Paul Moore [p.f.moore@gmail.com] Sent: Sunday, January 12, 2014 17:04 To: Kristján Valur Jónsson Cc: Nick Coghlan; Georg Brandl; python-dev@python.org Subject: Re: [Python-Dev] PEP 460: allowing %d and %f and mojibake On 12 January 2014 16:52, Kristján Valur Jónsson <kristjan@ccpgames.com> wrote: But that's not what the current PEP says. It uses %s for interpolating bytes values. It looks like you're saying that b'abc %s' % (b'def') will *not* produce b'abc def', but rather will produce b'abc b\'def\'' (because str(b'def'') is "b'def'").

On 12/01/14 16:52, Kristján Valur Jónsson wrote:

Now you're just splitting hairs, Nick.

An explicit operator, %s, _defined_ to be "encode a string object using strict ascii",

I don't like this because '%s' reads to me as "insert *string* here". I think '%a' which reads as "encode as ASCII and insert here" would be better.

how is that any less explicit than the .encode('ascii', 'strict') spelt out in full? The language is full of constructs that are shorthands for others, more lengthy but equivalent things.

I mean, basically what I am suggesting is that in addition to %b with

def helper(o):

return str(o).encode('ascii', 'strict')

b'foo*%b*bar'%(helper(myobj), )

you have

b'foo*%s*bar'%(myobj, )

There is no "data driven change in assumptions." Just an interpolation operator with a clearly defined meaning.

I don't think anyone is trying to compromise the text model. All people are asking for is that the _boundary_ is made a little easier to deal with.

K

------------------------------------------------------------------------ *From:* Nick Coghlan [ncoghlan@gmail.com] *Sent:* Sunday, January 12, 2014 16:09 *To:* Kristján Valur Jónsson *Cc:* python-dev@python.org; Georg Brandl *Subject:* Re: [Python-Dev] PEP 460: allowing %d and %f and mojibake

It is not explicit, it is implicit - whether or not the resulting string assumes ASCII compatibility or not depends on whether you pass a binary value (no assumption) or a string value (assumes ASCII compatibility). This kind of data driven change in assumptions about correctness is utterly unacceptable in the core text and binary types in Python 3.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org

Mark Lawrence wrote:

I entirely agree. This would also parallel the conversion flags given here http://docs.python.org/3/library/string.html#format-string-syntax, I quote "Three conversion flags are currently supported: '!s' which calls str() on the value, '!r' which calls repr() and '!a' which calls ascii()".

Except that ascii() does something rather different -- it's a variation on repr() rather than str(), and it doesn't imply any encoding operation. I think this parallel would be more confusing than helpful. -- Greg

Nick Coghlan wrote:

On 13 Jan 2014 01:22, "Kristján Valur Jónsson" <kristjan@ccpgames.com <mailto:kristjan@ccpgames.com>> wrote:

...

Well, my suggestion would that we _should_ make it work, by having the %s format specifyer on bytes objects mean: str(arg).encode('ascii', 'strict')

It is not explicit, it is implicit - whether or not the resulting string assumes ASCII compatibility or not depends on whether you pass a binary value (no assumption) or a string value (assumes ASCII compatibility).

How do you make that out? As far as I can see, Kristjan's proposal will *always* call str() on the argument of a %s format, regardless of its type. The *result* of that str() is then *required* (not assumed) to be encodable as ascii. I don't see any type-dependent changes in behaviour here. Interpolating a bytes object as-is, without a conversion to text, should be done by a different format specifier, such as %b. All text/bytes conversions are then explicit: if you write %s, then you're encoding something as ascii, but if you write %b, you're just inserting something that's already binary. -- Greg

11.01.14 21:40, Kristján Valur Jónsson написав(ла):

How about a compromise? Personally, I think adding the full complement of integer/float formatting to bytes is a bit over the top. How about just supporting two format specifiers? %b : interpolate a bytes object. If it doesn't have the buffer interface, error. %s : interpolate a str object, encoded to ASCII using 'strict' conversion.

%b is not supported in Python 2.7. And compatibility with Python 2.7 is only the purpose of this feature.

No, I don't think it is. The purpose is to make it easier to work with bytes objects. There can be no python 2 compatibility when it comes to bytes/unicode conversion. ________________________________________ From: Python-Dev [python-dev-bounces+kristjan=ccpgames.com@python.org] on behalf of Serhiy Storchaka [storchaka@gmail.com] Sent: Saturday, January 11, 2014 21:01 To: python-dev@python.org Subject: Re: [Python-Dev] PEP 460: allowing %d and %f and mojibake 11.01.14 21:40, Kristján Valur Jónsson написав(ла):

How about a compromise? Personally, I think adding the full complement of integer/float formatting to bytes is a bit over the top. How about just supporting two format specifiers? %b : interpolate a bytes object. If it doesn't have the buffer interface, error. %s : interpolate a str object, encoded to ASCII using 'strict' conversion.

%b is not supported in Python 2.7. And compatibility with Python 2.7 is only the purpose of this feature.

On 2014-01-11 22:09, Nick Coghlan wrote:

For Python 2 folks trying to grok where the "bright line" is in terms of the Python 3 text model: if your proposal includes *any* kind of implicit serialisation of non binary data to binary, it is going to be rejected as an addition to the core bytes type. If it avoids crossing that line (as the buffer-API-only version of PEP 460 does), then we can talk.

To take such a hard-line stance, I would expect you to author a PEP to strip the ASCII conveniences from the bytes and bytearray types. Otherwise, I find it a bit schizophrenic to argue that methods like lower, upper, title, and etc. don't implicitly assume encoding:

...

...

a = "scott".encode('utf-16') b = a.title() c = b.decode('utf-16') 'SCOTT'

So, clearly title() not only depends on the bytes characters encoded in a superset of ASCII characters, it depends on the bytes being a sequence of ASCII characters, which looks an awful lot like an operation on an implicit encoded string.

...

...

b"文字化け" File "<stdin>", line 1 SyntaxError: bytes can only contain ASCII literal characters.

There is an implicit serialization right there. My terminal is utf8 (or even if my source encoding is utf8), so why would that not be: b'\xe6\x96\x87\xe5\xad\x97\xe5\x8c\x96\xe3\x81\x91' I sympathize with Ethan that the bytes and bytearray types already seem to concede that bytes is the type you want to use for 7-bit ASCII manipulations. If that is not what we want, then we are not doing a good job communicating that to developers with the API. At the onset, the bytes literal itself seems to be an attractive nuisance as it gives a nod to using bytes for ASCII character sequences (a.k.a ASCII strings). Regards, -Scott -- Scott Dial scott@scottdial.com

As best I can tell, some people (apparently including Guido and PEP author Antoine) are taking some assumptions almost for granted, while other people (including me, before Nick's messages) were not assuming them at all. Since these assumptions (or, possibly, rejections of them?) are likely to decide the outcome, the assumptions should be explicit in the PEP. (1) The bytes-related classes do include methods that are only useful when the already-contained data is encoded ASCII. They do not (and will not) include any operations that *require* an encoding assumption. This implies that no non-bytes data can be added without an explicit encoding. (1a) Not even by assuming ASCII with strict error handling. (1b) Not even for numbers, where ASCII/strict really is sufficient. Note that this doesn't rule out a solution where objects (or maybe just numbers and ASCII-kind text) provide their own encoding to bytes -- but that has to be done by the objects themselves, not by the bytes container or by the interpreter. (2) Most python programmers are still in the future. So an API that confuses people who are still learning about Unicode and the text model is bad -- even if it would work fine for those who do already understand it. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ