Dmitry Vasiliev wrote:

Is it a good time to consider the entrie tree of ALL types then?

Please, no. One of the best things about Python is that, for most purposes, you can get the appropriate reaction from the standard library simply by providing the relevant magic methods (e.g. __iter__ if you want to be treated like a sequence). Normal class inheritance means you are inheriting the *whole* interface, as well as the implementation of any parts you don't override. So 'sequence' for instance, couldn't have any methods or do much of anything, without causing potentially undesirable behaviour. About the only exception that makes any sense to me is for exceptions, where using inheritance for classification seems to be universal practice in all languages with exceptions (since 'catch X' generally means catch X, or any of its subclasses). (Bad pun noted, and not avoided). Localised type hierarchies where it is useful, OK. Inheriting from object as a way to say 'new-style class please', OK. But please, please, please, lets keep interface introspection the dominant way of determining what can be done with a particular object. Creating a 'grand hierarchy of everything' runs directly counter to that goal. Regards, Nick. -- Nick Coghlan | Eugene, Oregon Email: ncoghlan@email.com | USA

Greg> I can enivisage a future in which 'string' means unicode string, Greg> and byte sequences are called something else entirely, such as Greg> 'bytevector' or 'bytearray' (or maybe just 'array' :-). Or just 'bytes'? Skip

1. Make bytes a synonuym for str.

Hmm... I worry that a simple alias would just encourage confused usage, since the compiler won't check. I'd rather see bytes an alias for a bytes array as defined by the array module.

2. Warn about the use of bytes as a variable name.

Is this really needed? Builtins don't byte variable names.

3. Introduce b"..." literals as a synonym for current string literals, and have them *not* generate warnings if non-ascii characters were used in them without a coding cookie.

I expecet all sorts of problems with that, such as what it would mean if Unicode or multibyte characters are used in the source. Do we really need byte array literals at all? I don't expect there to be much of a demand. Rather, byte arrays would eventually be returned by the read() method when a file is opened in binary mode. (Isn't this roughly how Java does this?) We could start doing this relatively soon if we used a new mode character ("B" anyone?). --Guido van Rossum (home page: http://www.python.org/~guido/)

Martin:

Now, it would also "work" if it were a character string, which then gets converted to a byte string using the system default encoding

Or the encoding associated with the file object, which could be set to ascii, either by default or explicitly. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Guido:

Anyway, if we really do have enough use cases for byte array literals, we might add them. I still think that would be confusing though, because byte arrays are most useful if they are mutable: and then we'd have mutable literals -- blechhhh!

Perhaps the constructor for a byte array could accept a string argument as long as it contained only ascii characters? h.putrequest(bytes('GET'), selector) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Guido:

...

Anyway, if we really do have enough use cases for byte array literals, we might add them. I still think that would be confusing though, because byte arrays are most useful if they are mutable: and then we'd have mutable literals -- blechhhh!

Greg:

Perhaps the constructor for a byte array could accept a string argument as long as it contained only ascii characters?

h.putrequest(bytes('GET'), selector)

Yeah, but that's what Martin called depending on the default encoding. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:

Anyway, if we really do have enough use cases for byte array literals, we might add them. I still think that would be confusing though, because byte arrays are most useful if they are mutable: and then we'd have mutable literals -- blechhhh!

I see. How would you like byte array displays then? This is the approach taken in the other languages: Everytime the array display is executed, a new array is created. There is then no problem with that being mutable. Of course, if the syntax is too similar to string literals, people might be tricked into believing they are actually literals. Perhaps bytes('G','E','T') would be sufficient, or even bytes("GET") which would implicitly convert each character to Latin-1. Regards, Martin

Skip Montanaro wrote:

That's why I think a special literal is necessary. There'd be no unicode foolishness involved. ;-) They'd just be raw uninterpreted bytes.

But you'd spell them b"GET", no? If so, which numeric value has "G"? Regards, Martin

Skip Montanaro wrote:

>> That's why I think a special literal is necessary. There'd be no >> unicode foolishness involved. ;-) They'd just be raw uninterpreted >> bytes.

Martin> But you'd spell them b"GET", no? If so, which numeric value has Martin> "G"?

Good point...

I don't think I understand the example... What's binary about 'GET' ? Why would you want to put non-ASCII into a binary literal definition ? If we switch the binding of 'yyy' to mean unicode('yyy') some day, why can't we just continue to use the existing implementation for 8-bit strings for b'xxx' (the current implementation is already doing the right thing, meaning that it is 8-bit safe regardeless of the source code encoding) ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 13 2004)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

M.-A. Lemburg wrote:

If we switch the binding of 'yyy' to mean unicode('yyy') some day, why can't we just continue to use the existing implementation for 8-bit strings for b'xxx' (the current implementation is already doing the right thing, meaning that it is 8-bit safe regardeless of the source code encoding) ?

Not exactly - the current implementation is not safe with respect to re-encoding source in a different encoding. Regards, Martin

Skip Montanaro wrote:

Can't such re-encoding tools (whatever they are) be intelligent about b"..."? If they are Python-aware that seems fairly trivial. If not, the job is more difficult.

What precisely should they do, though? A byte sequence in one encoding might be invalid in another. Regards, Martin

M.-A. Lemburg wrote:

It is if you stick to writing your binary data using an ASCII compatible encoding -- I wouldn't expect any other encoding for binary data anyway. The most common are ASCII + escape sequences, base64 or hex, all of which are ASCII compatible.

We probably have a different notion of "ASCII compatible" then. I would define it as: An encoding E is "ASCII compatbible" if strings that only consist of ASCII characters use the same byte representation in E that they use in ASCII. In that sense, ISO-8859-1 and UTF-8 are also ASCII compatible. Notice that this is also the definition that PEP 263 assumes. However, byte strings used in source code are not "safe" if they are encoded in ISO-8859-1 under recoding: If the source code is converted to UTF-8 (including the encoding declaration), then the length of the strings changes, as do the byte values inside the string. Regards, Martin

This suggests that byte string literals should be restricted to ASCII characters and \x escapes. Would that be safe enough?

Make that printable ASCII and I'm +0. (I'm still not sold on the concept of bytes literals at all.) --Guido van Rossum (home page: http://www.python.org/~guido/)

Anthony Baxter wrote:

Ok. Here's a case - in shtoom, I generate audio data. Lots of audio data. This is broken into packets, then gets a small header put onto each RTP packet. Right now, I'm using strings for this. If there was a 'byte literal', I'd use it. This isn't a huge problem right now, because strings are good enough. But if we end up in an 'all the strings are unicode', I'll need _some_ way to construct these packets.

Maybe you are missing the point here, maybe not: there is no debate that Python should always have a byte string type (although there is debate on whether that type should be mutable). The current question is whether you want to denote objects of the byte string type *in source code*. I.e. do you have the "Lots of audio data" stored in .py files? Regards, Martin

Guido van Rossum wrote:

...

(I'm still not sold on the concept of bytes literals at all.)

[Anthony]

Ok. Here's a case - in shtoom, I generate audio data. Lots of audio data. This is broken into packets, then gets a small header put onto each RTP packet. Right now, I'm using strings for this. If there was a 'byte literal', I'd use it. This isn't a huge problem right now, because strings are good enough. But if we end up in an 'all the strings are unicode', I'll need _some_ way to construct these packets.

I see that as a huge case for a bytes type, which I've proposed myself; but what's the use case for bytes literals, assuming you can write bytes("foo")? Does b"foo" really make much of a difference? Is it so hard to have to write bytes([0x66, 0x6f, 0x6f]) instead of b"\x66\x6f\x6f"? IOW, how many *literal* packet fragments are in shtoom? --Guido van Rossum (home page: http://www.python.org/~guido/)

On Tuesday 2004-08-17 12:13, Anthony Baxter wrote:

Guido van Rossum wrote:

...

I see that as a huge case for a bytes type, which I've proposed myself; but what's the use case for bytes literals, assuming you can write bytes("foo")? Does b"foo" really make much of a difference? Is it so hard to have to write bytes([0x66, 0x6f, 0x6f]) instead of b"\x66\x6f\x6f"?

It's a pretty marginal case for it. I just played with it a bit, and I think after playing with it, I actually prefer the non b'' case.

Another option, with pros and cons of its own: something along the lines of b"666f6f". -- g

Michael> Is this getting to (hopefully uncontroverisal!) PEP time? One would hope. I sent in a skeletal PEP last week asking for a number but haven't heard back. Michael> Is there any consensus forming on whether bytes() instances are Michael> mutable or not? ISTR Guido thought mutable was the way to go. I don't think that efficiency concerns will be a huge deal since it won't be used all over the place the way strings are. Will they need to be used as dict keys? Doesn't seem likely to me. Skip

Skip Montanaro writes:

Michael> Is this getting to (hopefully uncontroverisal!) PEP time?

One would hope. I sent in a skeletal PEP last week asking for a number but haven't heard back.

Goody.

Michael> Is there any consensus forming on whether bytes() instances are Michael> mutable or not?

ISTR Guido thought mutable was the way to go.

OK.

I don't think that efficiency concerns will be a huge deal since it won't be used all over the place the way strings are.

Who cares? :-)

Will they need to be used as dict keys? Doesn't seem likely to me.

This is indeed the question. I agree it's unlikely, and you can presumably do something like convert a bytes() instance into a tuple of small integers if you really want to key into a dict. Cheers, mwh -- Clue: You've got the appropriate amount of hostility for the Monastery, however you are metaphorically getting out of the safari jeep and kicking the lions. -- coonec -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html

Is this getting to (hopefully uncontroverisal!) PEP time?

Sure.

Is there any consensus forming on whether bytes() instances are mutable or not?

Mutable! --Guido van Rossum (home page: http://www.python.org/~guido/)

So, how will it be different from:

from array import array

def bytes(*initializer): return array('B',*initializer)

Even if it's desirable for 'bytes' to be an actual type (e.g. subclassing ArrayType), it might help the definition process to describe the difference between the new type and a byte array.

Not a whole lot different, except for the ability to use a string as alternate argument to the constructor, and the fact that it's going to be an actual type, and that it should support the buffer API (which array mysteriously doesn't?). The string argument support may not even be necessary -- an alternative way to spell that would be to let s.decode() return a bytes object, which has the advantage of being explicit about the encoding; there's even a base64 encoding already! But it would be a bigger incompatibility, more likely to break existing code using decode() and expecting to get a string. --Guido van Rossum (home page: http://www.python.org/~guido/)

Bob Ippolito wrote:

On Aug 17, 2004, at 5:33 PM, Guido van Rossum wrote:

...

...

So, how will it be different from:

from array import array

def bytes(*initializer): return array('B',*initializer)

Even if it's desirable for 'bytes' to be an actual type (e.g. subclassing ArrayType), it might help the definition process to describe the difference between the new type and a byte array.

Not a whole lot different, except for the ability to use a string as alternate argument to the constructor, and the fact that it's going to be an actual type, and that it should support the buffer API (which array mysteriously doesn't?).

The string argument support may not even be necessary -- an alternative way to spell that would be to let s.decode() return a bytes object, which has the advantage of being explicit about the encoding; there's even a base64 encoding already! But it would be a bigger incompatibility, more likely to break existing code using decode() and expecting to get a string.

IMHO current uses of decode and encode are really confusing. Many decodes are from str -> unicode, and many encodes are from unicode -> str (or str -> unicode -> str implicitly, which is usually going to fail miserably)... while yet others like zlib, base64, etc. are str <-> str. Technically unicode.decode(base64) should certainly work, but it doesn't because unicode doesn't have a decode method.

They do in 2.4. Note that in 2.4 .decode() and .encode() guarantee that you get a basestring instance. If you want more flexibility in terms of return type, the new codecs.encode() and codecs.decode() will allow arbitrary types as return value.

I don't have a proposed solution at the moment, but perhaps these operations should either be outside of the data types altogether (i.e. use codecs only) or there should be separate methods for doing separate things (character translations versus data->data transformations).

It all depends on whether you are discussing placing binary data into the Python source file (by some means of using literals) or just working with bytes you got from a file, generator, socket, etc. The current discussion is mixing these contexts a bit too much, I believe, which is probably why people keep misunderstanding each other (at least that's how I perceive the debate). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 17 2004)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Guido> The string argument support may not even be necessary -- an Guido> alternative way to spell that would be to let s.decode() return a Guido> bytes object, which has the advantage of being explicit about the Guido> encoding; there's even a base64 encoding already! I'm sorry folks, but I still don't understand all this discussion overlap between unicode/string objects (which require explicit or implicit decoding) and bytes objects (which clearly must not). Everyone keeps talking about decoding stuff into bytes objects and whether or not bytes literals would be compatible with the current source encoding. My understanding is that bytes objects are just that, raw sequences of bytes in the range 0x00 to 0xff, inclusive, with no interpretation of any type. Skip

Martin v. Löwis wrote:

Skip Montanaro wrote:

...

My understanding is that bytes objects are just that, raw sequences of bytes in the range 0x00 to 0xff, inclusive, with no interpretation of any type.

Yes, but your understanding is limited :-) This idea is good, but it falls short once we talk about source code, because source code does have an encoding. So if you don't want to incorporate the notion of encodings into the byte string types, yet be able to declare them in source code, you have to go for a numeric representation. I.e. you write bytes(71,69, 84) instead of b"GET"

As soon as you use some kind of string notation for denoting byte code values, you immediately *have* to deal with encodings.

Of course you do, but aren't you making things too complicated, Martin ? If you write your string literal using just ASCII characters and escapes, I don't see much of a problem with different source code encodings. If it makes you feel better, we could even enforce this by only allowing these characters in binary string literals. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 18 2004)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

>> > Is there any consensus forming on whether bytes() instances are >> > mutable or not? >> >> Mutable! Phillip> So, how will it be different from: Phillip> from array import array Phillip> def bytes(*initializer): Phillip> return array('B',*initializer) That might well be a decent trial implementation, though it seems that if we allow strings as initializers we should also allow strings in assignment: >>> b = bytes("abc\xff") >>> b array('B', [97, 98, 99, 255]) >>> print buffer(b) abcÿ >>> b[3] = '\xfe' Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: an integer is required >>> b += "\xfe" Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: can only extend array with array (not "str") I must admit I'm having a bit of trouble getting past this point with a more traditional subclass (can array objects not be subclassed?) in part I think because I don't understand new-style classes very well. In particular, I couldn't find a description of __new__, and once I fumbled past that, I didn't seem to be able to override append() or extend(). Skip

[GvR]

Alas, the array type doesn't seem completely fit for subclassing;

Famous last words: It doesn't look like it would be hard to make array's subclassable. I'll work on that this weekend. Reminder: there is an outstanding array feature request awaiting your adjudication: www.python.org/sf/992967 Raymond

array.array is the only type in the core that's a read buffer but not a character buffer, and I can't find any semantic differences between read and character buffers. If someone can explain the differences or confirm that there aren't any, I'll make this work. The easiest thing to do would be to make array support the character buffer API (but maybe only for [cbBu] types?).

Ah, that makes sense. And no please, keep it that way. It's compatible with the idea that bytes arrays should be read from / written to binary files. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Skip Montanaro wrote:

>>> b[3] = '\xfe'

I think you meant to write b[3] = 0xfe # or byte(0xfe) here. Assigning to an index always takes the element type in Python. It's only strings where reading an index returns the container type.

>>> b += "\xfe"

And here, you probably meant to write b += bytes("\xfe") Regards, Martin

On Tue, 17 Aug 2004, M.-A. Lemburg wrote:

Michael Hudson wrote:

...

Anthony Baxter writes:

...

A big +1 for a bytes() type, though. I'm not sure on the details, but it'd be nice if it was possible to pass a bytes() object to, for instance, write() directly.

If bytes() doesn't implement the read buffer interface, someone somewhere is going to need shooting :-)

Is there any reason you cannot use buffer() ?!

Is it mutable? My guess: no:

...

...

d = u'123124' ddd[0] '1' ddd[1] '\x00' ddd[1] = '1' Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: buffer is read-only

It already implements all the necessary things and has been available for many years.

It was in the shadows because we had byte-strings. Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by GNU/Linux RedHat 7.3

M.-A. Lemburg wrote:

...

We already have a number of read-write types for storing binary data, e.g. arrays, cStringIO and buffers. Inventing yet another way to spell binary data won't make life easier.

The point is canonicalize one and start to make APIs that expect it. Otherwise we will never make the leap from the bad habit of using strings as byte arrays. Can you pass arrays to the write() function? Can you decode buffers to strings? A byte array type would have a certain mix of functions and API compatibility that is missing from the plethora of similar thingees. Paul Prescod

M.-A. Lemburg wrote:

Wouldn't it be possible to extend the existing buffer type to meet those standards ?

Yes; you then need to change all codecs to return buffers from .encode. Regards, Martin

M.-A. Lemburg wrote:

I'm still not convinced that we can simply drop the existing immutable 8-bit string type and replace it with a mutable bytes or buffer type, e.g. would buffer.lower() work on the buffer itself or return a lowered copy ?

The byte string type would not have a .lower method, as "lowering" is not meaningful for bytes, only for characters. Regards, Martin

M.-A. Lemburg wrote:

Indeed... and the same is true for almost all other methods (except maybe .replace()).

Sounds like a lot of code will break.

We will see. The default string type will be Unicode, so code using .lower will continue to work in many cases. The question is what open(path,"r").read() will return. It seems that Guido wants users to specify "rb" if they want that to be byte strings. Regards, Martin

I'm still not convinced that we can simply drop the existing immutable 8-bit string type and replace it with a mutable bytes or buffer type, e.g. would buffer.lower() work on the buffer itself or return a lowered copy ?

Byte strings probably shouldn't have a lower() method at all, or any other method that assumes the contents represent characters. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Roman Suzi wrote:

On Tue, 17 Aug 2004, M.-A. Lemburg wrote:

...

Roman Suzi wrote:

...

On Tue, 17 Aug 2004, M.-A. Lemburg wrote:

...

...

It was in the shadows because we had byte-strings.

Right, so why not revive it ?!

Anyway, this whole discussion about a new bytes type doesn't really solve the problem that the b'...' literal was intended for: that of having a nice way to define (read-only) 8-bit binary string literals.

I think new _mutable_ bytes() type is better than old 8-bit binary strings for binary data processing purposes. Or do we need them for legacy text-procesing software?

Hmm, who ever said that we are going to drop the current 8-bit string implementation ? I'm only suggesting to look at what's there instead of trying to redo everything in slightly different way, e.g. you can already get the bytes() functionality from buffer type at C level - it's just that this functionality is not exposed at Python level.

...

We already have a number of read-write types for storing binary data, e.g. arrays, cStringIO and buffers. Inventing yet another way to spell binary data won't make life easier.

However, what will be missing is a nice way to spell read-only binary data.

Since 'tada' will return a Unicode object in Py3k, I think we should reuse the existing 8-bit string object under the new literal constructor b'tada\x00' (and apply the same source code encoding semantics we apply today for 'tada\x00').

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 18 2004)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Guido van Rossum wrote:

...

Hmm, who ever said that we are going to drop the current 8-bit string implementation ?

I expect that in Python 3.0 aka Python 3000 we'll have only Unicode strings and byte arrays, so yes, the current 8-bit string implementation will eventually die. Jython and IronPython are ahead of us in this respect...

Ok, so I suppose that we can learn from Jython and IronPython in this respect... How do they handle binary data and the interfacing between various I/O facilities, e.g. files, sockets, pipes, user input, etc. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 19 2004)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Guido van Rossum wrote:

...

Ok, so I suppose that we can learn from Jython and IronPython in this respect...

How do they handle binary data and the interfacing between various I/O facilities, e.g. files, sockets, pipes, user input, etc.

I'm not sure, but I expect that in most cases they use Unicode strings in order to be compatibly with Python's standard library. That's not the outcome I'd like to see though. I believe Jython at least also has a bytes-like type (probably a thin wrapper around Java's byte array) that's used for interfacing to java classes.

I've had a discussion with Jack Janssen about using bytes as default return value for I/O operations where no encoding is specified (or unknown). He raised the issue of bytes not being usable as dictionary keys due to their mutability. He was also concerned about the increase in complexity when writing programs that work with non-text data or mixed text/data I/O. If we want to make the move from Python 2.x to 3.0 feasable for large code bases, then we have to do something about these issues. It seems that the simple solution of going with Unicode + bytes type is not going to be a suitable approach. Anyway, we still have 4-5 years to think about this :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 23 2004)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

Martin v. Löwis wrote:

M.-A. Lemburg wrote:

...

Anyway, this whole discussion about a new bytes type doesn't really solve the problem that the b'...' literal was intended for: that of having a nice way to define (read-only) 8-bit binary string literals.

But why do you need a way to spell 8-bit string literals?

You can always do

"string".encode("L1")

If that is too much typing, do

def b(s):return s.encode("L1")

b("string")

You need to think about the important use-case of having to convert Py2 applications to Py3 style. In many cases, the application can be made to run under Py3 be adding the small 'b' in front of the used string literals. Even better: if we add the b'xxx' notation now, we could start working towards the switch-over by slowly converting the Python standard library to actually work in -U mode (which basically implements the switch-over by making 'abc' behave as u'abc'). Since the code is already in place and the change is minimal, I don't see any reason not to use it. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 18 2004)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

A big +1 for a bytes() type, though. I'm not sure on the details, but it'd be nice if it was possible to pass a bytes() object to, for instance, write() directly.

That should already possible with the array module, but somehow it doesn't quite work, even though the array type appears to support the buffer API. (Does anybody understand why not?) --Guido van Rossum (home page: http://www.python.org/~guido/)

Bill Janssen wrote:

Yes. My guess is that if you leave it out, you'll see

var = u"foo".encode("ASCII")

all over the place (assuming that encode() will produce a bytes type).

If you also had var = bytes(u"foo") then I guess people would prefer that. People who want to save typing can do b = bytes and, given that the u prefix will be redundant, write var = b("foo") Regards, Martin

Bob Ippolito wrote:

How would you embed raw bytes if the string was unicode?

The most direct notation would be bytes("delimited packet\x00") However, people might not understand what is happening, and Guido doesn't like it if the bytes are >127. Regards, Martin

Bob Ippolito wrote:

I guess that was a bad example, what if the delimiter was \xff?

I would write bytes("delimited packet\xff") Repeating myself: Guido wouldn't. Regards, Martin

On Aug 17, 2004, at 5:18 PM, Bob Ippolito wrote:

On Aug 17, 2004, at 5:11 PM, Martin v. Löwis wrote:

...

Bob Ippolito wrote:

...

How would you embed raw bytes if the string was unicode?

The most direct notation would be

bytes("delimited packet\x00")

However, people might not understand what is happening, and Guido doesn't like it if the bytes are >127.

I guess that was a bad example, what if the delimiter was \xff?

Indeed, if all strings are unicode, the question becomes: what encoding does bytes() use to translate unicode characters to bytes. Two alternatives have been proposed so far: 1) ASCII (translate chars as their codepoint if < 128, else error) 2) ISO-8859-1 (translate chars as their codepoint if < 256, else error) I think I'd choose #2, myself.

I know that map(ord, u'delimited packet\xff') would get correct results.. but I don't think I like that either.

Why would you consider that wrong? ord(u'\xff') *should* return 255. Just as ord(u'\u1000') returns 4096. There's nothing mysterious there. James

Greg Ewing wrote:

This suggests that byte string literals should be restricted to ASCII characters and \x escapes. Would that be safe enough?

Yes, that should work fine (Guidos restriction to *printable* characters is also useful, as line endings are easily changed, too, when moving from one system to another). Regards, Martin

Guido van Rossum wrote:

...

Anyway, if we really do have enough use cases for byte array literals, we might add them. I still think that would be confusing though, because byte arrays are most useful if they are mutable: and then we'd have mutable literals -- blechhhh!

Martin:

I see. How would you like byte array displays then?

Not as characters (not by default anyway), because more often than not they will contain binary or encoded gibberish!

This is the approach taken in the other languages: Everytime the array display is executed, a new array is created. There is then no problem with that being mutable.

The downside of that is that then for performance reasons you might end up having to move bytes literals out of expressions if they are in fact used read-only (which the compiler can't know but the user can).

Of course, if the syntax is too similar to string literals, people might be tricked into believing they are actually literals. Perhaps

bytes('G','E','T')

would be sufficient, or even

bytes("GET")

which would implicitly convert each character to Latin-1.

The first form would also need an encoding, since Python doesn't have character literals! I don't think we should use Latin-1, for the same reasons that the default encoding is ASCII. Better would be to have a second argument that's the encoding, so you can write bytes(u"<chinese text>", "utf-8") Hm, u"<chinese text>".encode("utf-8") should probably return a bytes array, and that might be sufficient. Perhaps bytes should by default be considered as arrays of tiny unsigned ints, so we could use bytes(map(ord, "GET")) and it would display itself as bytes([71, 69, 84]) Very long ones should probably use ellipses rather than print a million numbers. --Guido van Rossum (home page: http://www.python.org/~guido/)

>> This is the approach taken in the other languages: Everytime the >> array display is executed, a new array is created. There is then no >> problem with that being mutable. Guido> The downside of that is that then for performance reasons you Guido> might end up having to move bytes literals out of expressions if Guido> they are in fact used read-only (which the compiler can't know Guido> but the user can). Wouldn't the compiler be able to tell it was to be treated specially if it saw b"GET"? In that case, the code generated for x = b"GET" would be something like LOAD_CONST "GET" LOAD_NAME bytes CALL_FUNCTION 1 STORE_FAST x Skip

Guido> Anyway, if we really do have enough use cases for byte array Guido> literals, we might add them. I still think that would be Guido> confusing though, because byte arrays are most useful if they are Guido> mutable: and then we'd have mutable literals -- blechhhh! Today I can initialize mutable objects from immutable strings: >>> print list("abc") ['a', 'b', 'c'] >>> print set("abc") set(['a', 'c', 'b']) I see no reason that mutable bytes objects couldn't be created from otherwise immutable sequences either. Would it be a problem to ensure that a = b"abc" b = b"abc" print a is b prints False? The main difference as I see it is that byte literals would be completely devoid of any sort of interpretation as unicode sequences. It would be nice if this was possible: # -*- coding: utf-8 -*- b = b"â" though that would probably wreak havoc with editors and hex escapes would have to be used in most situations. Skip

Martin> Java also supports byte arrays in the source, although they are Martin> difficult to type: Martin> byte[] request = {'G', 'E', 'T'}; Seems to me that b"GET" would be more Pythonic given existing Python string literals. Martin> As for reading from streams: Java has multiple reader APIs; some Martin> return byte strings, some character strings. I think Guido's proposed 'B' might make sense here. OTOH, today's 'b' might work as well, though a switch of that magnitude could probably not be made until 3.0 if bytes objects are not synonyms for strings. Skip

>> 1. Make bytes a synonuym for str.

Guido> Hmm... I worry that a simple alias would just encourage confused Guido> usage, since the compiler won't check. I'd rather see bytes an Guido> alias for a bytes array as defined by the array module.

You're right. This could probably be added now ("now" being 2.5) with little or no problem. My thought was to get the name in there quickly (could be done in 2.4) with some supporting documentation so people could begin modifying their code.

I think very few people would do so until the semantics of bytes were clearer. Let's just put it in meaning byte array when we're ready.

>> 2. Warn about the use of bytes as a variable name.

Guido> Is this really needed? Builtins don't byte variable names.

I suppose this could be dispensed with. Let pychecker handle it.

>> 3. Introduce b"..." literals as a synonym for current string >> literals, and have them *not* generate warnings if non-ascii >> characters were used in them without a coding cookie.

Guido> I expecet all sorts of problems with that, such as what it would Guido> mean if Unicode or multibyte characters are used in the source.

My intent in proposing b"..." literals was that they would be allowed in any source file. Their contents would not be interpreted in any way.

But they would be manipulated if they were non-ASCII and the source file was converted to a different encoding. Better be safe and only allow printable ASCII and hex escapes there.

One simple use case: identifying the magic number of a binary file type of some sort. That might well be a constant to programmers manipulating that sort of file and have nothing to do with Unicode at all.

Not a very strong use case, this could easily be done using just hex.

Guido> Do we really need byte array literals at all?

I think so. Martin already pointed out an example where a string literal is used today for a sequences of bytes that's put out on the wire as-is. It's just convenient that the protocol was developed in such a way that most of its meta-data is plain ASCII.

See my response to that. --Guido van Rossum (home page: http://www.python.org/~guido/)

This is why "integer" is a better name than "baseinteger". For now it can be the common supertype of int and long. In the future, it can be the name of the single integer type.

No, that will be int, of course! Like 'basestring', 'baseinteger' is intentionally cumbersome, because it is only the base class of all *built-in* integral types. Note that UserString is *not* subclassing basestring, and likewise if you wrote an integer-like class from scratch it should not inherit from baseinteger. Code testing for these types is interested in knowing whether something is a member of one of the *built-in* types, which is often needed because other built-in operations (e.g. many extension modules) only handle the built-in types. If you want to test for integer-like or string-like behavior, you won't be able to use isinstance(), but instead you'll have to check for the presence of certain methods. I know this is not easy in the case of integers, but I don't want to start requiring inheritance from a marker base type now. Python is built on duck typing. (Google for it.) --Guido van Rossum (home page: http://www.python.org/~guido/)