Author: guido.van.rossum Date: Wed Sep 26 23:55:16 2007 New Revision: 58263 Added: peps/trunk/pep-3137.txt (contents, props changed) Modified: peps/trunk/pep-0000.txt Log: PEP 3137: immutable bytes and mutable buffer. Modified: peps/trunk/pep-0000.txt ============================================================================== --- peps/trunk/pep-0000.txt (original) +++ peps/trunk/pep-0000.txt Wed Sep 26 23:55:16 2007 @@ -96,6 +96,7 @@ S 3116 New I/O Stutzbach, Verdone, GvR S 3134 Exception Chaining and Embedded Tracebacks Yee S 3135 New Super Spealman, Delaney + S 3137 Immutable Bytes and Mutable Buffer GvR S 3141 A Type Hierarchy for Numbers Yasskin Finished PEPs (done, implemented in Subversion) @@ -509,6 +510,7 @@ S 3134 Exception Chaining and Embedded Tracebacks Yee S 3135 New Super Spealman, Delaney SR 3136 Labeled break and continue Chisholm + S 3137 Immutable Bytes and Mutable Buffer GvR S 3141 A Type Hierarchy for Numbers Yasskin Added: peps/trunk/pep-3137.txt ============================================================================== --- (empty file) +++ peps/trunk/pep-3137.txt Wed Sep 26 23:55:16 2007 @@ -0,0 +1,239 @@ +PEP: 3137 +Title: Immutable Bytes and Mutable Buffer +Version: $Revision$ +Last-Modified: $Date$ +Author: Guido van Rossum <guido@python.org> +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 26-May-2007 +Python-Version: 3.0 +Post-History: 26-may-2007 + +Introduction +============ + +After releasing Python 3.0a1 with a mutable bytes type, pressure +mounted to add a way to represent immutable bytes. Gregory P. Smith +proposed a patch that would allow making a bytes object temporarily +immutable by requesting that the data be locked using the new buffer +API from PEP 3118. This did not seem the right approach to me. + +Jeffrey Yasskin, with the help of Adam Hupp, then prepared a patch to +make the bytes type immutable (by crudely removing all mutating APIs) +and fix the fall-out in the test suite. This showed that there aren't +all that many places that depend on the mutability of bytes, with the +exception of code that builds up a return value from small pieces. + +Thinking through the consequences, and noticing that using the array +module as an ersatz mutable bytes type is far from ideal, and +recalling a proposal put forward earlier by Talin, I floated the +suggestion to have both a mutable and an immutable bytes type. (This +had been brought up before, but until seeing the evidence of Jeffrey's +patch I wasn't open to the suggestion.) + +Moreover, a possible implementation strategy became clear: use the old +PyString implementation, stripped down to remove locale support and +implicit conversions to/from Unicode, for the immutable bytes type, +and keep the new PyBytes implementation as the mutable bytes type. + +The ensuing discussion made it clear that the idea is welcome but +needs to be specified more precisely. Hence this PEP. + +Advantages +========== + +One advantage of having an immutable bytes type is that code objects +can use these. It also makes it possible to efficiently create hash +tables using bytes for keys; this may be useful when parsing protocols +like HTTP or SMTP which are based on bytes representing text. + +Porting code that manipulates binary data (or encoded text) in Python +2.x will be easier using the new design than using the original 3.0 +design with mutable bytes; simply replace ``str`` with ``bytes`` and +change '...' literals into b'...' literals. + +Naming +====== + +I propose the following type names at the Python level: + + - ``bytes`` is an immutable array of bytes (PyString) + + - ``buffer`` is a mutable array of bytes (PyBytes) + + - ``memoryview`` is a bytes view on another object (PyMemory) + +The old type named ``buffer`` is so similar to the new type +``memoryview``, introduce by PEP 3118, that it is redundant. The rest +of this PEP doesn't discuss the functionality of ``memoryview``; it is +just mentioned here to justify getting rid of the old ``buffer`` type +so we can reuse its name for the mutable bytes type. + +While eventually it makes sense to change the C API names, this PEP +maintains the old C API names, which should be familiar to all. + +Literal Notations +================= + +The b'...' notation introduced in Python 3.0a1 returns an immutable +bytes object, whatever variation is used. To create a mutable bytes +buffer object, use buffer(b'...') or buffer([...]). The latter may +use a list of integers in range(256). + +Functionality +============= + +PEP 3118 Buffer API +------------------- + +Both bytes and buffer support the PEP 3118 buffer API. The bytes type +only supports read-only requests; the buffer type allows writable and +data-locked requests as well. The element data type is always 'B' +(i.e. unsigned byte). + +Constructors +------------ + +There are four forms of constructors, applicable to both bytes and +buffer: + + - ``bytes(<bytes>)``, ``bytes(<buffer>)``, ``buffer(<bytes>)``, + ``buffer(<buffer>)``: simple copying constructors, with the note + that ``bytes(<bytes>)`` might return its (immutable) argument. + + - ``bytes(<str>, <encoding>[, <errors>])``, ``buffer(<str>, + <encoding>[, <errors>])``: encode a text string. Note that the + ``str.encode()`` method returns an *immutable* bytes object. + The <encoding> argument is mandatory; <errors> is optional. + + - ``bytes(<memory view>)``, ``buffer(<memory view>)``: construct a + bytes or buffer object from anything that supports the PEP 3118 + buffer API. + + - ``bytes(<iterable of ints>)``, ``buffer(<iterable of ints>)``: + construct an immutable bytes or mutable buffer object from a + stream of integers in range(256). + + - ``buffer(<int>)``: construct a zero-initialized buffer of a given + lenth. + +Comparisons +----------- + +The bytes and buffer types are comparable with each other and +orderable, so that e.g. b'abc' == buffer(b'abc') < b'abd'. + +Comparing either type to a str object raises an exception. This +turned out to be necessary to catch common mistakes. + +Slicing +------- + +Slicing a bytes object returns a bytes object. Slicing a buffer +object returns a buffer object. + +Slice assignment to a mutable buffer object accept anything that +supports the PEP 3118 buffer API, or an iterable of integers in +range(256). + +Indexing +-------- + +**Open Issue:** I'm undecided on whether indexing bytes and buffer +objects should return small ints (like the bytes type in 3.0a1, and +like lists or array.array('B')), or bytes/buffer objects of length 1 +(like the str type). The latter (str-like) approach will ease porting +code from Python 2.x; but it makes it harder to extract values from a +bytes array. + +Assignment to an item of a mutable buffer object accepts an int in +range(256); if we choose the str-like approach for indexing above, it +also accepts an object implementing the PEP 3118 buffer API, if it has +length 1. + +Str() and Repr() +---------------- + +The str() and repr() functions return the same thing for these +objects. The repr() of a bytes object returns a b'...' style literal. +The repr() of a buffer returns a string of the form "buffer(b'...')". + +Methods +------- + +The following methods are supported by bytes as well as buffer, with +similar semantics. They accept anything that implements the PEP 3118 +buffer API for bytes arguments, and return the same type as the object +whose method is called ("self"):: + + .capitalize(), .center(), .count(), .decode(), .endswith(), + .expandtabs(), .find(), .index(), .isalnum(), .isalpha(), .isdigit(), + .islower(), .isspace(), .istitle(), .isupper(), .join(), .ljust(), + .lower(), .lstrip(), .partition(), .replace(), .rfind(), .rindex(), + .rjust(), .rpartition(), .rsplit(), .rstrip(), .split(), + .splitlines(), .startswith(), .strip(), .swapcase(), .title(), + .translate(), .upper(), .zfill() + +This is exactly the set of methods present on the str type in Python +2.x, with the exclusion of .encode(). The signatures and semantics +are the same too. However, whenever character classes like letter, +whitespace, lower case are used, the ASCII definitions of these +classes are used. (The Python 2.x str type uses the definitions from +the current locale, settable through the locale module.) The +.encode() method is left out because of the more strict definitions of +encoding and decoding in Python 3000: encoding always takes a Unicode +string and returns a bytes sequence, and decoding always takes a bytes +sequence and returns a Unicode string. + +Bytes and the Str Type +---------------------- + +Like the bytes type in Python 3.0a1, and unlike the relationship +between str and unicode in Python 2.x, any attempt to mix bytes (or +buffer) objects and str objects without specifying an encoding will +raise a TypeError exception. This is the case even for simply +comparing a bytes or buffer object to a str object (even violating the +general rule that comparing objects of different types for equality +should just return False). + +Conversions between bytes or buffer objects and str objects must +always be explicit, using an encoding. There are two equivalent APIs: +``str(b, <encoding>[, <errors>])`` is equivalent to +``b.encode(<encoding>[, <errors>])``, and +``bytes(s, <encoding>[, <errors>])`` is equivalent to +``s.decode(<encoding>[, <errors>])``. + +There is one exception: we can convert from bytes (or buffer) to str +without specifying an encoding by writing ``str(b)``. This produces +the same result as ``repr(b)``. This exception is necessary because +of the general promise that *any* object can be printed, and printing +is just a special case of conversion to str. There is however no +promise that printing a bytes object interprets the individual bytes +as characters (unlike in Python 2.x). + +The str type current supports the PEP 3118 buffer API. While this is +perhaps occasionally convenient, it is also potentially confusing, +because the bytes accessed via the buffer API represent a +platform-depending encoding: depending on the platform byte order and +a compile-time configuration option, the encoding could be UTF-16-BE, +UTF-16-LE, UTF-32-BE, or UTF-32-LE. Worse, a different implementation +of the str type might completely change the bytes representation, +e.g. to UTF-8, or even make it impossible to access the data as a +contiguous array of bytes at all. Therefore, support for the PEP 3118 +buffer API will be removed from the str type. + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: