On Sat, Jan 11, 2014 at 8:20 PM, Terry Reedy <tjreedy@udel.edu> wrote:

The following function interpolates bytes, bytearrays, and formatted strings, the latter two auto-converted to bytes, into a bytes (or auto-converted bytearray) format. This function automates much of what some people have recommended for combining ascii text and binary blogs. The test passes on 2.7.6 as well as 3.3.3, though a 2.7-only version would be simpler. ===============

# bf.py -- Terry Jan Reedy, 2014 Jan 11 "Define byteformat(): a bytes version of str.format as a function." import re

def byteformat(form, obs): '''Return bytes-formated objects interpolated into bytes format.

The bytes or bytearray format has two types of replacement fields. b'{}' and b'{:}': The object can be any raw bytes or bytearray object. b'{:<format_spec>}: The object can by any object ob that can be string-formated with <format_spec>. Bytearray are converted to bytes.

The text encoding is the default (encoding="utf-8", errors="strict"). Users should be explicitly encode to bytes for any other encoding. The struct module can by used to produce bytes, such as binary-formated integers, that are not encoded text.

Test passes on both 2.7.6 and 3.3.3. '''

if isinstance(form, bytearray): form = bytes(form) fields = re.split(b'{:?([^}]*)}', form) # print(fields) if len(fields) != 2*len(obs)+1: raise ValueError('Number of replacement fields not same as len(obs)') j = 1 # index into fields for ob in obs: if isinstance(ob, bytearray): ob = bytes(ob) field = fields[j] fields[j] = format(ob, field.decode()).encode() if field else ob j += 2 return b''.join(fields)

# test code bformat = b"bytes: {}; bytearray: {:}; unicode: {:s}; int: {:5d}; float: {:7.2f}; end" objects = (b'abc', bytearray(b'def'), u'ghi', 123, 12.3) result = byteformat(bformat, objects) result2 = byteformat(bytearray(bformat), objects) strings = (ob.decode() if isinstance(ob, (bytes, bytearray)) else ob for ob in objects) expect = bformat.decode().format(*strings).encode()

#print(result) #print(result2) print(expect) assert result == result2 == expect

===== This has been edited from what I posted to issue 3982 to expand the docstrings and to work the same with both bytes and bytearrays on both 2.7 and 3.3. When I posted before, I though of it merely as a proof-of-concept prototype. After reading the seemingly endless discussion of possible variations of byte formatting with % and .format, I now present it as a real, concrete, proposal.

There are, of course, details that could be tweaked. The encoding uses the default, which on 3.x is (encoding='utf-8', errors='strict'). This could be changed to an explicit encoding='ascii'. If that were done, the encoding could be made a parameter that defaults to 'ascii'. The joiner could be defined as type(form)() so the output type matches the input form type. I did not do that because it complicates the test.

With that flexibility this matches what I have been mulling in the back of my head all day. Basically everything that goes in is assumed to be bytes unless {:s} says to expect something which can be passed to str() and then use some specified encoding in all instances (stupid example following as it might be easier with bytes.join, but it gets the point across):: formatter = format_bytes('latin1', 'strict') http_response = formatter(b'Content-Type: {:s}\r\n\r\nContent-Length: {:s}\r\n\r\n{}', 'image/jpeg', len(data), data) Nothing fancy, just an easy way to handle having to call str.encode() on every text argument that is to end up as bytes as Terry is proposing (and I'm fine with defaulting to ASCII/strict with no arguments). Otherwise you do what R. David Murray suggested and just have people rely on their own API which accepts what they want and then spits out what they want behind the scenes. It basically comes down to how much tweaking of existing Python 2.7 %/.format() calls people will be expected to make. I'm fine with asking people to call a function like what Terry is proposing as it can do away with baking in that ASCII is reasonable as well as not require a bunch of work without us having to argue over what bytes.format() should or should not do. Personally I say bytes.format() is fine but it shouldn't do any text encoding which makes its usefulness rather minor (much like the other text-like methods that got carried forward in hopes that they would be useful to people porting code; maybe we should consider taking them out in Python 4 or something if we find out no one is using them).

The coercion of interpolated bytearray objects to bytes is needed for 2.7 because in 2.7, str/bytes.join raises TypeError for bytearrays in the input sequence. A 3.x-only version could drop this.

One objection to the function is that it is neither % or .format. To me, this is an advantage in that a new function will not be expected to exactly match the % or .format behavior in either 2.x or 3.x. It eliminates the 'matching the old' arguments so we can focus on what actual functionality is needed.

Agreed.

There is no need to convert true binary bytes to text with either latin-1 or surrogates. There is no need to add anything to bytes. The code above uses the built-in facilities that we already have, which to me should be the first thing to try, not the last.

I think we are all losing sight of the fact that we are talking about Python 3.5 here. Even with an accelerated release schedule of a year that is still a year away! I think any proposal being made should be prototyped in pure Python and tried on a handful or real world examples to see how the results end up looking like to measure how useful they are on their own and how much work it is to port to using it. I think the goal should be a balance and not going to an extreme to minimize porting work from Python 2.7 at the cost of polluting the bytes/string separation and letting people entirely ignore encoding of strings.

One new feature that does not match old behavior is that {} and {:} are changed (in 3.x) to indicate bytes whereas {:s} continues to indicate (in 3.x) unicode text. ({:s} might be changed to mean unicode for 2.7 also, but I did not explore that idea.) Similarly, a new function is free to borrow only the format_spec part of replace of replacement fields and use format(ob, format_spec) to format each object. Anyone who needs the full power of str.format is free to use it explicitly. I think format_specs cover most of what people have asked for.

For future releases, the function could go in the string module. It could otherwise be added to existing or future 2&3 porting packages.

I don't think the string module is the right place since this is meant to operate on bytes, but then again I don't know where it would end up if it went into the stdlib. If we have it take the string encoding arguments it could be a method on the bytes type by being a factory method:: formatter = bytes.formatter('latin1', 'strict') ... I would be willing to go as far as making 'strict' the default 'error' argument, but I would say it's still go to make people specify even 'ascii', otherwise people lose sight that bytes([ord(1)]) == b'1' == '1'.encode('ascii') != 1 .to_bytes(1, 'big') and that is a key thing to grasp.