[New-bugs-announce] [issue5419] urllib.request.open(someURL).read() returns a bytes object so writing it requires binary mode

[Mitchell Model]

New submission from Mitchell Model <mlm at acm.org>:

There needs to be something somewhere in the documentation that makes 
the simple point that data coming in from the web is bytes not strings, 
which is a big change from Python 2, and that it needs to be manipulated 
as such, including writing in binary mode.

I am not sure what documentation should be changed, but I do think 
something is missing, because I just ran around in circles on this one 
for quite some time. Perhaps the Unicode HOWTO needs more information; 
possibly urllib.request does; maybe a combination of things have to be 
added to several documentation files. Here's what happened:

I wanted to read from a web page, make some string replacements, and 
save to a file, so I wrote code that boils down to something like:

    with open('url.html', 'w') as fil:
        fil.write(urllib.request.open(aURL).read()).replace(str1, str2)

The first thing that happened was an error telling me that I can't write 
bytes to a text stream, so I realized that read() was returning a bytes 
object, which makes sense.

So I converted it to a string, but that put a b' at the beginning of the 
file and a ' at the end! Bad.

Instead of str(thebytes) I did the proper thing: thebytes.decode(), and 
wrote that to the file.

But then I found that Non-ASCII characters created problems -- they were 
saved in the file as \xNN\xNN or even three \x's, then displayed as 
garbage when the page was opened in a browser. 

So I tried decoding using different codecs but couldn't find one that 
worked for the é and the emdash that were in the response.

Finally I realized that the whole thing was a delusion: obviously 
urlopen responses have to return bytes objects, and adding 'b' to the 
'w' when opening the output file fixed everything. (I also had to change 
my replacement strings to bytes.)

I went back to the relevant documentation multiple times, including 
after I figured everything out, and I can't convince myself that it 
makes the connection anywhere between bytes coming in, manipulating the 
bytes as bytes, and writing out in binary. Yes, in retrospect this all 
makes sense and perhaps even should have been obvious, but I am quite 
sure I won't be the only experienced Python 2 programmer to trip over 
this when making the transition to Python 3.

I apologize in advance if the requested documentation exists and I 
didn't find it, in which case I would appreciate a pointer to where it 
is lies.

----------
assignee: georg.brandl
components: Documentation
messages: 83179
nosy: MLModel, georg.brandl
severity: normal
status: open
title: urllib.request.open(someURL).read() returns a  bytes object so writing it requires binary mode
versions: Python 3.0, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5419>
_______________________________________