sys.argv as a list of bytes

[Olive]

On Wed, 18 Jan 2012 09:05:42 +0100
Peter Otten <__peter__ at web.de> wrote:

> Olive wrote:
> 
> > In Unix the operating system pass argument as a list of C strings.
> > But C strings does corresponds to the bytes notions of Python3. Is
> > it possible to have sys.argv as a list of bytes ? What happens if I
> > pass to a program an argumpent containing funny "character", for
> > example (with a bash shell)?
> > 
> > python -i ./test.py $'\x01'$'\x05'$'\xFF'
> 
> Python has a special errorhandler, "surrogateescape" to deal with
> bytes that are not valid UTF-8. If you try to print such a string you
> get an error:
> 
> $ python3 -c'import sys; print(repr(sys.argv[1]))'
> $'\x01'$'\x05'$'\xFF' '\x01\x05\udcff'
> $ python3 -c'import sys; print(sys.argv[1])' $'\x01'$'\x05'$'\xFF'
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in
> position 2: surrogates not allowed
> 
> It is still possible to get the original bytes:
> 
> $ python3 -c'import sys; print(sys.argv[1].encode("utf-8",
> "surrogateescape"))' $'\x01'$'\x05'$'\xFF' b'\x01\x05\xff'
> 
> 

But is it safe even if the locale is not UTF-8? I would like to be able
to pass a file name to a script. I can use bytes for file names in the
open function. If I keep the filename as bytes everywhere it will work
reliably whatever the locale or strange character the file name may
contain. 

Olive