[issue11681] -b option undocumented
New submission from Éric Araujo merwok@netwok.org:
Python 2 has a -b command-line option which is undocumented.
It is used by the warnings machinery to decide to warn about (-b) or raise an error (-bb) if someone uses str(bytes) or str(buffer) (explanation courtesy of grep and Python/pythonrun.c).
---------- assignee: docs@python components: Documentation keywords: easy messages: 132195 nosy: docs@python, eric.araujo priority: normal severity: normal stage: needs patch status: open title: -b option undocumented versions: Python 2.7
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
R. David Murray rdmurray@bitdance.com added the comment:
Are you sure? (see issue 10471). I don't see how -b could do anything in python2, since bytes and string are the same there (and, indeed, from command line experimentation it doesn't seem to).
---------- nosy: +r.david.murray
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Éric Araujo merwok@netwok.org added the comment:
The code definitely is in 2.7.
Python/_warnings.c:861: if (Py_BytesWarningFlag > 1) Python/_warnings.c:863: else if (Py_BytesWarningFlag) Python/_warnings.c:867: PyList_SET_ITEM(filters, pos++, create_filter(PyExc_BytesWarning, Python/sysmodule.c:1257: SetFlag(Py_BytesWarningFlag); Python/pythonrun.c:81:int Py_BytesWarningFlag; /* Warn on str(bytes) and str(buffer) */ Include/pydebug.h:14:PyAPI_DATA(int) Py_BytesWarningFlag; Include/pyerrors.h:178:PyAPI_DATA(PyObject *) PyExc_BytesWarning; Modules/main.c:285: Py_BytesWarningFlag++; Objects/bytearrayobject.c:1003: if (Py_BytesWarningFlag) { Objects/bytearrayobject.c:1004: if (PyErr_WarnEx(PyExc_BytesWarning, Objects/bytearrayobject.c:1028: if (Py_BytesWarningFlag && op == Py_EQ) { Objects/bytearrayobject.c:1029: if (PyErr_WarnEx(PyExc_BytesWarning, Objects/exceptions.c:2018: * BytesWarning extends Warning Objects/exceptions.c:2020:SimpleExtendsException(PyExc_Warning, BytesWarning, Objects/exceptions.c:2110: PRE_INIT(BytesWarning) Objects/exceptions.c:2178: POST_INIT(BytesWarning) Lib/warnings.py:399: simplefilter(bytes_action, category=BytesWarning, append=1) Lib/test/test_bytes.py:25: with test.test_support.check_warnings(('', BytesWarning)): Lib/test/exception_hierarchy.txt:50: +-- BytesWarning Include/pydebug.h:14:PyAPI_DATA(int) Py_BytesWarningFlag; Include/pyerrors.h:178:PyAPI_DATA(PyObject *) PyExc_BytesWarning; Doc/library/warnings.rst:164:* :exc:`BytesWarning` is ignored unless the :option:`-b` option is given once or
---------- nosy: +georg.brandl
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
R. David Murray rdmurray@bitdance.com added the comment:
rdmurray@hey:~/python/p27>cat temp.py x = bytes('abc') print x print str(x) rdmurray@hey:~/python/p27>./python -bb temp.py abc abc
----------
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Éric Araujo merwok@netwok.org added the comment:
test_bytes has a branch for -b and passes without it, with -b and with -bb. I really don’t know.
----------
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Changes by Ezio Melotti ezio.melotti@gmail.com:
---------- nosy: +ezio.melotti
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Nick Coghlan ncoghlan@gmail.com added the comment:
Even "from __future__ import unicode_literals" doesn't make it do anything.
Perhaps Christian merged it by mistake in [5341b30b1812]?
---------- nosy: +ncoghlan
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Marc-Andre Lemburg mal@egenix.com added the comment:
Nick Coghlan wrote:
Nick Coghlan ncoghlan@gmail.com added the comment:
Even "from __future__ import unicode_literals" doesn't make it do anything.
Perhaps Christian merged it by mistake in [5341b30b1812]?
It looks more like some parts were left out in the merge by accident: The 2.7 code only raises the warning for bytearrays, not for bytes and buffers, whereas 3.2 issues the warning for all bytes and bytearray (but not buffers; which makes sense, since Python 3 no longer has a text buffer interface).
---------- nosy: +lemburg
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
moijes12 moijes12@gmail.com added the comment:
Hi Marc
I tried reproducing this for bytearray using Python 2.7.2 but I can't see a warning.
devel@moses:~$ python --version Python 2.7.2+ devel@moses:~$ cat test_py.py k = range(10) kb = bytearray(k) print kb kb = bytearray("hi") print kb devel@moses:~$ python -b test_py.py
hi devel@moses:~$ python -bb test_py.py
hi
Please correct me if I'm wrong anywhere here. I'd like to work on this.
---------- nosy: +moijes12
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Changes by moijes12 moijes12@gmail.com:
---------- nosy: -moijes12
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Martin Panter added the comment:
Try this to trigger a warning:
python2 -b -c 'bytearray("3") == u"3"' -c:1: BytesWarning: Comparison between bytearray and string
---------- nosy: +vadmium
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Emily Zhao added the comment:
Might be worth making this addition from 3 (I'm not sure how to add this to 2) -b : issue warnings about str(bytes_instance), str(bytearray_instance) and comparing bytes/bytearray with str. (-bb: issue errors)
Building on Martin's example:
On all of these, python2 is Python 2.7.6 (default, Apr 6 2014, 23:14:26) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin Type "help", "copyright", "credits" or "license" for more information.
emily-mba:cpython emily$ python2
bytearray("3") == u"3"
False
emily-mba:cpython emily$ python2 -b
bytearray("3") == u"3"
__main__:1: BytesWarning: Comparison between bytearray and string False
emily-mba:cpython emily$ python2 -bb
bytearray("3") == u"3"
Traceback (most recent call last): File "<stdin>", line 1, in <module> BytesWarning: Comparison between bytearray and string
---------- nosy: +emily.zhao
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Emily Zhao added the comment:
Here's an attempt (based on 3's main.c http://hg.python.org/cpython/file/8866ac6f2269/Modules/main.c)
---------- keywords: +patch Added file: http://bugs.python.org/file35528/issue11681.patch
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Martin Panter added the comment:
Trouble is, in Python 2 bytes() and str() are the same thing, and most of those conditions don’t apply. Maybe something like this is more correct:
-b : issue warnings about comparing bytearray with unicode. (-bb: issue errors)
----------
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Changes by Arfrever Frehtes Taifersar Arahesis Arfrever.FTA@GMail.Com:
---------- nosy: +Arfrever
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Martin Panter added the comment:
As well as the usage string, it is missing from the list of options in the main documentation at Doc/using/cmdline.rst. There are a couple of places (sys.rst and warnings.rst) that reference :option:`-b`, and newer versions of Sphinx complain about this.
----------
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Greg Bengeult added the comment:
I have attached a patch for 2.7 that adds -b and -bb to the command line documentation in Modules/main.c and Doc/library/cmdline.rst, following the suggested text provided by Martin.
Interestingly, unicode(), bytearray(), and str() don't seem to be transitive in 2.7.
u"3" == str(3)
True
str(3) == bytearray("3")
True
u"3" == bytearray("3")
False
---------- nosy: +gbengeult Added file: http://bugs.python.org/file46261/b_option.patch
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Nick Coghlan added the comment:
Right, the lack of transitivity comes from the fact that:
>>> u"3" == str(3) True
Is really shorthand for:
>>> u"3" == str(3).decode("ascii") True
However, the implicit decoding only triggers for *exactly* bytes instances, so in the bytearray case it needs to be explicit:
>>> u"3" == bytearray(b"3").decode("ascii") True
----------
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Nick Coghlan added the comment:
Added some review comments on the patch. The only critical one was changing a :class:`bytes` reference to :class:`unicode`, but I also suggested it may be worth mentioning that it *doesn't* enable warnings about all binary/text comparisons the way the Python 3 one does.
----------
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Greg Bengeult added the comment:
Thanks for the :class: markup edits. I'm still learning this stuff.
----------
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Changes by Zachary Ware zachary.ware@gmail.com:
---------- pull_requests: +1660
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Zachary Ware added the comment:
New changeset f6c6d1e2304a06efdf65c60c3e1c6bbe1f192fd0 by Zachary Ware in branch '2.7': bpo-11681: Document the `-b` and `-bb` options (GH-1562) https://github.com/python/cpython/commit/f6c6d1e2304a06efdf65c60c3e1c6bbe1f1...
---------- nosy: +zach.ware
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
Zachary Ware added the comment:
Thanks for the patch, Greg! I implemented Nick's suggestions and merged it.
---------- resolution: -> fixed stage: needs patch -> resolved status: open -> closed
_______________________________________ Python tracker report@bugs.python.org http://bugs.python.org/issue11681 _______________________________________
participants (11)
-
Arfrever Frehtes Taifersar Arahesis -
Emily Zhao
-
Ezio Melotti
-
Greg Bengeult
-
Marc-Andre Lemburg
-
Martin Panter
-
moijes12
-
Nick Coghlan
-
R. David Murray
-
Zachary Ware
-
Éric Araujo