[issue11681] -b option undocumented
New submission from Éric Araujo <merwok@netwok.org>: Python 2 has a -b command-line option which is undocumented. It is used by the warnings machinery to decide to warn about (-b) or raise an error (-bb) if someone uses str(bytes) or str(buffer) (explanation courtesy of grep and Python/pythonrun.c). ---------- assignee: docs@python components: Documentation keywords: easy messages: 132195 nosy: docs@python, eric.araujo priority: normal severity: normal stage: needs patch status: open title: -b option undocumented versions: Python 2.7 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
R. David Murray <rdmurray@bitdance.com> added the comment: Are you sure? (see issue 10471). I don't see how -b could do anything in python2, since bytes and string are the same there (and, indeed, from command line experimentation it doesn't seem to). ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Éric Araujo <merwok@netwok.org> added the comment: The code definitely is in 2.7. Python/_warnings.c:861: if (Py_BytesWarningFlag > 1) Python/_warnings.c:863: else if (Py_BytesWarningFlag) Python/_warnings.c:867: PyList_SET_ITEM(filters, pos++, create_filter(PyExc_BytesWarning, Python/sysmodule.c:1257: SetFlag(Py_BytesWarningFlag); Python/pythonrun.c:81:int Py_BytesWarningFlag; /* Warn on str(bytes) and str(buffer) */ Include/pydebug.h:14:PyAPI_DATA(int) Py_BytesWarningFlag; Include/pyerrors.h:178:PyAPI_DATA(PyObject *) PyExc_BytesWarning; Modules/main.c:285: Py_BytesWarningFlag++; Objects/bytearrayobject.c:1003: if (Py_BytesWarningFlag) { Objects/bytearrayobject.c:1004: if (PyErr_WarnEx(PyExc_BytesWarning, Objects/bytearrayobject.c:1028: if (Py_BytesWarningFlag && op == Py_EQ) { Objects/bytearrayobject.c:1029: if (PyErr_WarnEx(PyExc_BytesWarning, Objects/exceptions.c:2018: * BytesWarning extends Warning Objects/exceptions.c:2020:SimpleExtendsException(PyExc_Warning, BytesWarning, Objects/exceptions.c:2110: PRE_INIT(BytesWarning) Objects/exceptions.c:2178: POST_INIT(BytesWarning) Lib/warnings.py:399: simplefilter(bytes_action, category=BytesWarning, append=1) Lib/test/test_bytes.py:25: with test.test_support.check_warnings(('', BytesWarning)): Lib/test/exception_hierarchy.txt:50: +-- BytesWarning Include/pydebug.h:14:PyAPI_DATA(int) Py_BytesWarningFlag; Include/pyerrors.h:178:PyAPI_DATA(PyObject *) PyExc_BytesWarning; Doc/library/warnings.rst:164:* :exc:`BytesWarning` is ignored unless the :option:`-b` option is given once or ---------- nosy: +georg.brandl _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
R. David Murray <rdmurray@bitdance.com> added the comment: rdmurray@hey:~/python/p27>cat temp.py x = bytes('abc') print x print str(x) rdmurray@hey:~/python/p27>./python -bb temp.py abc abc ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Éric Araujo <merwok@netwok.org> added the comment: test_bytes has a branch for -b and passes without it, with -b and with -bb. I really don’t know. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Changes by Ezio Melotti <ezio.melotti@gmail.com>: ---------- nosy: +ezio.melotti _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Nick Coghlan <ncoghlan@gmail.com> added the comment: Even "from __future__ import unicode_literals" doesn't make it do anything. Perhaps Christian merged it by mistake in [5341b30b1812]? ---------- nosy: +ncoghlan _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Marc-Andre Lemburg <mal@egenix.com> added the comment: Nick Coghlan wrote:
Nick Coghlan <ncoghlan@gmail.com> added the comment:
Even "from __future__ import unicode_literals" doesn't make it do anything.
Perhaps Christian merged it by mistake in [5341b30b1812]?
It looks more like some parts were left out in the merge by accident: The 2.7 code only raises the warning for bytearrays, not for bytes and buffers, whereas 3.2 issues the warning for all bytes and bytearray (but not buffers; which makes sense, since Python 3 no longer has a text buffer interface). ---------- nosy: +lemburg _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
moijes12 <moijes12@gmail.com> added the comment: Hi Marc I tried reproducing this for bytearray using Python 2.7.2 but I can't see a warning. devel@moses:~$ python --version Python 2.7.2+ devel@moses:~$ cat test_py.py k = range(10) kb = bytearray(k) print kb kb = bytearray("hi") print kb devel@moses:~$ python -b test_py.py hi devel@moses:~$ python -bb test_py.py hi Please correct me if I'm wrong anywhere here. I'd like to work on this. ---------- nosy: +moijes12 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Changes by moijes12 <moijes12@gmail.com>: ---------- nosy: -moijes12 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Martin Panter added the comment: Try this to trigger a warning: python2 -b -c 'bytearray("3") == u"3"' -c:1: BytesWarning: Comparison between bytearray and string ---------- nosy: +vadmium _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Emily Zhao added the comment: Might be worth making this addition from 3 (I'm not sure how to add this to 2) -b : issue warnings about str(bytes_instance), str(bytearray_instance) and comparing bytes/bytearray with str. (-bb: issue errors) Building on Martin's example: On all of these, python2 is Python 2.7.6 (default, Apr 6 2014, 23:14:26) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin Type "help", "copyright", "credits" or "license" for more information. emily-mba:cpython emily$ python2
bytearray("3") == u"3" False
emily-mba:cpython emily$ python2 -b
bytearray("3") == u"3" __main__:1: BytesWarning: Comparison between bytearray and string False
emily-mba:cpython emily$ python2 -bb
bytearray("3") == u"3" Traceback (most recent call last): File "<stdin>", line 1, in <module> BytesWarning: Comparison between bytearray and string
---------- nosy: +emily.zhao _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Emily Zhao added the comment: Here's an attempt (based on 3's main.c http://hg.python.org/cpython/file/8866ac6f2269/Modules/main.c) ---------- keywords: +patch Added file: http://bugs.python.org/file35528/issue11681.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Martin Panter added the comment: Trouble is, in Python 2 bytes() and str() are the same thing, and most of those conditions don’t apply. Maybe something like this is more correct: -b : issue warnings about comparing bytearray with unicode. (-bb: issue errors) ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA@GMail.Com>: ---------- nosy: +Arfrever _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Martin Panter added the comment: As well as the usage string, it is missing from the list of options in the main documentation at Doc/using/cmdline.rst. There are a couple of places (sys.rst and warnings.rst) that reference :option:`-b`, and newer versions of Sphinx complain about this. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Greg Bengeult added the comment: I have attached a patch for 2.7 that adds -b and -bb to the command line documentation in Modules/main.c and Doc/library/cmdline.rst, following the suggested text provided by Martin. Interestingly, unicode(), bytearray(), and str() don't seem to be transitive in 2.7.
u"3" == str(3) True
str(3) == bytearray("3") True
u"3" == bytearray("3") False
---------- nosy: +gbengeult Added file: http://bugs.python.org/file46261/b_option.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Nick Coghlan added the comment: Right, the lack of transitivity comes from the fact that: >>> u"3" == str(3) True Is really shorthand for: >>> u"3" == str(3).decode("ascii") True However, the implicit decoding only triggers for *exactly* bytes instances, so in the bytearray case it needs to be explicit: >>> u"3" == bytearray(b"3").decode("ascii") True ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Nick Coghlan added the comment: Added some review comments on the patch. The only critical one was changing a :class:`bytes` reference to :class:`unicode`, but I also suggested it may be worth mentioning that it *doesn't* enable warnings about all binary/text comparisons the way the Python 3 one does. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Greg Bengeult added the comment: Thanks for the :class: markup edits. I'm still learning this stuff. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Changes by Zachary Ware <zachary.ware@gmail.com>: ---------- pull_requests: +1660 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Zachary Ware added the comment: New changeset f6c6d1e2304a06efdf65c60c3e1c6bbe1f192fd0 by Zachary Ware in branch '2.7': bpo-11681: Document the `-b` and `-bb` options (GH-1562) https://github.com/python/cpython/commit/f6c6d1e2304a06efdf65c60c3e1c6bbe1f1... ---------- nosy: +zach.ware _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
Zachary Ware added the comment: Thanks for the patch, Greg! I implemented Nick's suggestions and merged it. ---------- resolution: -> fixed stage: needs patch -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue11681> _______________________________________
participants (11)
-
Arfrever Frehtes Taifersar Arahesis -
Emily Zhao
-
Ezio Melotti
-
Greg Bengeult
-
Marc-Andre Lemburg
-
Martin Panter
-
moijes12
-
Nick Coghlan
-
R. David Murray
-
Zachary Ware
-
Éric Araujo