[New-bugs-announce] [issue22767] `separators` argument to json.dumps() behaves unexpectedly across 2.x vs 3.x

[Tom Christie]

New submission from Tom Christie:

This is one of those behavioural issues that is a borderline bug.

The seperators argument to `json.dumps()` behaves differently across python 2 and 3.

* In python 2 it should be provided as a bytestring, and can cause a UnicodeDecodeError otherwise.
* In python 3 it should be provided as unicode,and can cause a TypeError otherwise.

Examples:

    Python 2.7.2
    >>> print json.dumps({'snowman': '☃'}, separators=(':', ','), ensure_ascii=False)
    {"snowman","☃"}
    >>> print json.dumps({'snowman': '☃'}, separators=(u':', u','), ensure_ascii=False)
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1: ordinal not in range(128)

And:

    Python 3.4.0
    >>> print(json.dumps({'snowman': '☃'}, separators=(':', ','), ensure_ascii=False))
    {"snowman","☃"}
    >>> print(json.dumps({'snowman': '☃'}, separators=(b':', b','), ensure_ascii=False))
    <...>
    TypeError: sequence item 2: expected str instance, bytes found

Technically this isn't out of line with the documentation - in both cases it uses `separators=(':', ',')` which is indeed the correct type in both v2 and v3. However it's unexpected behaviour that it changes types between versions, without being called out.

Working on a codebase with `from __future__ import unicode_literals` this is particularly unexpected because we get a `UnicodeDecodeError` when running code that otherwise looks correct.

It's also slightly awkward to fix because it's a bit of a weird branch condition.

The fix would probably be to forcibly coerce it to the correct type regardless of if it is supplied as unicode or a bytestring, or at least to do so for python 2.7.

Possibly related to http://bugs.python.org/issue22701 but wasn't able to understand if that ticket was in fact a different user error.

----------
messages: 230274
nosy: Tom.Christie
priority: normal
severity: normal
status: open
title: `separators` argument to json.dumps() behaves unexpectedly across 2.x vs 3.x
type: behavior
versions: Python 2.7, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22767>
_______________________________________