Marc-Andre Lemburg added the comment: On 28.11.2016 12:24, Serhiy Storchaka wrote:

...

Why are you removing the introductory section on how mappings work ?

Because it is not correct. I copied it to descriptions of concrete functions with correcting it according to the peculiarity of particular function.

The only part that is not correct is "single string characters". This should read "single bytes" or "bytes strings of length 1". I also don't see where you copied the description. Without some description of what "mappings" are in the context of the charmap codec, it's not easy to understand what the purpose of these APIs is. Please just fix the bytes wording instead of removing the whole intro.

...

Also, this wording needs to be corrected: "bytes (integers in the range from 0 to 255)". Bytes are not integers. I'd suggest to use the more correct wording "bytes strings of length 1".

The word "bytes" means here not Python bytes object, but is used in more common meaning: an integer in the range from 0 to 255.

That's confusing, since we use the term "bytes" as referring to the bytes object in Python. Please use "integers in the range 0-255". Aside: The deprecation of PyUnicode_EncodeCharmap() also seems misplaced in this context, since only the Py_UNICODE version of the API is deprecated. The functionality still exists and is useful. An API similar to the _PyUnicode_EncodeCharmap() API should be made publicly available to accommodate for the deprecation, since the mentioned PyUnicode_AsCharmapString() and PyUnicode_AsEncodedString() APIs are not suitable as replacement. PyUnicode_AsCharmapString() doesn't support error handling (strange, BTW) and PyUnicode_AsEncodedString() has a completely unrelated meaning (no idea why it's mentioned here at all). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28749> _______________________________________

Marc-Andre Lemburg added the comment:

...

The only part that is not correct is "single string characters". This should read "single bytes" or "bytes strings of length 1".

This is not correct. Decoding mappings map not bytes strings, but integers.

Looking at the implementation, you're right. AFAIR, the first incarnation of the charmap codec used single chars in Python 2. I guess the documentation was never updated when the change was made to use integers instead.

And this is not the only incorrect part. Decoding mappings can map to multicharacter Unicode strings, not to single Unicode characters. Not just None, but the integer 0xfffe and Unicode string '\ufffe' mean "undefined mapping".

Yes, this was added later on as well. Apparently the docs were never updated.

There are similar incorrectnesses about encoding mappings.

...

I also don't see where you copied the description. Without some description of what "mappings" are in the context of the charmap codec, it's not easy to understand what the purpose of these APIs is. Please just fix the bytes wording instead of removing the whole intro.

Decoding mappings were desribed in the introduction and in the description of PyUnicode_DecodeCharmap() (both are outdated and incomplete). I merged and corrected descriptions and left it only in one place, since PyUnicode_DecodeCharmap() is the only function that needs this. Same for encoding mappings. Both decoding and encoding mappings do not have a relation to PyUnicode_Translate(). The paragraph about a LookupError in the introduction was totally wrong. I left in the introduction only common

Ok, fair enough, let's remove the two paragraphs. part.

Other details are too different in decoding, encoding and translation mappings.

...

...

...

Also, this wording needs to be corrected: "bytes (integers in the range from 0 to 255)". Bytes are not integers. I'd suggest to use the more correct wording "bytes strings of length 1".> The word "bytes" means here not Python bytes object, but is used in more common meaning: an integer in the range from 0 to 255. That's confusing, since we use the term "bytes" as referring to the bytes object in Python. Please use "integers in the range 0-255".

Okay, I'll remove the word "bytes" here. But how would you formulate the following sentence: "Unmapped bytes (ones which cause a :exc:`LookupError`) as well as mapped to ``None``, ``0xFFFE`` or ``'\ufffe'`` are treated as "undefined mapping" and cause an error."?

Better: """ If *mapping* is *NULL*, Latin-1 decoding will be applied. Else *mapping* must map bytes ordinals (integers in the range from 0 to 255) to Unicode strings, integers (which are then interpreted as Unicode ordinals) or ``None``. Unmapped data bytes - ones which cause a :exc:`LookupError`, as well as ones which get mapped to ``None``, ``0xFFFE`` or ``'\ufffe'``, are treated as undefined mappings and cause an error. """

...

Aside: The deprecation of PyUnicode_EncodeCharmap() also seems misplaced in this context, since only the Py_UNICODE version of the API is deprecated. The functionality still exists and is useful. An API similar to the _PyUnicode_EncodeCharmap() API should be made publicly available to accommodate for the deprecation, since the mentioned PyUnicode_AsCharmapString() and PyUnicode_AsEncodedString() APIs are not suitable as replacement. PyUnicode_AsCharmapString() doesn't support error handling (strange, BTW) and PyUnicode_AsEncodedString() has a completely unrelated meaning (no idea why it's mentioned here at all).

Only PyUnicode_EncodeCharmap() is deprecated, PyUnicode_AsCharmapString() is not deprecated. I placed the deprecated function just after its non-deprecated counerpart following the pattern for other deprecated functions. If you prefer I'll move both deprecated functions (PyUnicode_EncodeCharmap and PyUnicode_TranslateCharmap) together at the end of this section.

No, I'd prefer this deprecation to be undone as long as we don't have a proper alternative for the API. Looking at the various deprecations for the Py_UNICODE APIs, I find that the Unicode API symmetry was severely broken. In the Python 2 API, we always have an PyUnicode_Encode...() and corresponding PyUnicode_Decode...() API for every codec. In Python 3, the encode APIs were apparently all deprecated due to their use of Py_UNICODE and only the the much less useful PyUnicode_As...String() APIs were left, which intentionally do not have an error argument, because they were intended as quick replacement for PyString_AsString() uses in Python 2.

I don't know why PyUnicode_AsCharmapString() don't support the errors argument. I added PyUnicode_AsEncodedString() as a replacement (issue19569) because this is the only public non-deprecated way to do a charmap encoding with errors handling. There is no exact equivalent, but PyUnicode_AsCharmapString() and PyUnicode_AsEncodedString() cover different areas of using PyUnicode_EncodeCharmap().

The only way around this is to have new APIs to reestablish the previous encoding functionality (only based on PyObject *unicode instead of Py_UNICODE) at the C API level. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28749> _______________________________________