[New-bugs-announce] [issue29818] Py_SetStandardStreamEncoding leads to a memory error in debug mode

[Nick Coghlan]

New submission from Nick Coghlan:

For PEP 538, setting PYTHONIOENCODING turned out to have undesirable side effects on Python 2 instances in subprocesses, since Python 2 has no 'surrogateescape' error handler.

So I switched to using the "Py_SetStandardStreamEncoding" API defined in http://bugs.python.org/issue16129 instead, but this turns out to have problematic interactions with the dynamic memory allocator management, so it fails with a fatal exception in debug mode. An example of the error can be seen here: https://travis-ci.org/python/cpython/jobs/211293576

The problem appears to be that between the allocation of the memory with `_PyMem_RawStrdup` in `Py_SetStandardStreamEncoding` and the release of that memory in `initstdio`, the active memory manager has changed (at least in a debug build), so the deallocation as part of the interpreter startup fails.

That interpretation is based on this comment in Programs/python.c:

```
    /* Force again malloc() allocator to release memory blocks allocated
       before Py_Main() */
    (void)_PyMem_SetupAllocators("malloc");
```

The allocations in Py_SetStandardStreamEncoding happen before the call to Py_Main/Py_Initialize, but the deallocation happens in Py_Initialize.

The "fix" I applied to the PEP branch was to make the default allocator conditional in Programs/python.c as well:

```
#ifdef Py_DEBUG
    (void)_PyMem_SetupAllocators("malloc_debug");
#  else
    (void)_PyMem_SetupAllocators("malloc");
#  endif
```

While that works (at least in the absence of a PYTHONMALLOC setting) it seems fragile. It would be nicer if there was a way for Py_SetStandardStreamEncoding to indicate which allocator should be used for the deallocation.

----------
messages: 289668
nosy: haypo, ncoghlan
priority: normal
severity: normal
stage: needs patch
status: open
title: Py_SetStandardStreamEncoding leads to a memory error in debug mode
type: crash

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue29818>
_______________________________________