[Python-Dev] Using environment variables problematic for embedded python.

[Campbell Barton]

On Thu, Oct 4, 2012 at 1:35 PM, Campbell Barton <ideasman42 at gmail.com> wrote:
> Hi,
>
> We've run into an issue recently with blender3d on ms-windows where we
> want to enforce the encoding is UTF-8 with the embedded python
> interpreter.
> (the encoding defaults to cp437).
>
> I naively thought setting the environment variable before calling
> Py_Initialize() would work, but the way python DLL loads, it gets its
> own environment variables that cant be modified directly [1].
> eg, _putenv("PYTHONIOENCODING=utf-8:surrogateescape");
>
> We had bug reports by windows users not able to export files because
> the stdout errors on printing paths with unsupported encoding. [2],[3]
>
> ---
>
> Of course we could distribute blender with a bat file launcher that
> sets env variables, or ask the user to set their env variable - but I
> dont think this is really a good option.
>
> I tried overriding the stderr & stdout, but this caused another bug in
> a part of out code that catches exception messages from the stderr.
> [4]
> import sys, io
> sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8',
> errors='surrogateescape', line_buffering=True)
> sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8',
> errors='surrogateescape', line_buffering=True)
>
>
>
> IMHO either of these solutions would be fine.
>
> * have a PyOS_PutEnv() function, gettext has gettext_putenv() to
> workaround this problem.
>
> * manage this the same as Py_GetPythonHome(), which can be defined by
> the embedding application to override the default.
>
>
> Id like to know if there is some known solution to workaround this
> issue, if not - would either of these would be acceptable in python
> (can write up a patch if it helps)
>
> Regards,
> Campbell
>
> ---
>
> [1] http://stackoverflow.com/questions/5153547/environment-variables-are-different-for-dll-than-exe
> [2] http://projects.blender.org/tracker/index.php?func=detail&aid=32750
> [3] http://projects.blender.org/tracker/index.php?func=detail&aid=31555
> [4] http://projects.blender.org/tracker/?func=detail&aid=32720

To follow up and give a correction to overwriting sys.stdout/stderr,
The issue seemed to be that __stderr__/__stdout__ was later
overwritten, loosing the original reference to the buffer (maybe
refcount issue here?), either way it would silence the output.


import sys, io
sys.__stdout__ = sys.stdout =
io.TextIOWrapper(io.open(sys.stdout.fileno(), "wb", -1),
encoding='utf-8', errors='surrogateescape', newline="\n",
line_buffering=True)
sys.__stderr__ = sys.stderr =
io.TextIOWrapper(io.open(sys.stderr.fileno(), "wb", -1),
encoding='utf-8', errors='surrogateescape', newline="\n",
line_buffering=True)

This all works as expected without bug [4] (above), however on exit I
get an assert in MSVCR90.DLL's write.c (called from python32_d.dll):
_VALIDATE_CLEAR_OSSERR_RETURN((_osfile(fh) & FOPEN), EBADF, -1);

I'd rather not loose more time debugging why this assert happens,
IMHO this is too low-level a way to change the encoing of stdio/stderr
and error-prone too, so some way to reliably set PYTHONIOENCODING from
a program embedding python is still needed.

-- 
- Campbell