[Python-Dev] Using environment variables problematic for embedded python.

[Campbell Barton]

Checked Python 3.4 (latest hg checkout) and this area of the code
hasn't changed: pythonrun,c initstdio()

reported the issue:
http://bugs.python.org/issue16129

On Thu, Oct 4, 2012 at 2:51 PM, Guido van Rossum <guido at python.org> wrote:
> This seems more fit for the tracker; can you file there? (Then post the
> issue link here.) I do think you have a legitimate use case to set the
> default encoding to utf-8. (Though there may be a way already.) Does Python
> 3.3 have te same bug?
>
>
> On Wednesday, October 3, 2012, Campbell Barton wrote:
>>
>> On Thu, Oct 4, 2012 at 1:35 PM, Campbell Barton <ideasman42 at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > We've run into an issue recently with blender3d on ms-windows where we
>> > want to enforce the encoding is UTF-8 with the embedded python
>> > interpreter.
>> > (the encoding defaults to cp437).
>> >
>> > I naively thought setting the environment variable before calling
>> > Py_Initialize() would work, but the way python DLL loads, it gets its
>> > own environment variables that cant be modified directly [1].
>> > eg, _putenv("PYTHONIOENCODING=utf-8:surrogateescape");
>> >
>> > We had bug reports by windows users not able to export files because
>> > the stdout errors on printing paths with unsupported encoding. [2],[3]
>> >
>> > ---
>> >
>> > Of course we could distribute blender with a bat file launcher that
>> > sets env variables, or ask the user to set their env variable - but I
>> > dont think this is really a good option.
>> >
>> > I tried overriding the stderr & stdout, but this caused another bug in
>> > a part of out code that catches exception messages from the stderr.
>> > [4]
>> > import sys, io
>> > sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8',
>> > errors='surrogateescape', line_buffering=True)
>> > sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8',
>> > errors='surrogateescape', line_buffering=True)
>> >
>> >
>> >
>> > IMHO either of these solutions would be fine.
>> >
>> > * have a PyOS_PutEnv() function, gettext has gettext_putenv() to
>> > workaround this problem.
>> >
>> > * manage this the same as Py_GetPythonHome(), which can be defined by
>> > the embedding application to override the default.
>> >
>> >
>> > Id like to know if there is some known solution to workaround this
>> > issue, if not - would either of these would be acceptable in python
>> > (can write up a patch if it helps)
>> >
>> > Regards,
>> > Campbell
>> >
>> > ---
>> >
>> > [1]
>> > http://stackoverflow.com/questions/5153547/environment-variables-are-different-for-dll-than-exe
>> > [2] http://projects.blender.org/tracker/index.php?func=detail&aid=32750
>> > [3] http://projects.blender.org/tracker/index.php?func=detail&aid=31555
>> > [4] http://projects.blender.org/tracker/?func=detail&aid=32720
>>
>> To follow up and give a correction to overwriting sys.stdout/stderr,
>> The issue seemed to be that __stderr__/__stdout__ was later
>> overwritten, loosing the original reference to the buffer (maybe
>> refcount issue here?), either way it would silence the output.
>>
>>
>> import sys, io
>> sys.__stdout__ = sys.stdout =
>> io.TextIOWrapper(io.open(sys.stdout.fileno(), "wb", -1),
>> encoding='utf-8', errors='surrogateescape', newline="\n",
>> line_buffering=True)
>> sys.__stderr__ = sys.stderr =
>> io.TextIOWrapper(io.open(sys.stderr.fileno(), "wb", -1),
>> encoding='utf-8', errors='surrogateescape', newline="\n",
>> line_buffering=True)
>>
>> This all works as expected without bug [4] (above), however on exit I
>> get an assert in MSVCR90.DLL's write.c (called from python32_d.dll):
>> _VALIDATE_CLEAR_OSSERR_RETURN((_osfile(fh) & FOPEN), EBADF, -1);
>>
>> I'd rather not loose more time debugging why this assert happens,
>> IMHO this is too low-level a way to change the encoing of stdio/stderr
>> and error-prone too, so some way to reliably set PYTHONIOENCODING from
>> a program embedding python is still needed.
>>
>> --
>> - Campbell
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)



-- 
- Campbell