[issue16633] os.environ updates only one copy of env vars under Windows (GetEnvironmentVariable vs. getenv)
New submission from Václav Šmilauer: On windows, environment variables exist in two copies: one is manipulated using win32 API (GetEnvironmentVariable, SetEnvironmentVariable), and another one is maintained by the C runtime (getenv, _putenv). This is explained in more depth in [1]. os.environ manipulates win32 environment variables, but *not* those seen by the CRT. This means that if I set an environment variable using os.environ and later read it, in the same process, using getenv in an extension module, it will not give the expected result. Child processes *do* see those vars in CRT, since it is copied over from the win32 version at process startup. Setting env vars has legitimate uses, since it is one of the few ways to influence initialization of extension modules: for instance, setting OMP_NUM_THREADS sets number of threads for OpenMP runtime (which cannot be changed once the module using it is loaded). It would be ideal to keep both CRT and win32 env vars in sync transparently. If that is not realistically achievable, this gotcha should be documented as a warning in the os.environ documentation. A workaround to this problem to set variables in both win32 and CRT using something like import ctypes, ctypes.util, os.environ ctypes.cdll[ctypes.util.find_msvcrt()]._putenv("%s=%s"%(name,value)) os.environ[name]=value [1] http://msmvps.com/blogs/senthil/archive/2009/10/13/when-what-you-set-is-not-... ---------- assignee: docs@python components: Documentation messages: 177081 nosy: docs@python, eudoxos priority: normal severity: normal status: open title: os.environ updates only one copy of env vars under Windows (GetEnvironmentVariable vs. getenv) type: behavior versions: Python 2.7, Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16633> _______________________________________
Andrew Svetlov added the comment: I think the problem is: windows process can contain single shared win32 API and several CRT copies — one for Visual Studio 2008, other for VS 2010 etc. We cannot know which CRT version is used by particular extension, so I doubt we can initialize it properly. Your workaround has the same problem. ---------- nosy: +asvetlov _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16633> _______________________________________
Tim Golden added the comment: Does the same problem obtain if you use os.putenv (which calls the crt putenv under the covers)? ---------- nosy: +tim.golden _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16633> _______________________________________
Václav Šmilauer added the comment: I checked on Windows 7 64bit with python 2.7 (sorry, not python 3.3 installed here) with the script attached. Each line sets a variable using the method in the very left column, then it attempts to read it back using all methods. os.environ win32 os.?etenv msvcrt msvcr? os.environ OK OK OK -- -- win32 -- OK -- -- -- os.?etenv -- OK -- -- -- msvcrt -- OK -- OK -- msvcr? -- OK -- -- -- Methods which can read back what they also set are os.environ, win32api, msvcrt. OTOH, msvcr? (which is ctypes.util.find_msvcr(), in my case msvcr90.dll) does not read its own values, just like os.getenv/os.putenv. It *seems* that reading with win32 API is very reliable (that is not a good news for writing cross-platform extension modules), though, but perhaps it just asks the CRT if it does not find the variable defined (my testing did not go that far). @Andrew: you're probably right, though it does not explain, why msvcr90.dll does not read back the values I set in there - that is the CRT python27.dll itself links to -- ---------- Added file: http://bugs.python.org/file28243/tryenv2.py _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16633> _______________________________________
Jerome Dubois added the comment: We encountered the same issue when upgrading our application's JRE from 1.5 to 1.6. We use a kind of grid, which has engines written in C, which uses an embedded JAVA JVM, which loads C dll. One of these uses python. Here is what happens: 1. C executable launches an embedded JVM, which loads its required dlls. 2. JVM sets PYTHONHOME environment variable through windows kernel API: SetEnvironmentVariable 3. JVM loads our C dlls to start computations 3. One of the dlls invokes Py_InitializeEx, which must read PYTHONHOME in some way (C runtime getenv?). Python 2.5 here, with MSVCR71.dll loaded. 4. Computations are done with invoked python With JRE 1.5, Python is able to get correct PYTHONHOME, and can do "import os" in our script. With JRE 1.6, this is not the case, as JRE 1.6 loads MSVCR71.dll @ step 1. JRE 1.5 did not. As stated in the previous comments, and from my understanding, in Windows there is the "Process Environment Variables Space" and possibly several "C Runtime Environment Variables Space". The first time a C runtime dll is loaded, it copies the Process Env Var into its own buffer. Our JRE 1.6 loads msvcr71.dll (C runtime), and so copies env var @ step 1. It happens before we set PYTHONHOME with JAVA @ step 2. To correct this, we had to use the Py_SetPythonHome function before calling Py_PyInit to set PYTHONHOME ourselves This way, Python executes our code fine when we use JRE 1.6. But this is because we do not call any getenv functionality within python at the moment, but it could happen in the future... As stated by eudoxos, the safest solution for windows is to use GetEnvironmentVariable (win32 kernel API). Is there any schedule for a fix for this problem? Thanks for you time and answer. Regards. ---------- nosy: +goungy _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16633> _______________________________________
Andrew Svetlov added the comment: I think the problem is: windows process can contain single shared win32 API and several CRT copies — one for Visual Studio 2008, other for VS 2010 etc. We cannot know which CRT version is used by particular extension, so I doubt we can initialize it properly. Your workaround has the same problem. ---------- nosy: +asvetlov _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16633> _______________________________________
Tim Golden added the comment: Does the same problem obtain if you use os.putenv (which calls the crt putenv under the covers)? ---------- nosy: +tim.golden _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16633> _______________________________________
Václav Šmilauer added the comment: I checked on Windows 7 64bit with python 2.7 (sorry, not python 3.3 installed here) with the script attached. Each line sets a variable using the method in the very left column, then it attempts to read it back using all methods. os.environ win32 os.?etenv msvcrt msvcr? os.environ OK OK OK -- -- win32 -- OK -- -- -- os.?etenv -- OK -- -- -- msvcrt -- OK -- OK -- msvcr? -- OK -- -- -- Methods which can read back what they also set are os.environ, win32api, msvcrt. OTOH, msvcr? (which is ctypes.util.find_msvcr(), in my case msvcr90.dll) does not read its own values, just like os.getenv/os.putenv. It *seems* that reading with win32 API is very reliable (that is not a good news for writing cross-platform extension modules), though, but perhaps it just asks the CRT if it does not find the variable defined (my testing did not go that far). @Andrew: you're probably right, though it does not explain, why msvcr90.dll does not read back the values I set in there - that is the CRT python27.dll itself links to -- ---------- Added file: http://bugs.python.org/file28243/tryenv2.py _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16633> _______________________________________
Jerome Dubois added the comment: We encountered the same issue when upgrading our application's JRE from 1.5 to 1.6. We use a kind of grid, which has engines written in C, which uses an embedded JAVA JVM, which loads C dll. One of these uses python. Here is what happens: 1. C executable launches an embedded JVM, which loads its required dlls. 2. JVM sets PYTHONHOME environment variable through windows kernel API: SetEnvironmentVariable 3. JVM loads our C dlls to start computations 3. One of the dlls invokes Py_InitializeEx, which must read PYTHONHOME in some way (C runtime getenv?). Python 2.5 here, with MSVCR71.dll loaded. 4. Computations are done with invoked python With JRE 1.5, Python is able to get correct PYTHONHOME, and can do "import os" in our script. With JRE 1.6, this is not the case, as JRE 1.6 loads MSVCR71.dll @ step 1. JRE 1.5 did not. As stated in the previous comments, and from my understanding, in Windows there is the "Process Environment Variables Space" and possibly several "C Runtime Environment Variables Space". The first time a C runtime dll is loaded, it copies the Process Env Var into its own buffer. Our JRE 1.6 loads msvcr71.dll (C runtime), and so copies env var @ step 1. It happens before we set PYTHONHOME with JAVA @ step 2. To correct this, we had to use the Py_SetPythonHome function before calling Py_PyInit to set PYTHONHOME ourselves This way, Python executes our code fine when we use JRE 1.6. But this is because we do not call any getenv functionality within python at the moment, but it could happen in the future... As stated by eudoxos, the safest solution for windows is to use GetEnvironmentVariable (win32 kernel API). Is there any schedule for a fix for this problem? Thanks for you time and answer. Regards. ---------- nosy: +goungy _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16633> _______________________________________
participants (4)
-
Andrew Svetlov -
Jerome Dubois
-
Tim Golden
-
Václav Šmilauer