[Python-bugs-list] [ python-Bugs-422686 ] sys.path initialization in embedded Pyth

Tue, 25 Sep 2001 06:45:12 -0700

Bugs item #422686, was opened at 2001-05-09 09:14
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=422686&group_id=5470

Category: Python Interpreter Core
Group: None
>Status: Closed
>Resolution: Wont Fix
Priority: 5
Submitted By: Frederic Giacometti (giacometti)
Assigned to: Guido van Rossum (gvanrossum)
Summary: sys.path initialization in embedded Pyth

Initial Comment:

Context:
======
Python 2.1 final release, working with build from source (c:\program files\python does not exist), 
and PYTHONHOME is set to point to the root source directory (c:\local\src\Python-2.1).
Problem shows up on Windows (other platforms?)

Problem:
======
The directory of the executable (which is not python.exe) shows up in second position in sys.path.

Ex:
==
When the process executable (__argv value) is c:\jdk1.3.0_02\bin\java.exe, I get:

sys.path = ['c:\local\src\jpe', 'c:\local\src\python-2.1\dlls', 'c:\local\src\python-2.1\lib', 
'c:\local\src\python-2.1\lib\plat-win', 'c:\local\src\python-2.1\lib\lib-tk', 
'c:\jdk1.3.0_02\bin', 'c:\local\src\python-2.1']

Note:
===
If PYTHONHOME is not set in the environment, I get a 'site module not found' message upon Python 
initialization, and then a corrupted import behavior, with corrupted values in sys.modules.
Maybe the Python VM should exit right away when the site module cannot be imported, instead of 
going on and working with a corrupted sys.modules list.

I did not have this problem on Python 1.5.2.

I looked at the sys.path initialization C code (getpath() function, windows-specific code...); the 
code seems quite complex, and I could not trace easily where the executable's directory was 
picked up.

By default of finding the source of the problem, an external fix could be a rule of the kind: Do not 
add the executable directory to sys.path when the executable basename not in (python, 
python.exe).

Frederic Giacometti

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-17 09:40

Message:
Logged In: YES 
user_id=93657

I'll try to trace the problem on Windows - but I only have
Unix boxes on the client site where I work this month; I
have to do it at home, and my wife now gets mad at me any
time she sees an xemacs window on my laptop :((.

I completely agree with the 'python -i' thing :))

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-17 08:32

Message:
Logged In: YES 
user_id=6380

Strange. I cannot reproduce this on Unix. Can you run your
test program under the debugger on Windows to see where
sys.path is modfied?

Let's forget about -i; if Java is doing some low-level
messing with stdin, you can't really blame Python if it goes
wrong!

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-17 08:20

Message:
Logged In: YES 
user_id=93657

I'm attaching a simple test program and its Makefile, for build with VC++.
Running 'nmake' after sourcing the VC++ environment (vcvars.bat) will do the build and print the test 
output.

Whatever be the configuration (PYTHONPATH set, not set, current working directory...), the directory
containing the executable (main.exe) will always be listed in sys.path.
This is a potential problem, to the extent that one cannot make assumption on the *.py files and directories 
contained in the executable directory.

I'm pasting below a sample run, where 'c:\local\src\testpath' shows up in sys.path (not normal), in addition 
ot the current working directory (c:\local\src - normal):

C:\local\src\testpath>nmake

        cl c:\local\src\Python\PCbuild\python21.lib main.c
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86
Copyright (C) Microsoft Corp 1984-1998. All rights reserved.

main.c
Microsoft (R) Incremental Linker Version 6.00.8168
Copyright (C) Microsoft Corp 1992-1998. All rights reserved.

/out:main.exe
c:\local\src\Python\PCbuild\python21.lib
main.obj
        cd .. && testpath\main.exe
['c:\program files\python21\lib\plat-win', 'c:\program files\python21\lib
', 'c:\program files\python21\dlls', 'c:\program files\python21\lib\lib-t
k', 'c:\local\src', 'c:\local\src\testpath']
PYTHONPATH=None
cwd=C:\local\src

C:\local\src\testpath>

-----------------------------

As for 'python -i', I was giving an exemple of probelm that occurs when modifying global process ressources 
[here, the -i option operates a setvbuf() call on stdin; the Java VM also seems to do some operation there 
(modifying stdin low-level file descriptor flags), that ends up in Python syntax errors on interactive carriage 
return, after the Java VM is started, but only with 'python -i'...].
I was giving this as an exemple in the same vein as modifying argv[]: one cannot help that there is an 
unlikely, but nevertheless existing possibility that there be sometimes another library that uses the variable 
in the same time; and that modifying it could mess up something somewhere...
Some would also quote 'Murphy's law' :))

Frederic Giacometti

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-05 11:19

Message:
Logged In: YES 
user_id=6380

Reassigning this to me, I don't think Tim can help.

Frederic, your explanations are inscrutable. "Py_Main()
isn't called, but it is called." "python -i and python -i"
does *what*?  Who are "the guys"? Sorry, I just don't see
the bug report in there.

Can you provide a small test program that exhibits the
problem?

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-04 19:16

Message:
Logged In: YES 
user_id=93657

In this context, only the Py_Initialize
(and the infamously versatile Py_ThreadInit at some point too) functions are called.

Py_Main() is not called there.

Py_Main() is called only in the python.PyRun.main() Java function,
so that 'java python.PyRun' emulates the python command
(provided as a turn around only because some Unix platform had problems finding and opening dynamically 
the Java shared library).

But, when python is called from a regular Java program,
only the standard embedded procedures are used
(Py_Initialize, Py_InitThreads...).

Sorry...

But, who knows, maybe Java also messes up some process global data ;-\
For curiosity, in this order of things:
 'python -i', and 'python -i' only yields a syntax error on the interactive carriage return right after the JVM 
is 
initialized... It sounds to me like some conflict on the stdin file descriptor flags setup (which seems only to 
occur with python -i)..
I advised the guys to use the cmd package instead of calling python -i in their applications.
I did not raise a bug report on this one...

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-04 18:32

Message:
Logged In: YES 
user_id=6380

PySys_SetArgv() is also called from Py_Main().

Don't you call that?  (I recall we had an argument about it.
:-)

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-04 17:47

Message:
Logged In: YES 
user_id=93657

I don't call PySys_SetArgv(), so this is the default behavior.

But it's true that calling PySys_SetArgv() could be a solution.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-04 11:19

Message:
Logged In: YES 
user_id=6380

The code that inserts the directory is not in getpath() but
in Python/sysmodule.c, in PySys_SetArgv(). (Most of this
function is devoted to this action, in fact.) I propose that
if you don't want the executable's directory in sys.argv,
you don't pass it to PySys_SetArgv().

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=422686&group_id=5470