[Python-Dev] Re: Path hacking

[Guido van Rossum]

[Guido]
> >         import Spam_path
> > 
> >     before importing anything else.

[JimA]
> This should not be necessary if you use the name "sitecustomize" instead
> of "Spam_path" right?  The file sitecustomize.py is automatically
> imported.
> Actually all this sounds like site.py all over again.

But the intention here is for the customization to be application
specific (hence the Spam in the name).  sitecustomize doesn't know
whethere I need the Mailman or the Knowbot root added to my path.

Or do you mean to imply that we can do this with zero text added to
the script, by simply dropping an appropriate sitecustomize.py in the
script dir?  Unfortunately this does currently *not* work, because
sys.path[0] is added after Py_Initialize() is run.

> >     Your installer, once it knows the absolute pathname of your
> >     application's root directory, crafts a file Spam_path.py which
> >     contains code that inserts the right absolute pathname into sys.path.
> 
> I don't think this is necessary either.  The sys module is available.
> So sitecustomize.py can say:
>   import sys
>   mydir = sys.path[0]
>   if not mydir:
>     import os
>     mydir = os.getcwd()
>   sys.path = [mydir]  # To be really extreme about it
>   # Note: inserting mydir as sys.path[0] should be redundant but is not

Hm, guessing based on the script directory might work, but seems less
reliable than hardcoding it through the installer.  But you can use
this if it works for your application.

> >     Your installer then installs a copy of this file (or a symbolic link
> >     to it) *in each bin directory where it installs top-level Python
> >     scripts*.
> > 
> >     Because the script's directory is first on the default path, the Spam
> >     scripts will pick up Spam_path without any help from $PYTHONPATH.
> 
> Hmmm.  Is this really true?  Nothing else, for example the registry, can
> change sys.path[0]?  Ever?  Please say yes.

Yes.  (The registry can add module-specific paths, which will be
searched before sys.path is even looked at, but this is only for
specific modules.  It cannot insert a general directory that is
searched.)  The only way this can fail is if an embedding app fails to 
call PySys_SetArgv().

> >     I know this doesn't resolve the relative import thread (how's that
> >     going by the way? :-) but Barry & Fred & I agree that this is the best
> >     solution to the problem stated in Barry's message to which I am
> >     following up here.
> 
> This is a good idea, but there are a few problems.
> 
> It depends on sys.path[0] being the directory of the Python
> file being executed as the main program.  I guess I never
> really trusted this before.  I think if this is the case it
> should never be ''.  A relative path or no path on the command
> line (the __main__ program) should be replaced by the full path
> in the sys module setup.  Then the "mydir = os.getcwd()" above
> is not necessary.  And inserting mydir as sys.path[0] is truly
> redundant should the current directory change (as it certainly will).
> This is currently a problem with sys.path[0] which should be
> fixed no matter what else happens.

I have always resisted forcing path items to be absolute, although I'm
not sure that my reasons are valid any more (it has to do with the
fact that getcwd() may fail and the fact that portable path
concatenation is a pain).  In any case, that's a separate issue -- I
agree that if sys.path[0] is '' (as it often is) it's better for
site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize
it (and everything else on the path) so that it will still work if the
app does a chdir().

> The files exceptions.py and site.py must be in all the bin
> directories as well as sitecustomize.py because they are
> automatically imported in Py_Initialize().

Yes.

> The above doesn't work when you start the Python command
> interpreter (no main).  I know, its a minor point.

You could add the "import Spam_path" to your $PYTHONSTARTUP file.

> It seems to me this totally solves Jim Fulton's and Marc's
> problem and makes "__" unnecessary.  You just install zope
> and mx in zopedir, perform the above, and presto you have a new
> private name space where you can control all your names.  But
> there must be some problem here I haven't thought of.

I think no simple solution that *I* can come up with will satisfy
JimF's and Marc's desire for obscurity :-)

> I still worry that this is not powerful enough.  Greg Stein
> has volunteered to re-write import.c in Python (using imputil.py)
> and this is a Great Idea.  Lots of Python could probably be
> written in itself.  I would like to try writing the main
> program in Python and eliminating the special freeze main
> program.  Once you start on this road (and I think it is a good road)
> you have Python code which is more truly part of the binary
> interpreter than a library.

Yes, this is the plan for Python 2.0, and some of it may be
implemented in Python 1.6.

> Proposal:
> 
> Use a special PYTHONPATH on startup to find "special" Python
> files which are really part of the interpreter.  There are
> three directories Python knows about.  Namely sys.path[0]
> (once it is fixed), sys.executable and sys.dllfullpath,
> the directory of python15.dll or other shared library (once it is
> added to sys).  How about prepending the single directory sys.executable
> to sys.path during Py_Initialize()?  And demanding that modules
> like the new Greg_importer.py[c], exceptions.py[c] and site.py[c]
> be placed there.

On Unix, this is a bin directory and it is strongly discouraged to put
non-program files there.  Python already does something similar --
it looks around in sys.executable's ancestors for a specific landmark, 
currently lib/python<version>/string.py.  Arguably, it should search
for execeptions.py instead.

> Actually I would prefer sys.dllfullpath if it exists, since that
> is where the interpreter is, and I am trying to associate these
> special internal Python files exactly with their correct Python
> interpreter.

Is the full DLL path available at any point?  This would certainly be
a good starting point -- especially when the DLL is loaded implicitly
as the result of some COM operation.

> Alternative Proposal:
> 
> Py_Initialize() first imports its files from sys.executable + '/' +
> PyInternal.pyl (again I prefer sys.dllfullpath).
> PyInternal.pyl is a Python library file (like a Java Jar
> file) which would contain modules like exceptions, etc.
> The PyInternal.pyl file has the standard Python library file
> format (whatever that turns out to be).  It is not an error if
> this file is absent.

I guess this is all up to the redesign of the import mechanism
(something like Greg Stein's imputil.py for sure).

--Guido van Rossum (home page: http://www.python.org/~guido/)