
"Guido van Rossum" wrote:
--> The solution:
Ah, finally a specific proposal...
This should not be necessary if you use the name "sitecustomize" instead of "Spam_path" right? The file sitecustomize.py is automatically imported. Actually all this sounds like site.py all over again.
I don't think this is necessary either. The sys module is available. So sitecustomize.py can say: import sys mydir = sys.path[0] if not mydir: import os mydir = os.getcwd() sys.path = [mydir] # To be really extreme about it # Note: inserting mydir as sys.path[0] should be redundant but is not
Hmmm. Is this really true? Nothing else, for example the registry, can change sys.path[0]? Ever? Please say yes.
This is a good idea, but there are a few problems. It depends on sys.path[0] being the directory of the Python file being executed as the main program. I guess I never really trusted this before. I think if this is the case it should never be ''. A relative path or no path on the command line (the __main__ program) should be replaced by the full path in the sys module setup. Then the "mydir = os.getcwd()" above is not necessary. And inserting mydir as sys.path[0] is truly redundant should the current directory change (as it certainly will). This is currently a problem with sys.path[0] which should be fixed no matter what else happens. The files exceptions.py and site.py must be in all the bin directories as well as sitecustomize.py because they are automatically imported in Py_Initialize(). The above doesn't work when you start the Python command interpreter (no main). I know, its a minor point. It seems to me this totally solves Jim Fulton's and Marc's problem and makes "__" unnecessary. You just install zope and mx in zopedir, perform the above, and presto you have a new private name space where you can control all your names. But there must be some problem here I haven't thought of. I still worry that this is not powerful enough. Greg Stein has volunteered to re-write import.c in Python (using imputil.py) and this is a Great Idea. Lots of Python could probably be written in itself. I would like to try writing the main program in Python and eliminating the special freeze main program. Once you start on this road (and I think it is a good road) you have Python code which is more truly part of the binary interpreter than a library. Proposal: Use a special PYTHONPATH on startup to find "special" Python files which are really part of the interpreter. There are three directories Python knows about. Namely sys.path[0] (once it is fixed), sys.executable and sys.dllfullpath, the directory of python15.dll or other shared library (once it is added to sys). How about prepending the single directory sys.executable to sys.path during Py_Initialize()? And demanding that modules like the new Greg_importer.py[c], exceptions.py[c] and site.py[c] be placed there. Actually I would prefer sys.dllfullpath if it exists, since that is where the interpreter is, and I am trying to associate these special internal Python files exactly with their correct Python interpreter. Alternative Proposal: Py_Initialize() first imports its files from sys.executable + '/' + PyInternal.pyl (again I prefer sys.dllfullpath). PyInternal.pyl is a Python library file (like a Java Jar file) which would contain modules like exceptions, etc. The PyInternal.pyl file has the standard Python library file format (whatever that turns out to be). It is not an error if this file is absent. Jim Ahlstrom

[Guido]
import Spam_path
before importing anything else.
[JimA]
But the intention here is for the customization to be application specific (hence the Spam in the name). sitecustomize doesn't know whethere I need the Mailman or the Knowbot root added to my path. Or do you mean to imply that we can do this with zero text added to the script, by simply dropping an appropriate sitecustomize.py in the script dir? Unfortunately this does currently *not* work, because sys.path[0] is added after Py_Initialize() is run.
Hm, guessing based on the script directory might work, but seems less reliable than hardcoding it through the installer. But you can use this if it works for your application.
Yes. (The registry can add module-specific paths, which will be searched before sys.path is even looked at, but this is only for specific modules. It cannot insert a general directory that is searched.) The only way this can fail is if an embedding app fails to call PySys_SetArgv().
I have always resisted forcing path items to be absolute, although I'm not sure that my reasons are valid any more (it has to do with the fact that getcwd() may fail and the fact that portable path concatenation is a pain). In any case, that's a separate issue -- I agree that if sys.path[0] is '' (as it often is) it's better for site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize it (and everything else on the path) so that it will still work if the app does a chdir().
Yes.
The above doesn't work when you start the Python command interpreter (no main). I know, its a minor point.
You could add the "import Spam_path" to your $PYTHONSTARTUP file.
I think no simple solution that *I* can come up with will satisfy JimF's and Marc's desire for obscurity :-)
Yes, this is the plan for Python 2.0, and some of it may be implemented in Python 1.6.
On Unix, this is a bin directory and it is strongly discouraged to put non-program files there. Python already does something similar -- it looks around in sys.executable's ancestors for a specific landmark, currently lib/python<version>/string.py. Arguably, it should search for execeptions.py instead.
Is the full DLL path available at any point? This would certainly be a good starting point -- especially when the DLL is loaded implicitly as the result of some COM operation.
I guess this is all up to the redesign of the import mechanism (something like Greg Stein's imputil.py for sure). --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Never mind, I'll use an imputil.py based approach to get relative imports to work in my packages. That is when I get imputil.py to work... it doesn't seem to be quite there yet (or I'm using an old version). BTW, I'm 100% behind you guys if you choose to reimplement Python's import mechanism in Python using a similar approach as the one Greg implemented in imputil. Should make everybody happy: those who want obscure syntactic add-ons and others with a taste for zlib'ed packaged byte code, plus those VMS freaks ;-) Perhaps we should start a new thread on that topic... Still needed are: · Python level APIs for the platform specific magic on Win32 and Macs (OS/2, BeOS ?), e.g. access to the Windows registry and the Mac forks · Patches to make the DirectoryImporter 100% backward compatible Greg's imputil.py can be found at: http://www.lyra.org/greg/small/ The trick would then be to install an application specific importer in the setup module Spam_path or MyAppSetup which then takes care of all the rest... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Guido van Rossum wrote:
Ah, you have multiple scripts in one directory and multiple Foo_path, Bar_path etc. I was thinking with my Windows head. A commercial Windows app generally has its own exclusive install directory, so I was thinking single directory so a single sitecustomize.py.
Yes, that is exactly what I was thinking.
Unfortunately this does currently *not* work, because sys.path[0] is added after Py_Initialize() is run.
Yikes! That kills using sitecustomize.py. Your Spam_path still works because it is imported later, but requires an import in each Python main script just as you said. Even worse, it means that exceptions.py and site.py can not be found at all except using the normal PYTHONPATH, and putting their path in Spam_path will *not* work.
Oh dear, I think I heard no instead of yes. Are you saying that if someone else installs a Python app on my customer's machine after I do, and sets a registry entry which sayes to use c:/other/path/to/site.py for site.py (as he may very well want to do), then if my Python program depends on getting my copy of site.py from my directory, it will then use the other copy instead and may very well fail?
Point on the curve: Windows apps generally start from an icon which contains their path and current working directory, and these are generally different. So a Windows app in general will *never* have had a getcwd() equal to the path of either the binary interpreter or the Python main script.
Well, *no* right? This fails unless the bin directories are in fact on PYTHONPATH. The only way to get exceptions.py is by using sys.path as it exists within Py_Initialize(). So there is no hacked sys.path[0] equal to the script dir. And since the path hacks in site.py haven't happened yet either, we have an incomplete sys.path at that point.
Ok, point taken.
I don't know about loading by COM, but if it is a file, its absolute path is reliably known in sys, the code is identical to that currently used for sys.executable (on Windows), and I have a patch if you want. JimA's conjecture: It is currently impossible to ship a Python app which can not be damaged by the installation of a second Python app without using a hacked custom binary. Jim Ahlstrom

[me]
[JimA]
Not too bad (who cares about one more line of boilerplate...).
Why would you want your own exceptions.py and site.py?
Again - why would anyone register their own site.py?
You're lucky. It turns out that on Windows, under those circumstances at least, sys.path[0] is the absolute pathname of the directory. You only see '' if sys.argv[0] doesn't have any pathname information; that's only possible if the script *does* live in the current directory.
Sorry, I've lost track of what we were after here. Indeed the scripts' directory (which I presume you meant by the bin directory) indeed doesn't occur in sys.path until after Py_Initialize() has run.
I presume using GetModuleFileName()? Please send me the patch!
Sounds right. All tricks to make the app unique require using a different registry key, which requires a change to the DLL. However, you can do this without recompiling! The version string is used is embedded in a resource, so you can patch it using some kind of resource editor. Mark Hammond planned it this way! --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
I don't. I never change Python library files. I am worried that they won't be found because I don't trust PYTHONPATH.
I wouldn't, I am worried that someone else will break my installation. Remember that site.py was invented as a site-specific module, although that function moved to sitecustomize.py.
I presume using GetModuleFileName()? Please send me the patch!
Yes, and OK.
I don't understand this. Is there documentation? Jim Ahlstrom

[me]
Hmm... PYTHONPATH gets inserted in front of the default sys.path. (Many moons ago that was different. But it has been like this for a loooooong time.) So are you worried that someone put a *different* exceptions.py or site.py on their path?
Hm, I dug out the oldest site.py I have (used in Python 1.4), and it doesn't encourage editing it at all -- it tells you to use sitecustomize.py. I guess they could break your installation anyway, but only by messing with the general Python installation.
The usual :-) Python/import.c shows that import calls PyWin_FindRegisteredModule() to find a registered module before looking in sys.path (but after checking for builtin and frozen modules). PC/import_nt.c shows that PyWin_FindRegisteredModule() uses a registry key of the form "Software\Python\PythonCore\<PyWin_DLLVersionString>\Modules\<modulename><debugstring>" where <modulename> is the module name, <debugstring> is empty or "\Debug" depending on whether we are compiled with _DEBUG define. The resource value points to a file (either .py, .pyc/.pyo, .pyd or .dll; in fact any of the prefixes returned by imp.get_suffixes()). PC/dl_nt.c shows that PyWin_DLLVersionString is set to string 1000 loaded from the string resource table. PC/python_nt.rc shows that there's a stringtable with item 1000 being the MS_DLL_ID string, set to "1.5" in that file. Note that this value (PyWin_DLLVersionString) is also to Python code as sys.winver. I hope that Mark Hammond can point you to a tool that you can use to edit a string resource in an executable or DLL. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
When you jam every file into a module archive, you still have to leave these two "outside" so that Python can find them when starting up. The problem then breaks down into two parts: 1) locating them 2) ensuring they're the right versions In my "small" distro, I relied on the current-dir always being in the path, so I could always find these. The two files were distributed as part of distro. Anyhow: JimA is saying that he doesn't trust PYTHONPATH. Not so much bad files/versions, but that he won't be able to find them because PYTHONPATH has been monkeyed in some way.
If the file exists, then somebody can mess with it. JimA is trying to create a bulletproof app here. To do this, you can distribute a python.exe, exceptions.py, site.py, and an archive of your other modules. site.py is scrapped in favor of installing an Importer to access the archive (this implies you also distribute imputil.py). These five files are the exact five in my "small" distro. It's pretty cool... no need for registry changes and a very small "file count" footprint. Gordon took this basis and added a bunch of features for bundling an application in there. JimA has also been mixing in some frozen modules (I forget exactly why/what). [oh, my small distro doesn't ship a python15.dll, although it easily could]
The win32api module has resource manipulation functions such as BeginUpdateResource, UpdateResource, and EndUpdateResource. Write a Python script to modify your version string :-) A demo of resource munging can be seen in <win32 source>/win32/scripts/VersionStamp/verstamp.py. Cheers, -g -- Greg Stein, http://www.lyra.org/

[Greg, replying to Guido's confusion on Jim's interest in site.py and exceptions.py]
In the soon-to-be-published beta version of my installer, I've got that down to exceptions.py. Background: my (Win32) installer has, as a stated goal, the ability to create quasi-frozen Python apps which won't interfere with (or be influenced by) existing Python installations (if any). And it doesn't require the programmer/user to have a compiler. Thanks to patches given me by Thomas Heller, my python.exe replacement is now a (minimal) embedding app, and I do the same things that Greg does in site.py directly from C code, (and turn off the SiteFlag, too). I believe that if I freeze in exceptions.py and tweak the resource in python15.dll (which is just the stock python15.dll), I can have a completely safe executable. I think the same techniques can be applied on *nix, (although I'm pretty sure programmer/users won't be able to get away without a compiler). This gives me a strong interest in import hooks for two distinct reasons: - I rely completely on Greg's imputil to make this work. - I rely on freeze's modulefinder to help build these things. Bizarre import hooks in a normal Python installation will fool modulefinder. Take a look at what Pmw does (fortunately, Pmw comes with it's own packager). So I'd like to see import hooks follow some sort of pattern that can be followed by a tool like modulefinder. I also want the hooks. Right now I use imputil with archives, but wouldn't it be cool if you could add another imputil importer that checks for more recent versions at some home site on the web and automatically updates the installation? Summary: I'm very interested in seeing import and import hooks get rationalized, and I think Greg's stuff goes a long, long way towards that goal. - Gordon

Gordon> In the soon-to-be-published beta version of my installer, I've Gordon> got that down to exceptions.py. Why not just run exceptions.py through Python2C, visually and experimentally verify that it works, then ship an exceptions.c as an optional module? People wanting to ship self-contained packages could then toss exceptions.py and build the C version of the exceptions module. Greg, is there anything in exceptions.py Python2C couldn't handle? Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented...

Skip Montanaro wrote:
Nah, shouldn't have any problem at all. P2C will even create true class objects and expose them in the interface. I think a person might want to consider hand-tuning the output, though :-) Cheers, -g -- Greg Stein, http://www.lyra.org/

Greg Stein wrote:
Optional module... Good idea. But it is easier to use freeze within the optional module. So I wrote a "boot" built in module, where bootmodule.c is created by the Python program bootmake.py. All this lives in ./Modules. Boot incorporates exceptions.pyc etc. as frozen modules without breaking the current freeze feature. I changed pythonrun.c to load "boot" after sys but before any other imports, but only if it exists as a built in module, otherwise no error. Boot has methods to turn it on and off, and to print its contents. Please take a look at ftp://ftp.interet.com/pub/bootmodule.html I think this is a good solution for how to build in imputil.py. Jim Ahlstrom

Jim> What is Python2C. Is it the same as freeze? Nope. Python in, compilable C out: http://www.mudlib.org/~rassilon/p2c/ Courtesy of Greg Stein and Bill Tutt. Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented...

Alternatively (and probably easier) it (and site.py) could be stored as frozen modules. All it takes is some edits to Python/frozen.c. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Alternatively (and probably easier) it (and site.py) could be stored as frozen modules. All it takes is some edits to Python/frozen.c.
An excellent design, very simple. A large list of critical Python-language modules including a new importer, imputils.py and a new main could be reliably linked to their interpreter. It would open the door writing a larger part of Python in Python. But editing Python/frozen.c will kill the current freeze feature since a user currently replaces this pointer with their own. We can have a second _PyImport_InternalFrozenModules[] array which is searched FIRST, so that the existing frozen modules feature is retained. Current logic only allows one frozen module array. This is easy. A little harder is turning it off, which may be important to developers. They will probably want to use site.py etc. in PYTHONPATH directories. I would hate to use another command line option. Maybe another method in the imp module, say imp.EnablePyInternalLib(x) to turn on/off the internal frozen modules. A method to print the names in the list wouldn't hurt either. At one time I had code for multiple frozen modules which I may be able to find. Jim Ahlstrom

Guido van Rossum wrote:
Yes, and/or (2) added a sitecustomize.py with their special import hook as has been proposed here over and over, or (3) PYTHONPATH is screwed up and doesn't find anything. Perhaps this is a prejudice of mine. I just look at "print sys.path" and marvel at what I don't understand. At least I can see it is not simple. I hate relying on thing that are not simple. And I hate custom import hooks. Unless they are mine of course ;-)
Just adding sitecustomize.py would do it. And this is encouraged. I think Gordon put his finger on the issue. Either try to co-exist with other installed Python software and take the risk that everyone is playing by the rules, or build your own black-box self-contained Python world and duplicate storage.
The usual :-) [Documentation goes here...]
Thanks. This is very useful. But it doesn't help, perhaps as a result of more of my prejudices. This registry entry is meant to be used by a by-the-rules shared Python installation, so I must not change it. And I hate changing any registry entries at all. My commercial software keeps all its settings in a regular .ini file in its install directory, and makes only minimal and required registry entries. IMHO the Windows registry is a software catastrophe which ranks right up there with JCL (anyone else here old enough to remember that?). Anyone who doesn't agree should go with me to our money center banking clients, and sit there while they grill you on every registry entry and why it is required. Money center banking clients do not like their registry messed with. I do however see your point that I could change the version string to something non-standard and use the registry to control imports. I will think about this further. Maybe it would work. My current "solution" is to use freeze to create a black-box install, and worry about second Python installations and wasted storage when it happens. I was hoping that this thread whould result in a consensis of what to do, but it has not. So now I am hoping that Python library (jar) files will turn out to be a practical solution, so I am pestering Greg and Gordon. We'll see. Jim Ahlstrom

James C. Ahlstrom wrote: [Guido explains Windows registry usage]
Without trying it, I doubt you have to. It looks like you could set the resource to something that won't be found in the registry, and then just use normal Python mechanisms.
Ah. In a rush to get it down to ops, tripping at the head of the stairs and sending the card deck flying... Those were the days. - Gordon

No - the point is that the "1.5" key is "reserved" by a standard install. Changing the string value actually allows you to have your own subtree, and you can assume you own that. For commercial apps I help with, re stick a "AppName" in this string (doesnt have to be a number) then simply use "HKLM\Software\Python\AppName"
And I hate changing any registry entries at all.
Well, you should learn to get over it! That is what it is designed for. sure, we dont want the users to muck with the registry, but if you dont like your install scripts working with the registry, I definately feel you should get over this prejudice of yours.
The registry is a huge bucket. Changing your own, custom kep under either HKLM\Software\ or HKCU\Software is perfectly reasonable. Many apps _require_ you to hit the registry - services, event log, performance monitor, etc. The fact the registry _also_ contains critical information is a problem, and sure, we dont want to mess with that.
well, IMO this is also the correct thing to do. any install that has
100 files is fragile. So I do both - freeze the app, _and_ a custom "sys.winver".
I was hoping that this thread whould result in a consensis of what to do, but it has not.
There is a consesus for people with the same problem. Different problems have different optimal solutions. Mark.

Perhaps you have heard about the East coast US hurricane. It really hammered us here in New Jersey. I had trouble getting home last night due to high water. When I went to our usual Japanese lunch restaurant it had broken windows and was full of tree branches. Wow... Anyway, Mark Hammond wrote:
OK, now that I know the rules I will think about doing that.
And I hate changing any registry entries at all.
Well, you should learn to get over it!
I will try. Of course, even if the registry helps ship Python apps on Windows, it is no help on Unix, and I care about that almost as much.
Yes, but this requires a compiler.
The problem is that there is no reliable way to ship bullet-proof Python apps without recompiling and rebuilding Python. Each of us has his own pet hack to solve our own problem. Now there is talk of custom import hooks, and this is likely to result in each package requiring its own import hook! Aren't packages supposed to be software IC's? I hate to be a nag, but I will keep pushing for a single solution. Python is totally cross platform, and with all that machinery, there must be portable way to do that. I think this is good for Python. Please don't think I am trying to solve my own selfish problems. I have a compiler, I am happy using freeze, and I don't have any problems. Its just that Python would "sell" better and be more popular if a developer could read the documentation "How to ship and install your Python app in five minutes and make millions". This documentation currently reads "You can't". Jim Ahlstrom

Vague documentation at http://www.python.org/windows/python/registry.html Doesnt explicitely say much about this particular issue, and if I remember I will update it. VC will allow you to edit these resources without rebuilding - simply open the DLL, but select "Resources" as the file type. I havent investigated other resource editors as I have MSVC :-) Greg pointed you to the win32api functions - they do work, as Greg provided code that uses them (the code he pointed at was done by him) - although I never actually worked out how to make them work ( I spent about an hour trying to get code to re-stamp the Python15.dll with a new sys.winver, and gave up. Would be cool to get going tho... Mark.

From: "Gordon McMillan" <gmcm@hypernet.com>
Hm. We had an email glitch. Apparently this message got lost: Subject: Re: Path hacking From: Guido van Rossum <guido@CNRI.Reston.VA.US> To: python-dev@python.org Date: Tue, 14 Sep 1999 15:57:51 -0400 I just had a long discussion with Barry and Fred, in response to his registry proposal. We quickly decided that a Python registry is overkill for the given problem. We also quickly came up with a nice variant of Mailman's approach which will work well in a variety of cases. --> The context: You have a large complicated application that contains many modules spread over many packages, and which has many "top-level" scripts that are invoked by the user (or via CGI, for example). All the code is properly packagized, with sufficiently globally unique package names being used all over the place. --> The problem: How to get the root directory of your application (where all your packages live) on sys.path. --> The rules: Using $PYTHONPATH is right out. You can't install new files in the core Python installation directory (nor modify existing ones), so using .pth files is also out. You don't want to have to edit each of the top-level scripts of your application. You want a cross-platform solution, in particular it should be amenable to Windows. --> The assumptions: You can use a reasonably intelligent installer. All your top-level scripts are installed in a single directory (or perhaps in a small number of separate bin directories, e.g. bin and cgi-bin). --> The solution: Suppose your application (as a whole, not the individual top-level script) is called Spam -- this may well also be the name of your top-level package. Then start each top-level script with the single line import Spam_path before importing anything else. Your installer, once it knows the absolute pathname of your application's root directory, crafts a file Spam_path.py which contains code that inserts the right absolute pathname into sys.path. Your installer then installs a copy of this file (or a symbolic link to it) *in each bin directory where it installs top-level Python scripts*. Because the script's directory is first on the default path, the Spam scripts will pick up Spam_path without any help from $PYTHONPATH. --> Notes: If you are Spam's developer, you probably want to be able to use its top-level scripts without having to install them. All you need to do is create a file Spam_path.py pointing to the top of your development tree, and set $PYTHONPATH to point to the directory that contains it. (Perhaps you already have $PYTHONPATH pointing to a personal directory of Python modules you like to have accessible -- then you can just drop Spam_path.py there, or link to it from there.) Note that adding a personal directory of Python goodies is about the only use of $PYTHONPATH that I approve of -- this way, you can set $PYTHONPATH in your .profile and never have to change it. I know this doesn't resolve the relative import thread (how's that going by the way? :-) but Barry & Fred & I agree that this is the best solution to the problem stated in Barry's message to which I am following up here. --Guido van Rossum (home page: http://www.python.org/~guido/)

Gordon McMillan wrote:
My mail system is flakey, so I have been reading this list directly on python.org. I didn't get it by list either, so I assumed my mailer ate it. See: http://www.python.org/pipermail/python-dev/1999-September/000851.html Jim Ahlstrom

"Gordo" == Gordon McMillan <gmcm@hypernet.com> writes:
Gordo> Jim Ahlstrom wtoe: >> "Guido van Rossum" wrote: >> --> The solution: Gordo> Did the dev-list miss something? The last I see is Barry's Gordo> post. I have a suspicion that python.org lost some email yesterday. We had a period of time where mail simply stopped getting delivered (thank you Solaris patch manager) and it took me a little while to realize that things weren't working correctly. Since there's nothing unexpected in the mail queue now, all I can say is that if you didn't get it by now, you ain't gonna. However, everything seemed to make it into the archives, so Guido's message is available at: http://www.python.org/pipermail/python-dev/1999-September/000880.html -Barry

"Gordo" == Gordon McMillan <gmcm@hypernet.com> writes:
Gordo> Hmm, I'm suspicious of the fact that no message from Barry Gordo> Warsaw ever gets "lost". Gordo> Stalin got started by being in charge of the Kremlin's Gordo> telephone system, you know... Well, I managed to get rid of Ken so my world domination plan is right on schedule! guido-may-be-benevolent-but-you-can-bet-i-won't-be-ly y'rs, -Barry

[Guido]
import Spam_path
before importing anything else.
[JimA]
But the intention here is for the customization to be application specific (hence the Spam in the name). sitecustomize doesn't know whethere I need the Mailman or the Knowbot root added to my path. Or do you mean to imply that we can do this with zero text added to the script, by simply dropping an appropriate sitecustomize.py in the script dir? Unfortunately this does currently *not* work, because sys.path[0] is added after Py_Initialize() is run.
Hm, guessing based on the script directory might work, but seems less reliable than hardcoding it through the installer. But you can use this if it works for your application.
Yes. (The registry can add module-specific paths, which will be searched before sys.path is even looked at, but this is only for specific modules. It cannot insert a general directory that is searched.) The only way this can fail is if an embedding app fails to call PySys_SetArgv().
I have always resisted forcing path items to be absolute, although I'm not sure that my reasons are valid any more (it has to do with the fact that getcwd() may fail and the fact that portable path concatenation is a pain). In any case, that's a separate issue -- I agree that if sys.path[0] is '' (as it often is) it's better for site.py or sitecustomize.py or Spam_path.py (or whoever) to absolutize it (and everything else on the path) so that it will still work if the app does a chdir().
Yes.
The above doesn't work when you start the Python command interpreter (no main). I know, its a minor point.
You could add the "import Spam_path" to your $PYTHONSTARTUP file.
I think no simple solution that *I* can come up with will satisfy JimF's and Marc's desire for obscurity :-)
Yes, this is the plan for Python 2.0, and some of it may be implemented in Python 1.6.
On Unix, this is a bin directory and it is strongly discouraged to put non-program files there. Python already does something similar -- it looks around in sys.executable's ancestors for a specific landmark, currently lib/python<version>/string.py. Arguably, it should search for execeptions.py instead.
Is the full DLL path available at any point? This would certainly be a good starting point -- especially when the DLL is loaded implicitly as the result of some COM operation.
I guess this is all up to the redesign of the import mechanism (something like Greg Stein's imputil.py for sure). --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Never mind, I'll use an imputil.py based approach to get relative imports to work in my packages. That is when I get imputil.py to work... it doesn't seem to be quite there yet (or I'm using an old version). BTW, I'm 100% behind you guys if you choose to reimplement Python's import mechanism in Python using a similar approach as the one Greg implemented in imputil. Should make everybody happy: those who want obscure syntactic add-ons and others with a taste for zlib'ed packaged byte code, plus those VMS freaks ;-) Perhaps we should start a new thread on that topic... Still needed are: · Python level APIs for the platform specific magic on Win32 and Macs (OS/2, BeOS ?), e.g. access to the Windows registry and the Mac forks · Patches to make the DirectoryImporter 100% backward compatible Greg's imputil.py can be found at: http://www.lyra.org/greg/small/ The trick would then be to install an application specific importer in the setup module Spam_path or MyAppSetup which then takes care of all the rest... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 107 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Guido van Rossum wrote:
Ah, you have multiple scripts in one directory and multiple Foo_path, Bar_path etc. I was thinking with my Windows head. A commercial Windows app generally has its own exclusive install directory, so I was thinking single directory so a single sitecustomize.py.
Yes, that is exactly what I was thinking.
Unfortunately this does currently *not* work, because sys.path[0] is added after Py_Initialize() is run.
Yikes! That kills using sitecustomize.py. Your Spam_path still works because it is imported later, but requires an import in each Python main script just as you said. Even worse, it means that exceptions.py and site.py can not be found at all except using the normal PYTHONPATH, and putting their path in Spam_path will *not* work.
Oh dear, I think I heard no instead of yes. Are you saying that if someone else installs a Python app on my customer's machine after I do, and sets a registry entry which sayes to use c:/other/path/to/site.py for site.py (as he may very well want to do), then if my Python program depends on getting my copy of site.py from my directory, it will then use the other copy instead and may very well fail?
Point on the curve: Windows apps generally start from an icon which contains their path and current working directory, and these are generally different. So a Windows app in general will *never* have had a getcwd() equal to the path of either the binary interpreter or the Python main script.
Well, *no* right? This fails unless the bin directories are in fact on PYTHONPATH. The only way to get exceptions.py is by using sys.path as it exists within Py_Initialize(). So there is no hacked sys.path[0] equal to the script dir. And since the path hacks in site.py haven't happened yet either, we have an incomplete sys.path at that point.
Ok, point taken.
I don't know about loading by COM, but if it is a file, its absolute path is reliably known in sys, the code is identical to that currently used for sys.executable (on Windows), and I have a patch if you want. JimA's conjecture: It is currently impossible to ship a Python app which can not be damaged by the installation of a second Python app without using a hacked custom binary. Jim Ahlstrom

[me]
[JimA]
Not too bad (who cares about one more line of boilerplate...).
Why would you want your own exceptions.py and site.py?
Again - why would anyone register their own site.py?
You're lucky. It turns out that on Windows, under those circumstances at least, sys.path[0] is the absolute pathname of the directory. You only see '' if sys.argv[0] doesn't have any pathname information; that's only possible if the script *does* live in the current directory.
Sorry, I've lost track of what we were after here. Indeed the scripts' directory (which I presume you meant by the bin directory) indeed doesn't occur in sys.path until after Py_Initialize() has run.
I presume using GetModuleFileName()? Please send me the patch!
Sounds right. All tricks to make the app unique require using a different registry key, which requires a change to the DLL. However, you can do this without recompiling! The version string is used is embedded in a resource, so you can patch it using some kind of resource editor. Mark Hammond planned it this way! --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
I don't. I never change Python library files. I am worried that they won't be found because I don't trust PYTHONPATH.
I wouldn't, I am worried that someone else will break my installation. Remember that site.py was invented as a site-specific module, although that function moved to sitecustomize.py.
I presume using GetModuleFileName()? Please send me the patch!
Yes, and OK.
I don't understand this. Is there documentation? Jim Ahlstrom

[me]
Hmm... PYTHONPATH gets inserted in front of the default sys.path. (Many moons ago that was different. But it has been like this for a loooooong time.) So are you worried that someone put a *different* exceptions.py or site.py on their path?
Hm, I dug out the oldest site.py I have (used in Python 1.4), and it doesn't encourage editing it at all -- it tells you to use sitecustomize.py. I guess they could break your installation anyway, but only by messing with the general Python installation.
The usual :-) Python/import.c shows that import calls PyWin_FindRegisteredModule() to find a registered module before looking in sys.path (but after checking for builtin and frozen modules). PC/import_nt.c shows that PyWin_FindRegisteredModule() uses a registry key of the form "Software\Python\PythonCore\<PyWin_DLLVersionString>\Modules\<modulename><debugstring>" where <modulename> is the module name, <debugstring> is empty or "\Debug" depending on whether we are compiled with _DEBUG define. The resource value points to a file (either .py, .pyc/.pyo, .pyd or .dll; in fact any of the prefixes returned by imp.get_suffixes()). PC/dl_nt.c shows that PyWin_DLLVersionString is set to string 1000 loaded from the string resource table. PC/python_nt.rc shows that there's a stringtable with item 1000 being the MS_DLL_ID string, set to "1.5" in that file. Note that this value (PyWin_DLLVersionString) is also to Python code as sys.winver. I hope that Mark Hammond can point you to a tool that you can use to edit a string resource in an executable or DLL. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
When you jam every file into a module archive, you still have to leave these two "outside" so that Python can find them when starting up. The problem then breaks down into two parts: 1) locating them 2) ensuring they're the right versions In my "small" distro, I relied on the current-dir always being in the path, so I could always find these. The two files were distributed as part of distro. Anyhow: JimA is saying that he doesn't trust PYTHONPATH. Not so much bad files/versions, but that he won't be able to find them because PYTHONPATH has been monkeyed in some way.
If the file exists, then somebody can mess with it. JimA is trying to create a bulletproof app here. To do this, you can distribute a python.exe, exceptions.py, site.py, and an archive of your other modules. site.py is scrapped in favor of installing an Importer to access the archive (this implies you also distribute imputil.py). These five files are the exact five in my "small" distro. It's pretty cool... no need for registry changes and a very small "file count" footprint. Gordon took this basis and added a bunch of features for bundling an application in there. JimA has also been mixing in some frozen modules (I forget exactly why/what). [oh, my small distro doesn't ship a python15.dll, although it easily could]
The win32api module has resource manipulation functions such as BeginUpdateResource, UpdateResource, and EndUpdateResource. Write a Python script to modify your version string :-) A demo of resource munging can be seen in <win32 source>/win32/scripts/VersionStamp/verstamp.py. Cheers, -g -- Greg Stein, http://www.lyra.org/

[Greg, replying to Guido's confusion on Jim's interest in site.py and exceptions.py]
In the soon-to-be-published beta version of my installer, I've got that down to exceptions.py. Background: my (Win32) installer has, as a stated goal, the ability to create quasi-frozen Python apps which won't interfere with (or be influenced by) existing Python installations (if any). And it doesn't require the programmer/user to have a compiler. Thanks to patches given me by Thomas Heller, my python.exe replacement is now a (minimal) embedding app, and I do the same things that Greg does in site.py directly from C code, (and turn off the SiteFlag, too). I believe that if I freeze in exceptions.py and tweak the resource in python15.dll (which is just the stock python15.dll), I can have a completely safe executable. I think the same techniques can be applied on *nix, (although I'm pretty sure programmer/users won't be able to get away without a compiler). This gives me a strong interest in import hooks for two distinct reasons: - I rely completely on Greg's imputil to make this work. - I rely on freeze's modulefinder to help build these things. Bizarre import hooks in a normal Python installation will fool modulefinder. Take a look at what Pmw does (fortunately, Pmw comes with it's own packager). So I'd like to see import hooks follow some sort of pattern that can be followed by a tool like modulefinder. I also want the hooks. Right now I use imputil with archives, but wouldn't it be cool if you could add another imputil importer that checks for more recent versions at some home site on the web and automatically updates the installation? Summary: I'm very interested in seeing import and import hooks get rationalized, and I think Greg's stuff goes a long, long way towards that goal. - Gordon

Gordon> In the soon-to-be-published beta version of my installer, I've Gordon> got that down to exceptions.py. Why not just run exceptions.py through Python2C, visually and experimentally verify that it works, then ship an exceptions.c as an optional module? People wanting to ship self-contained packages could then toss exceptions.py and build the C version of the exceptions module. Greg, is there anything in exceptions.py Python2C couldn't handle? Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented...

Skip Montanaro wrote:
Nah, shouldn't have any problem at all. P2C will even create true class objects and expose them in the interface. I think a person might want to consider hand-tuning the output, though :-) Cheers, -g -- Greg Stein, http://www.lyra.org/

Greg Stein wrote:
Optional module... Good idea. But it is easier to use freeze within the optional module. So I wrote a "boot" built in module, where bootmodule.c is created by the Python program bootmake.py. All this lives in ./Modules. Boot incorporates exceptions.pyc etc. as frozen modules without breaking the current freeze feature. I changed pythonrun.c to load "boot" after sys but before any other imports, but only if it exists as a built in module, otherwise no error. Boot has methods to turn it on and off, and to print its contents. Please take a look at ftp://ftp.interet.com/pub/bootmodule.html I think this is a good solution for how to build in imputil.py. Jim Ahlstrom

Jim> What is Python2C. Is it the same as freeze? Nope. Python in, compilable C out: http://www.mudlib.org/~rassilon/p2c/ Courtesy of Greg Stein and Bill Tutt. Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/~skip/ 847-971-7098 | Python: Programming the way Guido indented...

Alternatively (and probably easier) it (and site.py) could be stored as frozen modules. All it takes is some edits to Python/frozen.c. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Alternatively (and probably easier) it (and site.py) could be stored as frozen modules. All it takes is some edits to Python/frozen.c.
An excellent design, very simple. A large list of critical Python-language modules including a new importer, imputils.py and a new main could be reliably linked to their interpreter. It would open the door writing a larger part of Python in Python. But editing Python/frozen.c will kill the current freeze feature since a user currently replaces this pointer with their own. We can have a second _PyImport_InternalFrozenModules[] array which is searched FIRST, so that the existing frozen modules feature is retained. Current logic only allows one frozen module array. This is easy. A little harder is turning it off, which may be important to developers. They will probably want to use site.py etc. in PYTHONPATH directories. I would hate to use another command line option. Maybe another method in the imp module, say imp.EnablePyInternalLib(x) to turn on/off the internal frozen modules. A method to print the names in the list wouldn't hurt either. At one time I had code for multiple frozen modules which I may be able to find. Jim Ahlstrom

Guido van Rossum wrote:
Yes, and/or (2) added a sitecustomize.py with their special import hook as has been proposed here over and over, or (3) PYTHONPATH is screwed up and doesn't find anything. Perhaps this is a prejudice of mine. I just look at "print sys.path" and marvel at what I don't understand. At least I can see it is not simple. I hate relying on thing that are not simple. And I hate custom import hooks. Unless they are mine of course ;-)
Just adding sitecustomize.py would do it. And this is encouraged. I think Gordon put his finger on the issue. Either try to co-exist with other installed Python software and take the risk that everyone is playing by the rules, or build your own black-box self-contained Python world and duplicate storage.
The usual :-) [Documentation goes here...]
Thanks. This is very useful. But it doesn't help, perhaps as a result of more of my prejudices. This registry entry is meant to be used by a by-the-rules shared Python installation, so I must not change it. And I hate changing any registry entries at all. My commercial software keeps all its settings in a regular .ini file in its install directory, and makes only minimal and required registry entries. IMHO the Windows registry is a software catastrophe which ranks right up there with JCL (anyone else here old enough to remember that?). Anyone who doesn't agree should go with me to our money center banking clients, and sit there while they grill you on every registry entry and why it is required. Money center banking clients do not like their registry messed with. I do however see your point that I could change the version string to something non-standard and use the registry to control imports. I will think about this further. Maybe it would work. My current "solution" is to use freeze to create a black-box install, and worry about second Python installations and wasted storage when it happens. I was hoping that this thread whould result in a consensis of what to do, but it has not. So now I am hoping that Python library (jar) files will turn out to be a practical solution, so I am pestering Greg and Gordon. We'll see. Jim Ahlstrom

James C. Ahlstrom wrote: [Guido explains Windows registry usage]
Without trying it, I doubt you have to. It looks like you could set the resource to something that won't be found in the registry, and then just use normal Python mechanisms.
Ah. In a rush to get it down to ops, tripping at the head of the stairs and sending the card deck flying... Those were the days. - Gordon

No - the point is that the "1.5" key is "reserved" by a standard install. Changing the string value actually allows you to have your own subtree, and you can assume you own that. For commercial apps I help with, re stick a "AppName" in this string (doesnt have to be a number) then simply use "HKLM\Software\Python\AppName"
And I hate changing any registry entries at all.
Well, you should learn to get over it! That is what it is designed for. sure, we dont want the users to muck with the registry, but if you dont like your install scripts working with the registry, I definately feel you should get over this prejudice of yours.
The registry is a huge bucket. Changing your own, custom kep under either HKLM\Software\ or HKCU\Software is perfectly reasonable. Many apps _require_ you to hit the registry - services, event log, performance monitor, etc. The fact the registry _also_ contains critical information is a problem, and sure, we dont want to mess with that.
well, IMO this is also the correct thing to do. any install that has
100 files is fragile. So I do both - freeze the app, _and_ a custom "sys.winver".
I was hoping that this thread whould result in a consensis of what to do, but it has not.
There is a consesus for people with the same problem. Different problems have different optimal solutions. Mark.

Perhaps you have heard about the East coast US hurricane. It really hammered us here in New Jersey. I had trouble getting home last night due to high water. When I went to our usual Japanese lunch restaurant it had broken windows and was full of tree branches. Wow... Anyway, Mark Hammond wrote:
OK, now that I know the rules I will think about doing that.
And I hate changing any registry entries at all.
Well, you should learn to get over it!
I will try. Of course, even if the registry helps ship Python apps on Windows, it is no help on Unix, and I care about that almost as much.
Yes, but this requires a compiler.
The problem is that there is no reliable way to ship bullet-proof Python apps without recompiling and rebuilding Python. Each of us has his own pet hack to solve our own problem. Now there is talk of custom import hooks, and this is likely to result in each package requiring its own import hook! Aren't packages supposed to be software IC's? I hate to be a nag, but I will keep pushing for a single solution. Python is totally cross platform, and with all that machinery, there must be portable way to do that. I think this is good for Python. Please don't think I am trying to solve my own selfish problems. I have a compiler, I am happy using freeze, and I don't have any problems. Its just that Python would "sell" better and be more popular if a developer could read the documentation "How to ship and install your Python app in five minutes and make millions". This documentation currently reads "You can't". Jim Ahlstrom

Vague documentation at http://www.python.org/windows/python/registry.html Doesnt explicitely say much about this particular issue, and if I remember I will update it. VC will allow you to edit these resources without rebuilding - simply open the DLL, but select "Resources" as the file type. I havent investigated other resource editors as I have MSVC :-) Greg pointed you to the win32api functions - they do work, as Greg provided code that uses them (the code he pointed at was done by him) - although I never actually worked out how to make them work ( I spent about an hour trying to get code to re-stamp the Python15.dll with a new sys.winver, and gave up. Would be cool to get going tho... Mark.

From: "Gordon McMillan" <gmcm@hypernet.com>
Hm. We had an email glitch. Apparently this message got lost: Subject: Re: Path hacking From: Guido van Rossum <guido@CNRI.Reston.VA.US> To: python-dev@python.org Date: Tue, 14 Sep 1999 15:57:51 -0400 I just had a long discussion with Barry and Fred, in response to his registry proposal. We quickly decided that a Python registry is overkill for the given problem. We also quickly came up with a nice variant of Mailman's approach which will work well in a variety of cases. --> The context: You have a large complicated application that contains many modules spread over many packages, and which has many "top-level" scripts that are invoked by the user (or via CGI, for example). All the code is properly packagized, with sufficiently globally unique package names being used all over the place. --> The problem: How to get the root directory of your application (where all your packages live) on sys.path. --> The rules: Using $PYTHONPATH is right out. You can't install new files in the core Python installation directory (nor modify existing ones), so using .pth files is also out. You don't want to have to edit each of the top-level scripts of your application. You want a cross-platform solution, in particular it should be amenable to Windows. --> The assumptions: You can use a reasonably intelligent installer. All your top-level scripts are installed in a single directory (or perhaps in a small number of separate bin directories, e.g. bin and cgi-bin). --> The solution: Suppose your application (as a whole, not the individual top-level script) is called Spam -- this may well also be the name of your top-level package. Then start each top-level script with the single line import Spam_path before importing anything else. Your installer, once it knows the absolute pathname of your application's root directory, crafts a file Spam_path.py which contains code that inserts the right absolute pathname into sys.path. Your installer then installs a copy of this file (or a symbolic link to it) *in each bin directory where it installs top-level Python scripts*. Because the script's directory is first on the default path, the Spam scripts will pick up Spam_path without any help from $PYTHONPATH. --> Notes: If you are Spam's developer, you probably want to be able to use its top-level scripts without having to install them. All you need to do is create a file Spam_path.py pointing to the top of your development tree, and set $PYTHONPATH to point to the directory that contains it. (Perhaps you already have $PYTHONPATH pointing to a personal directory of Python modules you like to have accessible -- then you can just drop Spam_path.py there, or link to it from there.) Note that adding a personal directory of Python goodies is about the only use of $PYTHONPATH that I approve of -- this way, you can set $PYTHONPATH in your .profile and never have to change it. I know this doesn't resolve the relative import thread (how's that going by the way? :-) but Barry & Fred & I agree that this is the best solution to the problem stated in Barry's message to which I am following up here. --Guido van Rossum (home page: http://www.python.org/~guido/)

Gordon McMillan wrote:
My mail system is flakey, so I have been reading this list directly on python.org. I didn't get it by list either, so I assumed my mailer ate it. See: http://www.python.org/pipermail/python-dev/1999-September/000851.html Jim Ahlstrom

"Gordo" == Gordon McMillan <gmcm@hypernet.com> writes:
Gordo> Jim Ahlstrom wtoe: >> "Guido van Rossum" wrote: >> --> The solution: Gordo> Did the dev-list miss something? The last I see is Barry's Gordo> post. I have a suspicion that python.org lost some email yesterday. We had a period of time where mail simply stopped getting delivered (thank you Solaris patch manager) and it took me a little while to realize that things weren't working correctly. Since there's nothing unexpected in the mail queue now, all I can say is that if you didn't get it by now, you ain't gonna. However, everything seemed to make it into the archives, so Guido's message is available at: http://www.python.org/pipermail/python-dev/1999-September/000880.html -Barry

"Gordo" == Gordon McMillan <gmcm@hypernet.com> writes:
Gordo> Hmm, I'm suspicious of the fact that no message from Barry Gordo> Warsaw ever gets "lost". Gordo> Stalin got started by being in charge of the Kremlin's Gordo> telephone system, you know... Well, I managed to get rid of Ken so my world domination plan is right on schedule! guido-may-be-benevolent-but-you-can-bet-i-won't-be-ly y'rs, -Barry
participants (9)
-
Barry A. Warsaw
-
Fredrik Lundh
-
Gordon McMillan
-
Greg Stein
-
Guido van Rossum
-
James C. Ahlstrom
-
M.-A. Lemburg
-
Mark Hammond
-
Skip Montanaro