DLL in the system directory on Windows.
The 1.6a1 installer on Windows copies Python16.dll into the Python directory, rather than the system32 directory like 1.5.x. We discussed too long ago on this list not why this was probably not going to work. I guess Guido decided to "suck it and see" - which is fine. But guess what - it doesnt work :-( I couldnt get past the installer! The win32all installer executes some Python code at the end of the install (to generate the .pyc files and install the COM objects). This Python code is executed directly to the installation .EXE, by loading and executing a "shim" DLL I wrote for the purpose. Problem is, try as I might, my shim DLL could not load Python16.dll. The shim DLL _was_ in the same directory as Python16.dll. The only way I could have solved it was to insist the WISE installation .EXE be run from the main Python directory - obviously not an option. And the problem is quite obviously going to exist with COM objects. The problem would appear to go away if the universe switched over the LoadLibraryEx() - but we dont have that control in most cases (eg, COM, WISE etc dictate this to us). So, my solution was to copy Python16.dll to the system directory during win32all installation. This results in duplicate copies of this DLL, so to my mind, it is preferable that Python itself go back to using the System32 directory. The problem this will lead to is that Python 1.6.0 and 1.6.1 will not be able to be installed concurrently. Putting entries on the PATH doesnt solve the underlying problem - you will only be able to have one Python 1.6 directory on your path, else you end up with the same coflicts for the DLL. I dont see any better answer than System32 :-( Thoughts? Mark.
On Tue, 4 Apr 2000, Mark Hammond wrote:
... The problem this will lead to is that Python 1.6.0 and 1.6.1 will not be able to be installed concurrently.
Same thing happened with Python 1.5, so we're no worse off. If we do want this behavior, then we need to add another version digit...
Putting entries on the PATH doesnt solve the underlying problem - you will only be able to have one Python 1.6 directory on your path, else you end up with the same coflicts for the DLL.
I dont see any better answer than System32 :-( Thoughts?
I don't have a better answer, as you and I explained on several occasions. Dunno why Guido decided to skip our recommendations, but hey... it happens :-). IMO, put the DLL back into System32. If somebody can *demonstrate* (not hypothesize) a mechanism that works, then it can be switched. The underlying issue is this: Python16.dll in the app directory works for Python as an executable. However, it completely disables any possibility for *embedding* Python. On Windows, embedding is practically required because of the COM stuff (sure... a person could avoid COM but...). Cheers, -g -- Greg Stein, http://www.lyra.org/
The problem this will lead to is that Python 1.6.0 and 1.6.1 will not be able to be installed concurrently.
Same thing happened with Python 1.5, so we're no worse off. If we do want this behavior, then we need to add another version digit...
Actually, I don't plan on releasing a 1.6.1. The next one will be 1.7. Of course, alpha and beta versions for 1.6 won't be able to live along, but I can live with that.
Putting entries on the PATH doesnt solve the underlying problem - you will only be able to have one Python 1.6 directory on your path, else you end up with the same coflicts for the DLL.
I dont see any better answer than System32 :-( Thoughts?
I don't have a better answer, as you and I explained on several occasions. Dunno why Guido decided to skip our recommendations, but hey... it happens :-).
Actually, I just wanted to get the discussion started. It worked. :-) I'm waiting for Tim Peters' response in this thread -- if I recall he was the one who said that python1x.dll should not go into the system directory. Note that I've made it easy to switch: the WISE script defines a separate variable DLLDEST which is currently set to MAINDIR, but which I could easily change to SYS32 to get the semantics you prefer. Hey, we could even give the user a choice here! <0.4 wink>
IMO, put the DLL back into System32. If somebody can *demonstrate* (not hypothesize) a mechanism that works, then it can be switched.
The underlying issue is this: Python16.dll in the app directory works for Python as an executable. However, it completely disables any possibility for *embedding* Python. On Windows, embedding is practically required because of the COM stuff (sure... a person could avoid COM but...).
Yes, I know this. I'm just not happy with it, and I've definitely heard people complain that it is evil to install directories in the system directory. Seems there are different schools of thought... Another issue: MSVCRT.DLL and its friend MSVCIRT.DLL will also go into the system directory. I will now be distributing with the VC++ 6.0 servicepack 1 versions of these files. Won't this be a problem for installations that already have an older version? (Now that I think of it, this is another reason why I decided that at least the alpha release should install everything in MAINDIR -- to limit the damage. Any informed opinions?) David Ascher: if you're listening, could you forward this to someone at ActiveState who might understand the issues here? They should have the same problems with ActivePerl, right? Or don't they have COM support? (Personally, I think that it wouldn't be so bad if we made it so that if you install just Python, the DLLs go into MAINDIR -- if you install the COM support, it can move/copy them to the system directory. But you may find this inelegant...) --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, 3 Apr 2000, Guido van Rossum wrote:
... Actually, I just wanted to get the discussion started. It worked. :-)
hehe. True :-)
I'm waiting for Tim Peters' response in this thread -- if I recall he was the one who said that python1x.dll should not go into the system directory.
What's his physical address again? I have this nice little package to send him...
...
IMO, put the DLL back into System32. If somebody can *demonstrate* (not hypothesize) a mechanism that works, then it can be switched.
The underlying issue is this: Python16.dll in the app directory works for Python as an executable. However, it completely disables any possibility for *embedding* Python. On Windows, embedding is practically required because of the COM stuff (sure... a person could avoid COM but...).
Yes, I know this. I'm just not happy with it, and I've definitely heard people complain that it is evil to install directories in the system directory. Seems there are different schools of thought...
It is evil, but it is also unavoidable. The alternative is to munge the PATH variable, but that is a Higher Evil than just dropping DLLs into the system directory.
Another issue: MSVCRT.DLL and its friend MSVCIRT.DLL will also go into the system directory. I will now be distributing with the VC++ 6.0 servicepack 1 versions of these files. Won't this be a problem for installations that already have an older version?
Not at all. In fact, Microsoft explicitly recommends including those in the distribution and installing them over the top of *previous* versions. They should never be downgraded (i.e. always check their version stamp!), but they should *always* be upgraded. Microsoft takes phenomenal pains to ensure that OLD applications are compatible with NEW runtimes. It is certainly possible that you could have a new app was built against a new runtime, and breaks when used against an old runtime. But that is why you always upgrade :-) And note that I do mean phenomenal pains. It is one of their ship requirements that you can always drop in a new RT without breaking old apps. So: regardless of where you decide to put python16.dll, you really should be upgrading the RT DLLs.
David Ascher: if you're listening, could you forward this to someone at ActiveState who might understand the issues here? They should have the same problems with ActivePerl, right? Or don't they have COM support?
ActivePerl does COM, but I dunno much more than that.
(Personally, I think that it wouldn't be so bad if we made it so that if you install just Python, the DLLs go into MAINDIR -- if you install the COM support, it can move/copy them to the system directory. But you may find this inelegant...)
Eek. Now you're talking about one guy reaching into another installation and munging it around. Especially for a move (boy, would that throw off the uninstall!). If you copied, then it is possible to have *two* copies of the DLL loaded into a process. The primary key is the pathname. I've had two pythoncom DLLs loaded in a process, and boy does that suck! The bugs are quite interesting, to say the least :-) And a total bear to track down until you have seen the double-load several times and can start to recognize the effects. In other words, moving is bad for elegance/uninstall reasons, and copy is bad for (potential) runtime reasons. Cheers, -g -- Greg Stein, http://www.lyra.org/
What's his physical address again? I have this nice little package to send him...
Now, now, you don't want to sound like Ted Kazinsky, do you? :-)
It is evil, but it is also unavoidable. The alternative is to munge the PATH variable, but that is a Higher Evil than just dropping DLLs into the system directory.
Another issue: MSVCRT.DLL and its friend MSVCIRT.DLL will also go into the system directory. I will now be distributing with the VC++ 6.0 servicepack 1 versions of these files. Won't this be a problem for installations that already have an older version?
Not at all. In fact, Microsoft explicitly recommends including those in the distribution and installing them over the top of *previous* versions. They should never be downgraded (i.e. always check their version stamp!), but they should *always* be upgraded.
Microsoft takes phenomenal pains to ensure that OLD applications are compatible with NEW runtimes. It is certainly possible that you could have a new app was built against a new runtime, and breaks when used against an old runtime. But that is why you always upgrade :-)
And note that I do mean phenomenal pains. It is one of their ship requirements that you can always drop in a new RT without breaking old apps.
So: regardless of where you decide to put python16.dll, you really should be upgrading the RT DLLs.
OK. That means I need two separate variables: where to install the MS DLLs and where to install the Py DLLs.
David Ascher: if you're listening, could you forward this to someone at ActiveState who might understand the issues here? They should have the same problems with ActivePerl, right? Or don't they have COM support?
ActivePerl does COM, but I dunno much more than that.
I just downloaded and installed it. I've never seen an installer like this -- they definitely put a lot of effort in it. Annoying nit: they tell you to install "MS Windows Installer" first, and of course, being a MS tool, it requires a reboot. :-( Anyway, ActivePerl installs its DLLs (all 5) in c:\Perl\bin\. So there. It also didn't change PATH for me, even though the docs mention that it does -- maybe only on NT? (PATH on Win9x is still a mystery to me. Is it really true that in order to change PATH an installer has to edit autoexec.bat? Or is there a better way? Anything that claims to change PATH for me doesn't seem to do so. Could I have screwed something up?)
(Personally, I think that it wouldn't be so bad if we made it so that if you install just Python, the DLLs go into MAINDIR -- if you install the COM support, it can move/copy them to the system directory. But you may find this inelegant...)
Eek. Now you're talking about one guy reaching into another installation and munging it around. Especially for a move (boy, would that throw off the uninstall!). If you copied, then it is possible to have *two* copies of the DLL loaded into a process. The primary key is the pathname. I've had two pythoncom DLLs loaded in a process, and boy does that suck! The bugs are quite interesting, to say the least :-) And a total bear to track down until you have seen the double-load several times and can start to recognize the effects.
In other words, moving is bad for elegance/uninstall reasons, and copy is bad for (potential) runtime reasons.
OK, got it. But I'm still hoping that there's something we can do differently. Didn't someone tell me that at least on Windows 2000 installing app-specific files (as opposed to MS-provided files) in the system directory is a no-no? What's the alternative there? Is the same mechanism supported on NT or Win98? --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido]
... (PATH on Win9x is still a mystery to me.
You're not alone.
Is it really true that in order to change PATH an installer has to edit autoexec.bat?
AFAIK, yes. A specific PATH setting can be associated with a specific exe via the registry, though.
... Anything that claims to change PATH for me doesn't seem to do so.
Almost always the same here; suspect documentation rot.
Could I have screwed something up?
Yes, but I doubt it.
... Didn't someone tell me that at least on Windows 2000 installing app-specific files (as opposed to MS-provided files) in the system directory is a no-no?
MS was threatening to do this in (the then-named) NT5, but I believe they backed down. Don't have (the now-named) W2000 here to check on for sure, though.
I just downloaded and installed it. I've never seen an installer like this -- they definitely put a lot of effort in it.
hehe - guess who "encouraged" that :-)
Annoying nit: they tell you to install "MS Windows Installer" first
that should be a good clue :-)
and of course, being a MS tool, it requires a reboot. :-(
Actually, MSI is very cool. Now MSI is installed, most future MSI installs should proceed without reboot. In Win2k it is finally close to perfect. I dont think an installer has ever wanted to reboot my PC since Win2k.
Anyway, ActivePerl installs its DLLs (all 5) in c:\Perl\bin\. So there. It also didn't change PATH for me, even though the docs mention that it does -- maybe only on NT?
In another mail you asked David to look into how Active State handle their DLLs. Well, Trent Mick started the ball rolling... The answer is that Perl extensions never import data from the core DLL. They always import functions. In many cases, they can hide this fact with the pre-processor. In the Python world, this qould be equivilent to never accessing Py_None directly - always via a "PyGetNone()" type function. As mentioned, this could possibly be hidden so that code still uses "Py_None". One advantage they mentioned a number of times is avoiding dependencies on differing Perl versions. By avoiding the import of data, they have far more possibilities, including the use of LoadLibrary(), and a new VC6 linker feature called "delay loading". To my mind, it would be quite difficult to make this work for Python. There are a a large number of data items we import, and adding a function call indirection to each one sounds a pain. [As a semi-related issue: This "delay loading" feature is very cool - basically, the EXE loader will not resolve external DLL references until actually used. This is the same trick mentioned on comp.lang.python, where they saw _huge_ startup increases (although the tool used there was a third-party tool). The thread in question on c.l.py resolved that, for some reason, the initialization of the Windows winsock library was taking many seconds on that particular PC. Guido - are you on VC6 yet? If so, I could look into this linker option, and see how it improves startup performance on Windows. Note - this feature only works if no data is imported - hence, we could use it in Python16.dll, as most of its imports are indeed functions. Python extension modules can not use it against Python16 itself as they import data.] Mark.
Anyway, ActivePerl installs its DLLs (all 5) in c:\Perl\bin\. So there. It also didn't change PATH for me, even though the docs mention that it does -- maybe only on NT?
In another mail you asked David to look into how Active State handle their DLLs. Well, Trent Mick started the ball rolling...
The answer is that Perl extensions never import data from the core DLL. They always import functions. In many cases, they can hide this fact with the pre-processor.
This doesn't answer my question. My question is how they support COM without having a DLL in the system directory. Or at least I don't understand how not importing data makes a difference.
By avoiding the import of data, they have far more possibilities, including the use of LoadLibrary(),
For what do they use LoadLibrary()? What is it? We use LoadLibraryEx() -- isn't that just as good?
and a new VC6 linker feature called "delay loading".
To my mind, it would be quite difficult to make this work for Python. There are a a large number of data items we import, and adding a function call indirection to each one sounds a pain.
Agreed.
[As a semi-related issue: This "delay loading" feature is very cool - basically, the EXE loader will not resolve external DLL references until actually used. This is the same trick mentioned on comp.lang.python, where they saw _huge_ startup increases (although the tool used there was a third-party tool). The thread in question on c.l.py resolved that, for some reason, the initialization of the Windows winsock library was taking many seconds on that particular PC.
Guido - are you on VC6 yet?
Yes -- I promised myself I'd start using VC6 for the 1.6 release cycle, and I did.
If so, I could look into this linker option, and see how it improves startup performance on Windows. Note - this feature only works if no data is imported - hence, we could use it in Python16.dll, as most of its imports are indeed functions. Python extension modules can not use it against Python16 itself as they import data.]
But what DLLs does python16 use that could conceivably be delay-loaded? Note that I have a feeling that there are a few standard extensions that should become separate PYDs -- e.g. socket (for the above reason) and unicodedata. This would greatly reduce the size of python16.dll. Since this way we manage our own DLL loading anyway, what's the point of delay-loading? --Guido van Rossum (home page: http://www.python.org/~guido/)
The answer is that Perl extensions never import data from the core DLL. They always import functions. In many cases, they can hide this fact with the pre-processor.
This doesn't answer my question. My question is how they support COM without having a DLL in the system directory. Or at least I don't understand how not importing data makes a difference.
By not using data, they can use either "delay load", or fully dynamic loading. Fully dynamic loading really just involves getting every API function via GetProcAddress() rather than having implicit linking via external references. GetProcAddress() can retrieve data items, but only their address, leaving us still in a position where "Py_None" doesnt work without magic. Delay Loading involves not loading the DLL until the first reference is used. This also lets you define code that locates the DLL to be used. This code is special in a "DllMain" kinda way, but does allow runtime binding to a statically linked DLL. However, it still has the "no data" limitation.
But what DLLs does python16 use that could conceivably be delay-loaded?
Note that I have a feeling that there are a few standard extensions that should become separate PYDs -- e.g. socket (for the above reason) and unicodedata. This would greatly reduce the size of python16.dll.
Agreed - these were my motivation. If these are moving to external modules then I am happy. I may have a quick look for other preloaded DLLs we can avoid - worth a look for the sake of a linker option :-) Mark.
The answer is that Perl extensions never import data from the core DLL. They always import functions. In many cases, they can hide this fact with the pre-processor.
This doesn't answer my question. My question is how they support COM without having a DLL in the system directory. Or at least I don't understand how not importing data makes a difference.
By not using data, they can use either "delay load", or fully dynamic loading.
Fully dynamic loading really just involves getting every API function via GetProcAddress() rather than having implicit linking via external references. GetProcAddress() can retrieve data items, but only their address, leaving us still in a position where "Py_None" doesnt work without magic.
Actually, Py_None is just a macro that expands to the address of some data -- isn't that exactly what we need?
Delay Loading involves not loading the DLL until the first reference is used. This also lets you define code that locates the DLL to be used. This code is special in a "DllMain" kinda way, but does allow runtime binding to a statically linked DLL. However, it still has the "no data" limitation.
But what DLLs does python16 use that could conceivably be delay-loaded?
Note that I have a feeling that there are a few standard extensions that should become separate PYDs -- e.g. socket (for the above reason) and unicodedata. This would greatly reduce the size of python16.dll.
Agreed - these were my motivation. If these are moving to external modules then I am happy. I may have a quick look for other preloaded DLLs we can avoid - worth a look for the sake of a linker option :-)
OK, I'll look into moving socket and unicodedata out of python16.dll. But, I still don't understand why Perl/COM doesn't need a DLL in the system directory. Or is it just because they change PATH? (I don't know zit about COM, so that may be it. I understand that a COM object is registered (in the registry) as an entry point of a DLL. Couldn't that DLL be specified by absolute pathname??? Then no search path would be necessary.) --Guido van Rossum (home page: http://www.python.org/~guido/)
But, I still don't understand why Perl/COM doesn't need a DLL in the system directory. Or is it just because they change PATH?
(I don't know zit about COM, so that may be it. I understand that a COM object is registered (in the registry) as an entry point of a DLL. Couldn't that DLL be specified by absolute pathname??? Then no search path would be necessary.)
Yes - but it all gets back to the exact same problem that got us here in the first place: * COM object points to \Python1.6\PythonCOM16.dll * PythonCOM16.dll has link-time reference to Python16.dll * As COM just uses LoadLibrary(), the path of PythonCOM16.dll is not used to resolve its references - only the path of the host .EXE, the system path, etc. End result is Python16.dll is not found, even though it is in the same directory. So, if you have the opportunity to intercept the link-time reference to a DLL (or, obviously, use LoadLibrary()/GetProcAddress() to reference the DLL), you can avoid override the search path. Thus, if PythonCOM16.dll could intercept its references to Python16.dll, it could locate the correct Python16.dll with runtime code. However, as we import data from Python16.dll rather then purely addresses, we can't use any of these interception solutions. If we could hide all data references behind macros, then we could possibly arrange it. Perl _does_ use such techniques, so can arrange for the runtime type resolution. (Its not clear if Perl uses "dynamic loading" via GetProcAddress(), or delayed loading via the new VC6 feature - I believe the former, but the relevant point is that they definately hide data references behind magic...) Mark.
Guido van Rossum wrote:
But, I still don't understand why Perl/COM doesn't need a DLL in the system directory. Or is it just because they change PATH?
Here is some generic info which may help, or perhaps you already know it. If you have a DLL head.dll or EXE head.exe which needs another DLL needed.dll, you can link needed.dll with head, and the system will find all data and module names automatically (well, almost). When head is loaded, needed.dll must be available, or head will fail to load. This can be confusing. For example, I once tried to port PIL to my new Python mini-GUI model, and my DLL failed. Only after some confusion did I realize that PIL is linked with Tk libs, and would fail to load if they were not present, even though I was not using them. I think what Mark is saying is that Microsoft now has an option to do delayed DLL loading. The load of needed.dll is delayed until a function in needed.dll is called. This would have meant that PIL would have worked provided I never called a Tk function. I think he is also saying that this feature can only trap function calls, not pointer access to data, so it won't work in the context of data access (maybe it would if a function call came first). Of course, if you access all data through a function call GetMyData(), it all works. As an alternative, head.[exe|dll] would not be linked with needed.dll, and so needed.dll need not be present. To access functions by name in needed.dll, you call LoadLibrary or LoadLibraryEx to open needed.dll, and then call GetProcAddress() to get a pointer to named functions. In the case of data items, the pointer is dereferenced twice, that is, data = **pt. Python uses this strategy to load PYD's, and accesses the sole function initmodule(). Then the rest of the data is available through Python mechanisms which effectively substitute for normal DLL access. The alternative search path available in LoadLibraryEx only affects head.dll, and causes the system to look in the directory of needed.dll instead of the directory of the ultimate executable for finding other needed DLL's. So on Windows, Python needs PYTHONPATH to find PYD's, and if the PYD's need further DLL's those DLL's can be in the directory of the PYD, or on the usual DLL search path provided the "alternate search path" is used. Probably you alread know this, but maybe it will help the Windozly-challenged follow along. JimA
[Guido]
... I'm waiting for Tim Peters' response in this thread -- if I recall he was the one who said that python1x.dll should not go into the system directory.
Not that I don't say a lot of dumb-ass things <wink>, but I strongly doubt I would have said this one. In my brief career as a Windows app provider, I learned four things, the first three loudly gotten across by seriously unhappy users: 1. Contra MS guidelines, dump the core DLLs in the system directory. 2. Contra MS guidelines, install the app by default in C:\name_of_app\. 3. Contra MS guidelines, put all the config options you can in a text file C:\name_of_app\name_of_app.ini instead of the registry. 4. This one was due to my boss: Contra MS guidelines, put a copy of every MS system DLL you rely on under C:\name_of_app\, so you don't get screwed when MS introduces an incompatible DLL upgrade. In the end, the last one is the only one I disagreed with (in recent years I believe MS DLL upgrades have gotten much more likely to fix bugs than to introduce incompatibilities; OTOH, from Tcl to Macsyma Pro I see 6 apps on my home machine that use their own copy of msvcrt.dll -- /F, if you're reading, how come the Pythonworks beta does this?).
... I've definitely heard people complain that it is evil to install directories in the system directory. Seems there are different schools of thought...
Well, mucking with the system directories is horrid! Nobody likes doing it. AFAIK, though, there's really no realistic alternative. It's the only place you *know* will be on the PATH, and if an app embedding Python can't rely on PATH, it will have to hardcode the Python DLL path itself.
Another issue: MSVCRT.DLL and its friend MSVCIRT.DLL will also go into the system directory. I will now be distributing with the VC++ 6.0 servicepack 1 versions of these files. Won't this be a problem for installations that already have an older version? (Now that I think of it, this is another reason why I decided that at least the alpha release should install everything in MAINDIR -- to limit the damage. Any informed opinions?)
You're using a std installer, and MS has rigid rules for these DLLs that the installer will follow by magic. Small comfort if things break, but this one is (IMO) worth playing along with.
2. Contra MS guidelines, install the app by default in C:\name_of_app\.
Ive got to agree here. While I also see Greg's point, the savvy user can place it where they want, while the "average user" is better of with a more reasonable default. However, I would tend to go for "\name_of_app" rooted from the Windows drive. It is likely that this will be the default drive when a command prompt is open, so a simple "cd \python1.6" will work. This is also generally the same drive the default "Program Files" is on too.
You're using a std installer, and MS has rigid rules for these DLLs that the installer will follow by magic. Small comfort if things break, but this one is (IMO) worth playing along with.
I checked the installer, and these MSVC dlls are indeed set to install only if the existing version is the "same or older". Annoyingly, it doesnt have an option for only "if older"! They are also set to correctly reference count in the registry. I believe that by installing a single custom DLL into the system directory, plus correctly installing some MS system DLLs into the system directory we are being perfect citizens. [Interestingly, Windows 2000 has a system process that continually monitors the system directory. If it detects that a "protected file" has been changed, it promptly copies the original back over the top! I believe the MSVC*.dlls are in the protected list, so can only be changed with a service pack release anyway. Everything _looks_ like it updates - Windows just copies it back!] Mark.
[Mark Hammond]
... However, I would tend to go for "\name_of_app" rooted from the Windows drive. It is likely that this will be the default drive when a command prompt is open, so a simple "cd \python1.6" will work. This is also generally the same drive the default "Program Files" is on too.
Yes, "C:\" doesn't literally mean "C:\" any more than "Program Files" literally means "Program Files" <0.1 wink>. By "C:\" I meant "drive where the thing we conveniently but naively call 'Program Files' lives"; naming the registry key whose value is this thing is more accurate but less helpful; the installer will have some magic predefined name which would be most helpful to give, but without the installer docs here I can't guess what that is.
... [Interestingly, Windows 2000 has a system process that continually monitors the system directory. If it detects that a "protected file" has been changed, it promptly copies the original back over the top! I believe the MSVC*.dlls are in the protected list, so can only be changed with a service pack release anyway. Everything _looks_ like it updates - Windows just copies it back!]
Thanks for making my day <wink>.
[Guido]
... I'm waiting for Tim Peters' response in this thread -- if I recall he was the one who said that python1x.dll should not go into the system directory.
[Tim]
Not that I don't say a lot of dumb-ass things <wink>, but I strongly doubt I would have said this one.
OK, it must be my overworked tired brain that is playing games with me. It might have been Jim Ahlstrom then, our resident Windows 3.1 supporter. :-)
In my brief career as a Windows app provider, I learned four things, the first three loudly gotten across by seriously unhappy users:
1. Contra MS guidelines, dump the core DLLs in the system directory. 2. Contra MS guidelines, install the app by default in C:\name_of_app\.
It's already been said that the drive letter could be chosen more carefully. I wonder if the pathname should also be an 8+3 (max) name, so that it can be relyably typed into a DOS window.
3. Contra MS guidelines, put all the config options you can in a text file C:\name_of_app\name_of_app.ini instead of the registry. 4. This one was due to my boss: Contra MS guidelines, put a copy of every MS system DLL you rely on under C:\name_of_app\, so you don't get screwed when MS introduces an incompatible DLL upgrade.
In the end, the last one is the only one I disagreed with (in recent years I believe MS DLL upgrades have gotten much more likely to fix bugs than to introduce incompatibilities; OTOH, from Tcl to Macsyma Pro I see 6 apps on my home machine that use their own copy of msvcrt.dll -- /F, if you're reading, how come the Pythonworks beta does this?).
Probably because Pythonworks doesn't care about COM or embedding. Anyway, I now agree with you on 1-2 and on not following 4. As for 3, I think that for Mark's COM support to work, the app won't necessarily be able to guess what \name_of_app\ is, so that's where the registry comes in handy. PATH info is really about all that Python puts in the registry, so I think we're okay here. (Also if you read PC\getpathp.c in 1.6, you'll see that it now ignores most of the registry when it finds the installation through a search based on argv[0].)
... I've definitely heard people complain that it is evil to install directories in the system directory. Seems there are different schools of thought...
Well, mucking with the system directories is horrid! Nobody likes doing it. AFAIK, though, there's really no realistic alternative. It's the only place you *know* will be on the PATH, and if an app embedding Python can't rely on PATH, it will have to hardcode the Python DLL path itself.
Another issue: MSVCRT.DLL and its friend MSVCIRT.DLL will also go into the system directory. I will now be distributing with the VC++ 6.0 servicepack 1 versions of these files. Won't this be a problem for installations that already have an older version? (Now that I think of it, this is another reason why I decided that at least the alpha release should install everything in MAINDIR -- to limit the damage. Any informed opinions?)
You're using a std installer, and MS has rigid rules for these DLLs that the installer will follow by magic. Small comfort if things break, but this one is (IMO) worth playing along with.
One more thing that I just realized. There are a few Python extension modules (_tkinter and the new pyexpat) that rely on external DLLs: _tkinter.pyd needs tcl83.dll and tk83.dll, and pyexpat.pyd needs xmlparse.dll and xmltok.dll. If I understand correctly how the path rules work, these have to be on PATH too (although the pyd files don't have to be). This worries me -- these aren't official MS DLLs and neither are the our own, so we could easily stomp on some other app's version of the same... (The tcl folks don't change their filename when the 3rd version digit changes, e.g. 8.3.0 -> 8.3.1, and expat has no versions at all.) Is there a better solution? --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <guido@python.org> wrote:
I wonder if the pathname should also be an 8+3 (max) name, so that it can be relyably typed into a DOS window.
"\py<number>" is reserved ;-)
OK, it'll be \python16 then. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
OK, it must be my overworked tired brain that is playing games with me. It might have been Jim Ahlstrom then, our resident Windows 3.1 supporter. :-)
I think I've been insulted. What's wrong with Windows 3.1?? :-)
1. Contra MS guidelines, dump the core DLLs in the system directory.
The Python DLL must really go in the Windows system directory. I don't see any other choice. This is in accordance with Microsoft guidelines AFAIK, or anyway, that's the only way it Just Works. The Python16.dll is a system file if you are using COM, and it supports an embedded scripting language, so it goes into the system dir. QED.
3. Contra MS guidelines, put all the config options you can in a text file C:\name_of_app\name_of_app.ini instead of the registry.
This is an excellent practice, and there should be a standard module to deal with .ini files. But, as you say, the registry is sometimes needed.
4. This one was due to my boss: Contra MS guidelines, put a copy of every MS system DLL you rely on under C:\name_of_app\, so you don't get screwed when MS introduces an incompatible DLL upgrade.
Yuk. More trouble than it's worth.
I've definitely heard people complain that it is evil to install directories in the system directory. Seems there are different schools of thought...
It is very illegal to install directories as opposed to DLL's. Do you really mean directories? If so, don't do that.
Another issue: MSVCRT.DLL and its friend MSVCIRT.DLL will also go into the system directory. I will now be distributing with the VC++ 6.0
If you distribute these, you must check version numbers and only replace old versions. Wise and other installers do this easily. Doing otherwise is evil and unacceptable. Checking file dates is not good enough either.
servicepack 1 versions of these files. Won't this be a problem for installations that already have an older version?
Probably not, thanks to Microsoft's valiant testing efforts.
(Now that I think of it, this is another reason why I decided that at least the alpha release should install everything in MAINDIR -- to limit the damage. Any informed opinions?)
Distribute these files with a valid Wise install script which checks VERSIONS.
One more thing that I just realized. There are a few Python extension modules (_tkinter and the new pyexpat) that rely on external DLLs: _tkinter.pyd needs tcl83.dll and tk83.dll, and pyexpat.pyd needs xmlparse.dll and xmltok.dll.
Welcome to the club.
If I understand correctly how the path rules work, these have to be on PATH too (although the pyd files don't have to be). This worries me -- these aren't official MS DLLs and neither are the our own, so we could easily stomp on some other app's version of the same... (The tcl folks don't change their filename when the 3rd version digit changes, e.g. 8.3.0 -> 8.3.1, and expat has no versions at all.)
Is there a better solution?
This is a daily annoyance and risk in the Windows world. If you require Tk, then you need to completely understand how to produce a valid Tk distribution. Same with PIL (which requires Tk). Often you won't know that some pyd requires some other obscure DLL. To really do this you need something high level. Like rpm's on linux. On Windows, people either write complex install programs with Wise et al, or run third party installers provided with (for example) Tk from simpler install scripts. It is then up to the Tk people to know how to install it, and how to deal with version upgrades. JimA
Hi! [...]
3. Contra MS guidelines, put all the config options you can in a text file C:\name_of_app\name_of_app.ini instead of the registry.
James C. Ahlstrom:
This is an excellent practice, and there should be a standard module to deal with .ini files. [...]
One half of it is already there in the standard library: 'ConfigParser'. From my limited knowledge about windows (shrug) this can at least read .ini files. Writing this info again out to a file shouldn't be too hard. Regards, Peter
[me]
One more thing that I just realized. There are a few Python extension modules (_tkinter and the new pyexpat) that rely on external DLLs: _tkinter.pyd needs tcl83.dll and tk83.dll, and pyexpat.pyd needs xmlparse.dll and xmltok.dll.
[Jim A]
Welcome to the club.
I'm not sure what you mean by this?
If I understand correctly how the path rules work, these have to be on PATH too (although the pyd files don't have to be). This worries me -- these aren't official MS DLLs and neither are the our own, so we could easily stomp on some other app's version of the same... (The tcl folks don't change their filename when the 3rd version digit changes, e.g. 8.3.0 -> 8.3.1, and expat has no versions at all.)
Is there a better solution?
This is a daily annoyance and risk in the Windows world. If you require Tk, then you need to completely understand how to produce a valid Tk distribution. Same with PIL (which requires Tk). Often you won't know that some pyd requires some other obscure DLL. To really do this you need something high level. Like rpm's on linux. On Windows, people either write complex install programs with Wise et al, or run third party installers provided with (for example) Tk from simpler install scripts. It is then up to the Tk people to know how to install it, and how to deal with version upgrades.
Calculating the set of required DLLs isn't the problem. I have a tool (Dependency Viewer) that shows me exactly the dependencies (it recurses down any DLLs it finds and shows their dependencies too, using the nice MFC tree widget). The problem is where should I install these extra DLLs. In 1.5.2 I included a full Tcl/Tk installer (the unadorned installer from Scriptics). The feedback over the past year showed that this was a bad idea: it stomped over existing Tcl/Tk installations, new Tcl/Tk installations stomped over it, people chose to install Tcl/Tk on a different volume than Python, etc. In 1.6, I am copying the necessary files from the Tcl/Tk installation into the Python directory. This actually installs fewer files than the full Tcl/Tk installation (but you don't get the Tcl/Tk docs). It gives me complete control over which Tcl/Tk version I use without affecting other Tcl/Tk installations that might exist. This is how professional software installations deal with inclusions. However the COM DLL issue might cause problems: if the Python directory is not in the search path because we're invoked via COM, there are only two places where the Tcl/Tk DLLs can be put so they will be found: in the system directory or somewhere along PATH. Assuming it is still evil to modify PATH, we would end up with Tcl/Tk in the system directory, where it could once again interfere with (or be interfered by) other Tcl/Tk installations! Someone suggested that COM should not use Tcl/Tk, and then the Tcl/Tk DLLs can live in the Python tree. I'm not so sure -- I can at least *imagine* that someone would use Tcl/Tk to give their COM object a bit of a GUI. Moreover, this argument doesn't work for pyexpat -- COM apps are definitely going to expect to be able to use pyexpat! It's annoying. I have noticed, however, that you can use os.putenv() (or assignment to os.environ[...]) to change the PATH environment variable. The FixTk.py script in Python 1.5.2 used this -- it looked in a few places for signs of a Tcl/Tk installation, and then adjusted PATH to include the proper directory before trying to import _tkinter. Maybe there's a solution here? The Python DLL could be the only thing in the system directory, and from the registry it could know where the Python directory was. It could then prepend this directory to PATH. This is not so evil as mucking with PATH at install time, I think, since it is only done when Python16.dll is actually loaded. Would this always work? (Windows 95, 98, NT, 2000?) Wouldn't it run out of environment space? Wouldn't it break other COM apps? Is the PATH truly separate per process? --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido]
Someone suggested that COM should not use Tcl/Tk, and then the Tcl/Tk DLLs can live in the Python tree. I'm not so sure -- I can at least *imagine* that someone would use Tcl/Tk to give their COM object a bit of a GUI. Moreover, this argument doesn't work for pyexpat -- COM apps are definitely going to expect to be able to use pyexpat!
Me. Would you have any sympathy for someone who wanted to make a GUI an integral part of a web server? Or would you tell them to get a brain and write a GUI that talks to the web server? Same issue. (Though not, I guess, for pyexpat).
It's annoying.
I have noticed, however, that you can use os.putenv() (or assignment to os.environ[...]) to change the PATH environment variable. The FixTk.py script in Python 1.5.2 used this -- it looked in a few places for signs of a Tcl/Tk installation, and then adjusted PATH to include the proper directory before trying to import _tkinter. Maybe there's a solution here? The Python DLL could be the only thing in the system directory, and from the registry it could know where the Python directory was. It could then prepend this directory to PATH. This is not so evil as mucking with PATH at install time, I think, since it is only done when Python16.dll is actually loaded.
The drawback of relying on PATH is that then some other jerk (eg you, last year <wink>) will stick something of the same name in the system directory and break your installation.
Would this always work? (Windows 95, 98, NT, 2000?) Wouldn't it run out of environment space? Wouldn't it break other COM apps? Is the PATH truly separate per process?
Are there any exceptions to this: - dynamically load a .pyd - .pyd implicitly loads the .dll ? If that's always the case, then you can temporarily cd to the right directory before the dynamic load, and the implicit load should work. As for the others: probably not; can't see how; yes. - Gordon
Me. Would you have any sympathy for someone who wanted to make a GUI an integral part of a web server? Or would you tell them to get a brain and write a GUI that talks to the web server? Same issue. (Though not, I guess, for pyexpat).
Not all COM objects are used in web servers. Some are used in GUI contexts (aren't Word and Excel and even IE really mostly COM objects these days?).
It's annoying.
I have noticed, however, that you can use os.putenv() (or assignment to os.environ[...]) to change the PATH environment variable. The FixTk.py script in Python 1.5.2 used this -- it looked in a few places for signs of a Tcl/Tk installation, and then adjusted PATH to include the proper directory before trying to import _tkinter. Maybe there's a solution here? The Python DLL could be the only thing in the system directory, and from the registry it could know where the Python directory was. It could then prepend this directory to PATH. This is not so evil as mucking with PATH at install time, I think, since it is only done when Python16.dll is actually loaded.
The drawback of relying on PATH is that then some other jerk (eg you, last year <wink>) will stick something of the same name in the system directory and break your installation.
Yes, that's a problem, especially since it appears that PATH is searched *last*. (I wonder if this could explain the hard-to-reproduce crashes that people report when quitting IDLE?)
Would this always work? (Windows 95, 98, NT, 2000?) Wouldn't it run out of environment space? Wouldn't it break other COM apps? Is the PATH truly separate per process?
Are there any exceptions to this: - dynamically load a .pyd - .pyd implicitly loads the .dll ?
I think this is always the pattern (except that some DLLs will implicitly load other DLLs, and so on).
If that's always the case, then you can temporarily cd to the right directory before the dynamic load, and the implicit load should work.
Hm, I would think that the danger of temporarily changing the current directory is at least as big as that of changing PATH. (What about other threads? What if you run into an error and don't get a chance to cd back?)
As for the others: probably not; can't see how; yes.
--Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
[Jim A]
Welcome to the club.
I'm not sure what you mean by this?
It sounded like you were joining the Microsoft afflicted...
In 1.5.2 I included a full Tcl/Tk installer (the unadorned installer from Scriptics). The feedback over the past year showed that this was a bad idea: it stomped over existing Tcl/Tk installations, new Tcl/Tk installations stomped over it, people chose to install Tcl/Tk on a different volume than Python, etc.
My first thought was that this was the preferred solution. It is up to Scriptics to provide an installer for Tk and for Tk customers to use it. Any problems with installing Tk are Scriptics' problem. I don't know the reasons it stomped over other installs etc. But either Tk customers are widely using non-standard installs, or the Scriptics installer is broken, or there is no such thing as a standard Tk install. This is fundamentally a Scriptics problem, but I understand it is a Python problem too. There may still be the problem that a standard Tk install might not be accessible to Python. This needs to be worked out with Scriptics. An environment variable could be set, the registry used etc. Assuming there is a standard Tk install and a way for external apps to use Tk, then we can still use the (fixed) Scriptics installer.
Assuming it is still evil to modify PATH, we would end up with Tcl/Tk in the system directory, where it could once again interfere with (or be interfered by) other Tcl/Tk installations!
I seems to me that the correct Tk install script would put Tk DLL's in the system dir, and use the registry to find the libraries and other needed files. The exe's could go in a program directory somewhere. This is what I have to come to expect from professional software for DLL's which are expected to be used from multiple apps, as opposed to DLL's which are peculiar to one app. If the Tk installer did this, Tk would Just Work, and it would Just Work with third party apps (Tk clients) like Python too. Sorry, I have to run to a class. To be continued tomorrow.... JimA
[Jim A]
Welcome to the club.
[me]
I'm not sure what you mean by this?
It sounded like you were joining the Microsoft afflicted...
Indeed :-(
In 1.5.2 I included a full Tcl/Tk installer (the unadorned installer from Scriptics). The feedback over the past year showed that this was a bad idea: it stomped over existing Tcl/Tk installations, new Tcl/Tk installations stomped over it, people chose to install Tcl/Tk on a different volume than Python, etc.
My first thought was that this was the preferred solution. It is up to Scriptics to provide an installer for Tk and for Tk customers to use it. Any problems with installing Tk are Scriptics' problem. I don't know the reasons it stomped over other installs etc. But either Tk customers are widely using non-standard installs, or the Scriptics installer is broken, or there is no such thing as a standard Tk install. This is fundamentally a Scriptics problem, but I understand it is a Python problem too.
There may still be the problem that a standard Tk install might not be accessible to Python. This needs to be worked out with Scriptics. An environment variable could be set, the registry used etc. Assuming there is a standard Tk install and a way for external apps to use Tk, then we can still use the (fixed) Scriptics installer.
The Tk installer has had these problems for a long time. I don't want to have to argue with them, I think it would be a waste of time.
Assuming it is still evil to modify PATH, we would end up with Tcl/Tk in the system directory, where it could once again interfere with (or be interfered by) other Tcl/Tk installations!
I seems to me that the correct Tk install script would put Tk DLL's in the system dir, and use the registry to find the libraries and other needed files. The exe's could go in a program directory somewhere. This is what I have to come to expect from professional software for DLL's which are expected to be used from multiple apps, as opposed to DLL's which are peculiar to one app. If the Tk installer did this, Tk would Just Work, and it would Just Work with third party apps (Tk clients) like Python too.
OK, you go argue with the Tcl folks. They create a vaguely unix-like structure under c:\Program Files\Tcl: subdirectories lib, bin, include, and then they dump their .exe and their .dll files in the bin directory. They also try to munge PATH to include their bin directory, but that often doesn't work (not on Windows 95/98 anyway). --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
OK, you go argue with the Tcl folks. They create a vaguely unix-like structure under c:\Program Files\Tcl: subdirectories lib, bin, include, and then they dump their .exe and their .dll files in the bin directory. They also try to munge PATH to include their bin directory, but that often doesn't work (not on Windows 95/98 anyway).
That is even worse than I thought. Obviously they are incompetent in Windows. Mark's suggestion is a great one! JimA
I wonder if the pathname should also be an 8+3 (max) name, so that it can be relyably typed into a DOS window.
To be honest, I can not see a good reason for this any more. The installation package only works on Win95/98/NT/2000 - all of these support long file names on all their supported file systems. So, any where that this installer will run, the "command prompt" on this system will correctly allow "cd \Python-1.6-and-any-thing-else-I-like-ly" :-) [OTOH, I tend to prefer "Python1.6" purely from an "easier to type" POV] Mark.
[Guido]
... I wonder if the pathname should also be an 8+3 (max) name, so that it can be relyably typed into a DOS window.
Yes, but for a different reason: Many sites still use older Novell file servers that screw up on non-8.3 names in a variety of unpleasant ways. Just went thru this at Dragon again, where the language modeling group created a new file format with a 4-letter extension; they had to back off to 3 letters because half the company couldn't get at the new files. BTW, two years ago it was much worse, and one group started using Python instead of Java partly because .java files didn't work over the network at all!
Tim Peters wrote:
4. This one was due to my boss: Contra MS guidelines, put a copy of every MS system DLL you rely on under C:\name_of_app\, so you don't get screwed when MS introduces an incompatible DLL upgrade.
In the end, the last one is the only one I disagreed with (in recent years I believe MS DLL upgrades have gotten much more likely to fix bugs than to introduce incompatibilities; OTOH, from Tcl to Macsyma Pro I see 6 apps on my home machine that use their own copy of msvcrt.dll -- /F, if you're reading, how come the Pythonworks beta does this?).
we've been lazy... in the pre-IE days, some machines came without any msvcrt.dll at all. so since we have to ship it, I guess it was easier to ship it along with all the other components, rather than implementing the "install in system directory only if newer" stuff... (I think it's on the 2.0 todo list ;-) </F>
[Guido]
I'm waiting for Tim Peters' response in this thread -- if I recall he was the one who said that python1x.dll should not go into the system directory.
Some time ago Tim and I said that the place for a DLL that is intimately tied to an EXE is in the EXE's directory. The search path: 1) the EXE's directory 2) the current directory (useless) 3) the system directory 4) the Windows directory 5) the PATH For a general purpose DLL, that makes the system directory the only sane choice (if modifying PATH was sane, then PATH would be saner, but a SpecTCL will just screw you up). Things that go in the system directory should maintain backwards compatibility. For a DLL, that means all the old entry points are still there, in the same order with new ones at the end. For Python, there's no crying need to conform for now, but if (when?) embedding Python becomes ubiquitous, this (or some other scheme) may need to be considered. - Gordon
Some time ago Tim and I said that the place for a DLL that is intimately tied to an EXE is in the EXE's directory.
But the conclusion seems to be that python1x.dll is not closely tied to python.exe -- it may be invoked via COM.
The search path: 1) the EXE's directory 2) the current directory (useless) 3) the system directory 4) the Windows directory 5) the PATH
For a general purpose DLL, that makes the system directory the only sane choice (if modifying PATH was sane, then PATH would be saner, but a SpecTCL will just screw you up).
Things that go in the system directory should maintain backwards compatibility. For a DLL, that means all the old entry points are still there, in the same order with new ones at the end. For Python, there's no crying need to conform for now, but if (when?) embedding Python becomes ubiquitous, this (or some other scheme) may need to be considered.
Where should I put tk83.dll etc.? In the Python\DLLs directory, where _tkinter.pyd also lives? --Guido van Rossum (home page: http://www.python.org/~guido/)
[Gordon]
Some time ago Tim and I said that the place for a DLL that is intimately tied to an EXE is in the EXE's directory. [Guido] But the conclusion seems to be that python1x.dll is not closely tied to python.exe -- it may be invoked via COM.
Right.
Where should I put tk83.dll etc.? In the Python\DLLs directory, where _tkinter.pyd also lives?
Won't work (unless there are some tricks in MSVC 6 I don't know about). Assuming no one is crazy enough to use Tk in a COM server, (or rather, that their insanity need not be catered to), then I'd vote for the directory where python.exe and pythonw.exe live. - Gordon
Where should I put tk83.dll etc.? In the Python\DLLs directory, where _tkinter.pyd also lives?
Won't work (unless there are some tricks in MSVC 6 I don't know about). Assuming no one is crazy enough to use Tk in a COM server, (or rather, that their insanity need not be catered to), then I'd vote for the directory where python.exe and pythonw.exe live.
What we can do is have Python itself use LoadLibraryEx() to load the .pyd files. This _will_ allow any dependant DLLs to be found in the same directory as the .pyd. [And as I mentioned, if the whole world would use LoadLibraryEx(), our problem would go away] LoadLibraryEx() is documented as working on all Win9x and NT from 3.1.
What we can do is have Python itself use LoadLibraryEx() to load the .pyd files. This _will_ allow any dependant DLLs to be found in the same directory as the .pyd. [And as I mentioned, if the whole world would use LoadLibraryEx(), our problem would go away]
Doh! [Sound of forehead being slapped violently] We already use LoadLibraryEx()! So we can drop all the dependent dlls in the DLLs directory which has the PYD files as well. Case closed. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Gordon writes]
Things that go in the system directory should maintain backwards compatibility. For a DLL, that means all the old entry points are still there, in the same order with new ones at the end.
Actually, the order is not important unless people link to you by ordinal (in which case you are likely to specify the ordinal in the .def file anyway). The Win32 loader is smart enough to be able to detect that all ordinals are the same as when it was linked, and use a fast-path. If ordinal name-to-number mappings have changed, the runtime loader takes a slower path that fixes up these differences. So what you suggest is ideal, but not really necessary.
For Python, there's no crying need to conform for now, but if (when?) embedding Python becomes ubiquitous, this (or some other scheme) may need to be considered.
I believe Python will already do this, almost by accident, due to the conservative changes with each minor Python release. Eg, up until Python 1.6 was branded as 1.6, I was still linking my win32all extensions against the CVS version. When I remembered I would switch back to the 1.5.2 release ones, but when I forgot I never had a problem. People running a release version 1.5.2 could happily use my extensions linked with the latest 1.5.2+ binaries. We-could-even-blame-the-time-machine-at-a-strecth-ly, Mark.
[Mark Hammond]
The 1.6a1 installer on Windows copies Python16.dll into the Python directory, rather than the system32 directory like 1.5.x. We discussed too long ago on this list not why this was probably not going to work. I guess Guido decided to "suck it and see" - which is fine.
But guess what - it doesnt work :-( ... I dont see any better answer than System32 :-( Thoughts?
Same as yours! Guido went off and innovated here -- always a bad sign <wink>. OTOH, I've got no use for "Program Files" -- make the cmdline version easy to use too.
participants (8)
-
Fredrik Lundh -
Gordon McMillan -
Greg Stein -
Guido van Rossum -
James C. Ahlstrom -
Mark Hammond -
pfï¼ artcom-gmbh.de -
Tim Peters