Adventures with x64, VS7 and VS8 on Windows
Hi all, I hope the cross-post is appropriate. I've started playing with getting the pywin32 extensions building under the AMD64 architecture. I started building with Visual Studio 8 (it was what I had handy) and I struck a few issues relating to the compiler version that I thought worth sharing. * In trying to build x64 from a 32bit VS7 (ie, cross-compiling via the PCBuild directory), the python.exe project fails with: pythoncore fatal error LNK1112: module machine type 'X86' conflicts with target machine type 'AMD64' is this a known issue, or am I doing something wrong? * The PCBuild8 project files appear to work without modification (I only tried native compilation here though, not a cross-compile) - however, unlike the PCBuild directory, they place all binaries in a 'PCBuild8/x64' directory. While this means that its possible to build for multiple architectures from the same source tree, it makes life harder for tools like 'distutils' - eg, distutils already knows about the 'PCBuild' directory, but it knows nothing about either PCBuild8 or PCBuild8/x64. A number of other build processes also know to look inside a PCBuild directory (eg, Mozilla), so instead of formalizing PCBuild8, I think we should merge PCBuild8 into PCBuild. This could mean PCBuild/vs7 and PCBuild/vs8 directories with the "project" files, but binaries would still be generated in the 'PCBuild' (or PCBuild/x64) directory. This would mean the same tree isn't capable of hosting 2 builds from different VS compilers, but I think that is reasonable (if it's a problem, just have multiple source directories). I understand that PCBuild8 is not "official", but in the assumption that future versions of Python will use a compiler later than VS7, it makes sense to me to clean this up now - what are others opinions on this? * Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to generate x64 binaries to their own directory - my expectation is that cross-compiling between platforms is a reasonable use-case, and we should support multiple achitectures for the same compiler version. This would mean formalizing the x64 directory in both 'PCBuild' and distutils, and leaving other external build processes to update as they support x64 builds. Does this make sense? Would this fatally break other scripts used for packaging (eg, the MSI framework)? * Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler versions. VS8 defines this as wchar_t, which I'm struggling to find a formal definition for beyond being 2 bytes. My problem is that code which assumes a 'Py_UNICODE *' could be used in place of a 'WCHAR *' now fails. I believe the intent on Windows has always been "PyUNICODE == 'native unicode'" - should PC/pyconfig.h reflect this (ie, should pyconfig.h grow a version specific definition of PyUNICODE as wchar_t)? * Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts? I know a couple of projects are doing this successfully, including Mozilla, so it has precendent. It does impose a greater burden on people trying to build on Windows, but I'd suggest that in recent times, many people who are likely to want to build Python on Windows are already likely to have a cygwin environment. Simpler mingw builds and nuking MSVC specific build stuff are among the advantages this would bring. It is not worth adding this as "yet another windows build option" - so IMO it is only worth progressing with if it became the "blessed" build process for windows - if there is support for this, I'll work on it as the opportunity presents itself... I'm (obviously) only suggesting we do this on the trunk and am happy to make all agreed changes - but I welcome all suggestions or critisisms of this endeavour... Cheers, Mark
* In trying to build x64 from a 32bit VS7 (ie, cross-compiling via the PCBuild directory), the python.exe project fails with:
pythoncore fatal error LNK1112: module machine type 'X86' conflicts with target machine type 'AMD64'
is this a known issue, or am I doing something wrong?
You are likely doing something wrong: a) I assume it's VS 7.1 (i.e. VS.NET 2003); VS 2002 is not supported at all b) you probably didn't install vsextcomp, but you should. In fact, you don't need all of it, but you do need the cl.exe and link.exe wrappers it comes with - they dispatch to the proper tools from the SDK c) in case it isn't clear: you also need an AMD64 compiler, e.g. from the platform SDK. Unfortunately, Microsoft keeps changing the registry settings for the SDK, so vsextcomp only knows about some selected SDKs. If that causes a problem, please let me know.
* The PCBuild8 project files appear to work without modification (I only tried native compilation here though, not a cross-compile) - however, unlike the PCBuild directory, they place all binaries in a 'PCBuild8/x64' directory. While this means that its possible to build for multiple architectures from the same source tree, it makes life harder for tools like 'distutils' - eg, distutils already knows about the 'PCBuild' directory, but it knows nothing about either PCBuild8 or PCBuild8/x64.
This is an issue to be discussed for Python 2.6. I'm personally hesitant to have the "official" build infrastructure deviate from the layout that has been in-use for so many years, as a lot of things depend on it. I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't.
A number of other build processes also know to look inside a PCBuild directory (eg, Mozilla), so instead of formalizing PCBuild8, I think we should merge PCBuild8 into PCBuild.
Right - PCbuild8 should not get formalized. It probably should continue to be maintained. For 2.6, the first question to answer is: what compiler should it use? I would personally like to see Python "skip" VS 2005 altogether, as it will be soon superceded by Orcas. Unfortunately, it's unclear how long Microsoft will need to release Orcas (and also, when Python 2.6 will be released), so I would like to defer that question by a few months.
I understand that PCBuild8 is not "official", but in the assumption that future versions of Python will use a compiler later than VS7, it makes sense to me to clean this up now - what are others opinions on this?
Not "official" really only means "not used to build the official binaries" - just like PC/VC6. It's still (somewhat) maintained. As for cleaning it up - see above. I would *really* like to skip VS 2005 altogether, as I expect that soon after we decide to use VS 2005, Microsoft will replace it with the next release, stop supporting VS 2005, take the free compiler off the next, and so on (just like they did for VS 2003, soon after we decided to use it for 2.5).
* Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to generate x64 binaries to their own directory - my expectation is that cross-compiling between platforms is a reasonable use-case, and we should support multiple achitectures for the same compiler version.
See above; I disagree. First, "multiple architectures" only means x86, AMD64, and Itanium, and I would like to drop "official" Itanium binaries from 2.6 (even though they could continue to be supported in the build process). Then, even if the Python build itself support multiple simultaneous architectures, the extension modules don't all (correct me if I'm wrong).
This would mean formalizing the x64 directory in both 'PCBuild' and distutils, and leaving other external build processes to update as they support x64 builds. Does this make sense? Would this fatally break other scripts used for packaging (eg, the MSI framework)?
The MSI packaging would need to be changed, certainly. It currently detects the architecture it needs to package by looking at the file type of python.exe; that would have to be changed to give it an explicit parameter what architecture to package, or have it package all architectures it can find.
* Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler versions. VS8 defines this as wchar_t, which I'm struggling to find a formal definition for beyond being 2 bytes.
In C or in C++? In C++, wchar_t is a builtin type, just like short, int, long. So there is no further formal definition. In C (including C99), wchar_t ought to be defined in stddef.h.
My problem is that code which assumes a 'Py_UNICODE *' could be used in place of a 'WCHAR *' now fails. I believe the intent on Windows has always been "PyUNICODE == 'native unicode'" - should PC/pyconfig.h reflect this (ie, should pyconfig.h grow a version specific definition of PyUNICODE as wchar_t)?
I'd rather make it a platform-specific definition (for platform=Windows API). Correct me if I'm wrong, but isn't wchar_t also available in VS 2003 (and even in VC6?). And doesn't it have the "right" definition in all these compilers? So +1 for setting Py_UNICODE to wchar_t on Windows.
* Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts?
What compiler would you use there? I very much like using the VS debugger when developing on Windows, so that capability should not go away. Regards, Martin
Hi Martin,
You are likely doing something wrong: a) I assume it's VS 7.1 (i.e. VS.NET 2003); VS 2002 is not supported at all b) you probably didn't install vsextcomp, but you should. In fact, you don't need all of it, but you do need the cl.exe and link.exe wrappers it comes with - they dispatch to the proper tools from the SDK c) in case it isn't clear: you also need an AMD64 compiler, e.g. from the platform SDK. Unfortunately, Microsoft keeps changing the registry settings for the SDK, so vsextcomp only knows about some selected SDKs. If that causes a problem, please let me know.
I'm using the full-blown VS.NET 2003, as given to a number of python-dev people by Microsoft a number of years ago. This appears to come with the SDK and a 64bit compiler. I'm guessing vsextcomp doesn't use the Visual Studio 'ReleaseAMD64' configuration - would it be OK for me to check in changes to the PCBuild projects for this configuration?
This is an issue to be discussed for Python 2.6. I'm personally hesitant to have the "official" build infrastructure deviate from the layout that has been in-use for so many years, as a lot of things depend on it.
Yes, I agree - although I consider x64 new enough that an opportunity exists to set a new 'standard'. However, if most 'external' build processes will not otherwise need to change for a 64bit environment, then I agree that nothing should change in Python's layout.
Right - PCbuild8 should not get formalized. It probably should continue to be maintained.
So is there something we can do to make distutils play better with binaries built from PCBuild8, even though it is considered temporary? It seems the best thing might be to modify the PCBuild8 build process so the output binaries are in the ../PCBuild' directory - this way distutils and others continue to work fine. Does that sound reasonable?
For 2.6, the first question to answer is: what compiler should it use?
I would personally like to see Python "skip" VS 2005 altogether, as it will be soon superceded by Orcas. Unfortunately, it's unclear how long Microsoft will need to release Orcas (and also, when Python 2.6 will be released), so I would like to defer that question by a few months.
I've no objection to that - but I'd like to help keep the pain to a minimum for people who find themselves trying to build 64bit extensions in the meantime. Anecdotally, VS8 is the compiler most people start trying to use for this (quite possibly because that is what they already have handy).
See above; I disagree. First, "multiple architectures" only means x86, AMD64, and Itanium, and I would like to drop "official" Itanium binaries from 2.6 (even though they could continue to be supported in the build process). Then, even if the Python build itself support multiple simultaneous architectures, the extension modules don't all (correct me if I'm wrong).
Yes, I agree that it is unlikely to work in practice - at least for a number of years as the external libs and extensions catch up.
* Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler versions. VS8 defines this as wchar_t, which I'm struggling to find a formal definition for beyond being 2 bytes.
In C or in C++? In C++, wchar_t is a builtin type, just like short, int, long. So there is no further formal definition.
This was in C++, but the problem was really WCHAR, as used by much of the win32 API.
I'd rather make it a platform-specific definition (for platform=Windows API). Correct me if I'm wrong, but isn't wchar_t also available in VS 2003 (and even in VC6?). And doesn't it have the "right" definition in all these compilers?
hrm - as above, I'm more concerned with the definition of WCHAR - which means my problem is related more to the Platform SDK version rather than the compiler. This is unfortunate - on one hand we do consider 'platform=Windows API', and WCHAR is very much an API concept. I'll need to dig some more into this, but at least I know I'm not wasting my time :)
* Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts?
What compiler would you use there? I very much like using the VS debugger when developing on Windows, so that capability should not go away.
You would use whatever compiler the autoconf toolset found. Recent versions know enough about MSVC for simple projects. Many people would need to take care that their environment pointed at the correct compiler - especially the person building releases. But assuming MSVC was found and had the appropriate switches passed, there would be no impact on the ability to use Visual Studio as a debugging environment. Thanks, Mark
-----Original Message----- This was in C++, but the problem was really WCHAR, as used by much of the win32 API.
I'd rather make it a platform-specific definition (for platform=Windows API). Correct me if I'm wrong, but isn't wchar_t also available in VS 2003 (and even in VC6?). And doesn't it have the "right" definition in all these compilers?
hrm - as above, I'm more concerned with the definition of WCHAR - which means my problem is related more to the Platform SDK version rather than the compiler. This is unfortunate - on one hand we do consider 'platform=Windows API', and WCHAR is very much an API concept. I'll need to dig some more into this, but at least I know I'm not wasting my time :)
Mark, your problem may be related to a setting in the "c/c++ -> language" tab in the settings, where "treat wchar_t as a builtin type" default has changed. I recommend that we do treat it as a builtin, but the VS2003 default was "no" and the 2005 is "yes". Could this be contributing to your problem? Kristjan
Kristján Valur Jónsson quoting me:
hrm - as above, I'm more concerned with the definition of WCHAR - which means my problem is related more to the Platform SDK version rather than the compiler. This is unfortunate - on one hand we do consider 'platform=Windows API', and WCHAR is very much an API concept. I'll need to dig some more into this, but at least I know I'm not wasting my time :)
Mark, your problem may be related to a setting in the "c/c++ -> language" tab in the settings, where "treat wchar_t as a builtin type" default has changed. I recommend that we do treat it as a builtin, but the VS2003 default was "no" and the 2005 is "yes". Could this be contributing to your problem?
Thanks for the suggestion and for introducing me to that option - but it made no difference. I'm guessing its related to C++ - code such as the following: static PyObject *TestIBuild() { // obviously nonsense - the point is to test if it compiles. WCHAR *wval = PyUnicode_AS_UNICODE(Py_None); return PyUnicode_FromUnicode(wval, wcslen(wval)); } works everywhere - except in a pywin32 .cpp file built on x64:) That code results in: win32/src/win32apimodule.cpp(81) : error C2440: 'initializing' : cannot convert from 'Py_UNICODE *' to 'WCHAR *' Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast win32/src/win32apimodule.cpp(82) : error C2664: 'PyUnicodeUCS2_FromUnicode' : cannot convert parameter 1 from 'WCHAR *' to 'const Py_UNICODE *' Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast Unfortunately I don't have pywin32 building under vc8 on x32, but expect it to happen there too. pywin32 uses distutils, but I inspected the options passed and can't find anything to make a difference. /Zc:wchar_t and/or /Zc:wchar_t- seem to be the command-line settings for this flag and it also makes no difference. I'm out of time to confirm is is simply "c++ with vs8", but did confirm that the patch below appears to solve the problem, and given Martin's previous +1, I decided to stop there. I failed in a quick attempt at replacing the literal 2 with something involving sizeof. Does this look reasonable? Cheers, Mark Index: pyconfig.h =================================================================== --- pyconfig.h (revision 55487) +++ pyconfig.h (working copy) @@ -492,10 +492,10 @@ #define Py_USING_UNICODE /* Define as the integral type used for Unicode representation. */ -#define PY_UNICODE_TYPE unsigned short +#define PY_UNICODE_TYPE wchar_t /* Define as the size of the unicode type. */ -#define Py_UNICODE_SIZE SIZEOF_SHORT +#define Py_UNICODE_SIZE 2 /* Define if you have a useable wchar_t type defined in wchar.h; useable means wchar_t must be 16-bit unsigned type. (see
-----Original Message----- below appears to solve the problem, and given Martin's previous +1, I decided to stop there. I failed in a quick attempt at replacing the literal 2 with something involving sizeof. Does this look reasonable?
+1. I should add that we have this local mod in our own patched python. It does seem like a mistake that needs fixing. Kristjan
I'm using the full-blown VS.NET 2003, as given to a number of python-dev people by Microsoft a number of years ago. This appears to come with the SDK and a 64bit compiler.
Not sure what it makes it appear to you that way - it doesn't. VS.NET 2003 is x86 only
I'm guessing vsextcomp doesn't use the Visual Studio 'ReleaseAMD64' configuration - would it be OK for me to check in changes to the PCBuild projects for this configuration?
Please don't. It exclusively relies on vsextcomp, and is only useful if you have that infrastructure installed. See for yourself: it uses the /USE_CL:MS_OPTERON command line switch, which isn't a Microsoft invention (but instead invented by Peter Tröger).
So is there something we can do to make distutils play better with binaries built from PCBuild8, even though it is considered temporary?
In what way better? It already supports them just fine, AFAICT. I guess you are referring to the support for building extensions on top of a source tree "installation". I doubt that is used that often (but I understand you are using it).
It seems the best thing might be to modify the PCBuild8 build process so the output binaries are in the ../PCBuild' directory - this way distutils and others continue to work fine. Does that sound reasonable?
I think Kristjan will have to say a word here: I think he just likes it the way it is right now. That would rather suggest that build_ext needs to be changed.
I've no objection to that - but I'd like to help keep the pain to a minimum for people who find themselves trying to build 64bit extensions in the meantime.
I recommend that those people install the official binaries. Why do you need to build the binaries from source, if all you want is to build extensions?
In C or in C++? In C++, wchar_t is a builtin type, just like short, int, long. So there is no further formal definition.
This was in C++, but the problem was really WCHAR, as used by much of the win32 API.
But in C, WCHAR should not be a problem (and I would like to see explicit source code and explicit compiler error message to be proven wrong).
* Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts? What compiler would you use there? I very much like using the VS debugger when developing on Windows, so that capability should not go away.
You would use whatever compiler the autoconf toolset found. Recent versions know enough about MSVC for simple projects. Many people would need to take care that their environment pointed at the correct compiler - especially the person building releases.
But assuming MSVC was found and had the appropriate switches passed, there would be no impact on the ability to use Visual Studio as a debugging environment.
I doubt that could work in practice. You will have to rewrite everything to make it pass the correct command line switches. Regards, Martin
Hi Martin, > > I'm using the full-blown VS.NET 2003, as given to a number > of python-dev > > people by Microsoft a number of years ago. This appears to > come with the > > SDK and a 64bit compiler. > > Not sure what it makes it appear to you that way - it doesn't. VS.NET > 2003 is x86 only Yes - I was confused by finding an x64 configuration option, and this looking very similar to VC8. My apologies for the confusion. > > So is there something we can do to make distutils play > better with binaries > > built from PCBuild8, even though it is considered temporary? > > In what way better? It already supports them just fine, AFAICT. > > I guess you are referring to the support for building extensions on > top of a source tree "installation". I doubt that is used that often > (but I understand you are using it). Yes, that is correct. I agree it will be rarely used by Python user's, but believe it is a common scenario for people who maintain extensions or libraries, particularly those who want debugging builds. > > It seems the > > best thing might be to modify the PCBuild8 build process so the output > > binaries are in the ../PCBuild' directory - this way distutils and others > > continue to work fine. Does that sound reasonable? > I think Kristjan will have to say a word here: I think he just likes > it the way it is right now. That would rather suggest that build_ext > needs to be changed. So this means PCBuild8 does indeed get formalized into distutils? I'm happy to live with it if it lets me work, but it seems contrary to our emails yesterday. It would also mean the PCBuild8 environment will not work with external build processes that assume the standard layout, but that really isn't something I'm willing to run the pydev gauntlet for at the current time. > > I've no objection to that - but I'd like to help keep the > pain to a minimum > > for people who find themselves trying to build 64bit > extensions in the > > meantime. > > I recommend that those people install the official binaries. > Why do you > need to build the binaries from source, if all you want is to build > extensions? Let's assume that people have a valid reason to build from source - it really is quite common in the open source world. The meta-question then becomes "is it reasonable to expect people be able to build extensions from a tree built with VC8, in the same way they can with VS7?". I think you are suggesting we do want to support this, but I want to be clear. > > >> In C or in C++? In C++, wchar_t is a builtin type, just like > >> short, int, > >> long. So there is no further formal definition. > > > > This was in C++, but the problem was really WCHAR, as used > by much of the > > win32 API. > > But in C, WCHAR should not be a problem (and I would like to see > explicit source code and explicit compiler error message to be > proven wrong). Please see my other mail to Kristjan - at this stage I can only reproduce it in C++ on VC8. C++ on VC7 and C on VC8 both work fine. Please let me know if thde code snippet I pasted or the compiler error aren't suitable. I've stopped further investigation since there seems support for a change that makes the problem go away. > >>> * Finally, as something from left-field which may well take > >>> 12 months or > >>> more to pull off - but would there be any interest to > >>> moving the Windows > >>> build process to a cygwin environment based on the > existing autoconf > >>> scripts? > >> What compiler would you use there? I very much like using the VS > >> debugger when developing on Windows, so that capability should not > >> go away. > > > > You would use whatever compiler the autoconf toolset found. > Recent versions > > know enough about MSVC for simple projects. Many people > would need to take > > care that their environment pointed at the correct compiler > - especially the > > person building releases. > > > > But assuming MSVC was found and had the appropriate > switches passed, there > > would be no impact on the ability to use Visual Studio as a > debugging > > environment. > > I doubt that could work in practice. You will have to rewrite > everything > to make it pass the correct command line switches. 'have to rewrite everything' sounds a little pessimistic, but that's fine with me - consider this idea dropped. Cheers, Mark
Yes, that is correct. I agree it will be rarely used by Python user's, but believe it is a common scenario for people who maintain extensions or libraries, particularly those who want debugging builds.
Ah, debugging builds. It's true that PCbuild does not support them for AMD64, and it's also true that you need such a build if you want the debug CRT. However, I think people ask much too often for a debugging build; in many cases, they could work happily with a release build that supports debugging. Depending on the problem you try to solve, you may or may not need debug information for pythonxy.dll as well, or just for your extension module. I'd like to repeat an offer that I have made several times in the past: if somebody contributes a patch to msi.py which packs all PDB files (and whatever you else you need) into a ZIP file (or whatever else works) from the release build, then I could happily release PDB files along with the actual installer files. (I would not like to release the PDB files *in* the installer files, as I expect they would blow up the msi size significantly).
I think Kristjan will have to say a word here: I think he just likes it the way it is right now. That would rather suggest that build_ext needs to be changed.
So this means PCBuild8 does indeed get formalized into distutils? I'm happy to live with it if it lets me work, but it seems contrary to our emails yesterday.
It would mean that - I'm willing to compromise to make Kristjan happy (he contributed PCbuild8, so he has to decide what changes to it are acceptable and which aren't). A middle ground might be an addition build step (e.g. as a .bat file) which copies all result files also into PCbuild.
Let's assume that people have a valid reason to build from source - it really is quite common in the open source world. The meta-question then becomes "is it reasonable to expect people be able to build extensions from a tree built with VC8, in the same way they can with VS7?". I think you are suggesting we do want to support this, but I want to be clear.
As long as people are willing to maintain it - why not? It's not "official" in the sense that if it breaks, nobody might fix it, but it doesn't hurt to have it (AFAICT). Also, it not being official might mean that we are not obliged to provide backwards compatibility for it, so it can go away without notice (e.g. when/if PCbuild8 is dropped). The same could be done for PC/VC6, if anybody cared to contribute and maintain it. I think there are several other cases in distutils to support special cases (e.g. the DISTUTILS_USE_SDK environment variable); if people want to see their setup supported, AND ARE WILLING TO CONTRIBUTE: more power to them.
Please see my other mail to Kristjan - at this stage I can only reproduce it in C++ on VC8. C++ on VC7 and C on VC8 both work fine. Please let me know if thde code snippet I pasted or the compiler error aren't suitable.
It's fine. I readily believe that it causes problems in C++. Regards, Martin
However, I think people ask much too often for a debugging build; in many cases, they could work happily with a release build that supports debugging. Depending on the problem you try to solve, you may or may not need debug information for pythonxy.dll as well, or just for your extension module.
While that is true in theory, I often find it is not the case in practice, generally due to the optimizer. Depending on what magic the compiler has done, it can be very difficult to set breakpoints (conditional or otherwise), inspect variable values, etc. It is useful in some cases, but very often I find myself falling back to a debug build after attempting to debug a local release build.
I think there are several other cases in distutils to support special cases (e.g. the DISTUTILS_USE_SDK environment variable); if people want to see their setup supported, AND ARE WILLING TO CONTRIBUTE: more power to them.
Yes, but I believe its also important to solicit feedback on the best way to achieve their aims. In this particular case, I believe it would have been misguided for me to simply check in whatever was necessary to have distutils work from the PCBuild8 directory. I hope it is clear that I am willing to contribute the outcome of these discussions... Cheers, Mark
While that is true in theory, I often find it is not the case in practice, generally due to the optimizer. Depending on what magic the compiler has done, it can be very difficult to set breakpoints (conditional or otherwise), inspect variable values, etc. It is useful in some cases, but very often I find myself falling back to a debug build after attempting to debug a local release build.
What I typically do is to disable optimization in a release build. It then essentially becomes a debug build, except that _DEBUG is not defined, so that it doesn't incorporate the debug CRT. This, in turn, means that you can mix such a "debug" build readily with non-debug binaries.
I think there are several other cases in distutils to support special cases (e.g. the DISTUTILS_USE_SDK environment variable); if people want to see their setup supported, AND ARE WILLING TO CONTRIBUTE: more power to them.
Yes, but I believe its also important to solicit feedback on the best way to achieve their aims. In this particular case, I believe it would have been misguided for me to simply check in whatever was necessary to have distutils work from the PCBuild8 directory. I hope it is clear that I am willing to contribute the outcome of these discussions...
Sure - all understood, and all fine. It's certainly good to build consensus first, but in cases of "should we support that borderline case of build setup as long as I'm willing to maintain it", the answer should always be "yes" (*), irrespective of the merits of the specific proposal. Heck, we would even accept patches to support HP-UX better :-) Regards, Martin (*) assuming it doesn't break anything P.S. I think there have been cases where I haven't been that supportive wrt. build issues. However, in this case I really feel "go ahead".
-----Original Message-----
It seems the best thing might be to modify the PCBuild8 build process so the output binaries are in the ../PCBuild' directory - this way distutils and others continue to work fine. Does that sound reasonable?
I think Kristjan will have to say a word here: I think he just likes it the way it is right now. That would rather suggest that build_ext needs to be changed.
Someone mentioned the idea to have a bat file do it. I like that idea. There is already a build.bat file which will build whatever configuration you choose (platform and configuration). Extending it to subsequently copy the resulting binaries up one level is easy. Perhaps this is an acceptable compromise? Kristjan
Someone mentioned the idea to have a bat file do it. I like that idea. There is already a build.bat file which will build whatever configuration you choose (platform and configuration). Extending it to subsequently copy the resulting binaries up one level is easy. Perhaps this is an acceptable compromise?
Sure - that will work for me. I'll check this out and contact you by private mail for further guidance. Cheers, Mark
I recommend that those people install the official binaries. Why do you need to build the binaries from source, if all you want is to build extensions?
I've been following this discussion and it seems like an appropriate place to mention such a scenario which I have encountered myself and am still trying to work around: I have a set of extensions that use SWIG to wrap my own C++ library. This library, on a day-to-day basis is built against VS8 since the rest of our product suite is. Right now I have no way to work with this code using VS8 since the standard distribution is built against VS7 which uses a different CRT. This is an absolute nightmare in practice since I currently have to maintain VS7 projects in parallel with the standard VS8 ones simply so that I can run and test my python code. I've downloaded the Python source and had a look at building up my own distributions for each case (ideally there would be an easy way to separate out Release / Debug products as well as the VS8 / VS7 variants, and I guess potentially for those cross-compiling we'd need to go a step further and do this per arch as well. Anyway, this isn't how it works at the moment, but I'm still searching for a way to be able to work on the python code in VS8. Building using the current projects I seem to get everything in the PCBuild8 / PCBuild dirs. How can I work with what is build? Is there a shell script to build a final distribution tree? If not, is there a simple way to build an MSI similar to the one found on the Python.org site for the official releases but using the PCBuild8 stuff? If not how do you recommend getting myself to a state where I have at least a feature complete distribution build against VS8? I'm happy with a one time build that I can just install into my source tree and upload to the SCM. Thanks in advance, and I hope that my thoughts proved useful in some way. Jamie
I have a set of extensions that use SWIG to wrap my own C++ library. This library, on a day-to-day basis is built against VS8 since the rest of our product suite is. Right now I have no way to work with this code using VS8 since the standard distribution is built against VS7 which uses a different CRT. This is an absolute nightmare in practice since I currently have to maintain VS7 projects in parallel with the standard VS8 ones simply so that I can run and test my python code.
If you know well enough what you are doing, and avoid using unsupported cases, you *can* mix different CRTs.
Building using the current projects I seem to get everything in the PCBuild8 / PCBuild dirs. How can I work with what is build?
Python works correctly out of its build directory. You can just run it from there.
Is there a shell script to build a final distribution tree?
Not sure what you mean by "tree". Distribution is in a single MSI file, not in a tree, and the tree that gets created on the target machine (of the MSI file) never exists on the source machine.
If not, is there a simple way to build an MSI similar to the one found on the Python.org site for the official releases but using the PCBuild8 stuff?
No, but it should be possible to adopt Tools/msi to package pcbuild8 instead. Just make sure you are using different GUIDs as the ProductCode. Regards, Martin
Martin v. Löwis wrote:
I have a set of extensions that use SWIG to wrap my own C++ library. This library, on a day-to-day basis is built against VS8 since the rest of our product suite is. Right now I have no way to work with this code using VS8 since the standard distribution is built against VS7 which uses a different CRT. This is an absolute nightmare in practice since I currently have to maintain VS7 projects in parallel with the standard VS8 ones simply so that I can run and test my python code.
If you know well enough what you are doing, and avoid using unsupported cases, you *can* mix different CRTs.
I can attest to this - I have SWIG-wrapped extensions built with VC6 running quite happily against the official VS7 binaries for Python 2.4. Moving from Python 2.2 to Python 2.4 was a simple matter of recompiling and relinking the modules. The important thing was to make sure to never pass memory ownership or standard lib data structures across the boundary. I haven't actually found this to be all that difficult in practice, as I am typically either copying data from standard library data structures into native Python data structures (e.g. between std::string and PyString) or else merely accessing the Python wrappers around my own C++ classes. In both cases memory ownership remains entirely within the extension module, and all interaction occurs through the Python C API, and never indirectly through the CRT. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
From: Jamie Kirkpatrick [mailto:jkp@kirkconsulting.co.uk] Sent: Wednesday, 23 May 2007 5:16 AM
I have a set of extensions that use SWIG to wrap my own C++ library. This library, on a day-to-day basis is built against VS8 since the rest of our product suite is. Right now I have no way to work with this code using VS8 since the standard distribution is built against VS7 which uses a different CRT. This is an absolute nightmare in practice since I currently have to maintain VS7 projects in parallel with the standard VS8 ones simply so that I can run and test my python code.
I've downloaded the Python source and had a look at building up my own distributions for each case (ideally there would be an easy way to separate out Release / Debug
VS8 / VS7 variants, and I guess potentially for those cross-compiling we'd need to go a step further and do this per arch as well. Anyway, this isn't how it works at
If you are brave and willing to ensure your module doesn't voilate certain constraints (such as never passing a CRT 'concept' - such as a file handle or memory block to be free'd) across mismatched CRT boundaries, you may find that you can happily load your VS8 built pronect with VS7 - but yes, your general point is valid but beyond the scope of this discussion. A separate discussion on making Python "crt agnostic" is almost certainly worthwhile though, but not directly related to this current discussion. products as well as the the moment, but I'm
still searching for a way to be able to work on the python code in VS8. Building using the current projects I seem to get everything in the PCBuild8 / PCBuild dirs. How can I work with what is build?
Is there a shell script to build a final distribution tree? If not, is
way to build an MSI similar to the one found on the Python.org site for
This is *exactly* the point of this thread, and what we are trying to resolve. In the short term, we have agreed a change to PCBuild8\build.bat that copies the build files into PCBuild is a solution that should "work", where "work" is defined as "allow a source tree built with VS8 to operate in the same way, from the POV of building extensions, as one built with VS8." My primary issue with this is solved by the change to the .bat file, but we welcome all feedback from people who believe this is not ideal. I've agreement from Kristjan on the specific change to that .bat file, I'm just yet to check it in (but it literally just copies everything from the PVBuild8 target dir into the PCBuild dir after checking the expected dirs do indeed exist) there a simple the official
releases but using the PCBuild8 stuff?
I believe not. In most cases, people who build from source on Windows will run directly from that source tree, rather than attempting the intermediate step of creating a .msi and installing it.
If not how do you recommend getting myself to a state where I have at least a feature complete distribution build against VS8? I'm happy with a one time build that I can just install into my source tree and upload to the SCM.
I'd suggest that once I check the .bat change in, you build the PCBuild8 directory via that .bat file, then continue to use the 'PCBuild' directory as it it were a VC7 build. Cheers, Mark
Is there a shell script to build a final distribution tree? If not, is there a simple way to build an MSI similar to the one found on the Python.org site for the official releases but using the PCBuild8 stuff?
I believe not.
It's actually not that difficult. You just have to run Tools/msi/msi.py, with a python interpreter that support PythonCOM (i.e. has PythonWin installed). Replacing all occurrences of PC[Bb]uild with PCbuild8 should do the trick. This will give you a "snapshot" MSI, i.e. one that you can install along with a regularly-release Python interpreter. If you put a config.py with testpackage=True into the msi directory, it will register the extensions .px, .pxw, .pxo, .pxc (rather than .py*), and overwrite the registry at Software\xPython (rather than Software\Python), so that the package shouldn't interfere at all with a regular installation. Regards, Martin
[MarkH]
I'm guessing vsextcomp doesn't use the Visual Studio 'ReleaseAMD64' configuration - would it be OK for me to check in changes to the PCBuild projects for this configuration?
[Martin v. Löwis]
Please don't. It exclusively relies on vsextcomp, and is only useful if you have that infrastructure installed. See for yourself: it uses the /USE_CL:MS_OPTERON command line switch, which isn't a Microsoft invention (but instead invented by Peter Tröger).
Aside: it isn't my experience that vsextcomp is necessary to cross-compile for AMD64 and IA64. My cross-build process basically equates to: - Run the appropriate environment setup for the correct compiler. E.g., for the Platform SDK AMD64 compiler and with the current Platform SDK this is: C:\Program Files\Microsoft Platform SDK\SetEnv.Cmd /X64 /RETAIL - Run the solution file with "devenv.com" (IIRC, devenv.exe doesn't take command-line args) and be sure to pass ing "/useenv" to pick up the environment changes. (*) set DEVENV_COM=path/to/devenv.com %DEVENV_COM% PCbuild\pcbuild.sln /useenv /build ReleaseAMD64 (*) Note that for a cross-build the "make_versioninfo" and "make_buildinfo" projects need to be built natively first: "C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\vcvars32.bat" %DEVENV_COM% PCbuild\pcbuild.sln /useenv /build ReleaseAMD64 /project make_versioninfo %DEVENV_COM% PCbuild\pcbuild.sln /useenv /build ReleaseAMD64 /project make_buildinfo This is all VS7.1, though. I don't yet know if VS8 throws a spanner into the works. For VS6 I use "msdev" instead of "devenv.com" and "PC\VC6\pcbuild.dsw" instead of "PCbuild\pcbuild.sln". I haven't looked into what vsextcomp does, so apologies if this is ignorant. Trent -- Trent Mick trentm at activestate.com
- Run the appropriate environment setup for the correct compiler. E.g., for the Platform SDK AMD64 compiler and with the current Platform SDK this is:
C:\Program Files\Microsoft Platform SDK\SetEnv.Cmd /X64 /RETAIL
- Run the solution file with "devenv.com" (IIRC, devenv.exe doesn't take command-line args) and be sure to pass ing "/useenv" to pick up the environment changes. (*)
set DEVENV_COM=path/to/devenv.com %DEVENV_COM% PCbuild\pcbuild.sln /useenv /build ReleaseAMD64
Yes, that should work equally fine.
I haven't looked into what vsextcomp does, so apologies if this is ignorant.
It spares you having to setup the environment; it provides a cl.exe wrapper that locates the SDK from the registry, and then invokes the cl.exe in the SDK if necessary. As a consequence, you can still just double-click the solution file, without having to run devenv.exe/com manually. Regards, Martin
It seems the best thing might be to modify the PCBuild8 build process so the output binaries are in the ../PCBuild' directory - this way distutils and others continue to work fine. Does that sound reasonable?
I think Kristjan will have to say a word here: I think he just likes it the way it is right now. That would rather suggest that build_ext needs to be changed.
I use this patch in ActivePython to get distutils to find the correct PCbuild dir (see attached). Trent -- Trent Mick trentm at activestate.com --- python/Lib/distutils/command/build_ext.py Tue Mar 13 03:19:35 2007 +++ python/Lib/distutils/command/build_ext.py Tue Apr 17 12:51:26 2007 @@ -176,7 +176,16 @@ # Append the source distribution include and library directories, # this allows distutils on windows to work in the source tree self.include_dirs.append(os.path.join(sys.exec_prefix, 'PC')) - self.library_dirs.append(os.path.join(sys.exec_prefix, 'PCBuild')) + from distutils.msvccompiler import get_build_version + msvc_version = get_build_version() + if msvc_version == 6: + self.library_dirs.append(os.path.join(sys.exec_prefix, 'PC', 'VC6')) + elif 6 < msvc_version < 8: + self.library_dirs.append(os.path.join(sys.exec_prefix, 'PCbuild')) + elif msvc_version >= 8: + self.library_dirs.append(os.path.join(sys.exec_prefix, 'PCbuild8')) + else: + log.warn("unexpected MSVC version: %r", msvc_version) # OS/2 (EMX) doesn't support Debug vs Release builds, but has the # import libraries in its "Config" subdirectory
Martin v. Löwis wrote:
I use this patch in ActivePython to get distutils to find the correct PCbuild dir (see attached).
Would you like to commit this to 2.6? (or perhaps 2.5 even?)
Sure, if others think it is a good thing. Will do tomorrow unless I hear a -1 before then. Trent -- Trent Mick trentm at activestate.com
I use this patch in ActivePython to get distutils to find
Martin v. Löwis wrote: the correct
PCbuild dir (see attached).
Would you like to commit this to 2.6? (or perhaps 2.5 even?)
Sure, if others think it is a good thing. Will do tomorrow unless I hear a -1 before then.
I'm not quite a '-1', but am a little confused about where this would leave us. To some extent, this would formalize PCBuild8 and VC6 directories. External tools would then slowly start growing support for these additional directories and the previous benefits of "PCBuild is the canonical location" appear to vanish. Further, I expect that such a patch would confuse any attempts to manually copy from PCBuild8 into PCBuild, for example (ie, some tools knowing about PCBuild8 while others assume PCBuild is likely to get confusing.) Trent: I assume you use the same source tree for multiple platforms and compilers, meaning that changing these "optional" build processes to copy from PCBuild8/VC6 into PCBuild would cause pain? If not, do you think that would be a reasonable solution? Cheers, Mark
I'm not quite a '-1', but am a little confused about where this would leave us. To some extent, this would formalize PCBuild8 and VC6 directories. External tools would then slowly start growing support for these additional directories and the previous benefits of "PCBuild is the canonical location" appear to vanish. Further, I expect that such a patch would confuse any attempts to manually copy from PCBuild8 into PCBuild, for example (ie, some tools knowing about PCBuild8 while others assume PCBuild is likely to get confusing.)
Trent: I assume you use the same source tree for multiple platforms and compilers, meaning that changing these "optional" build processes to copy from PCBuild8/VC6 into PCBuild would cause pain? If not, do you think that would be a reasonable solution?
Changing to have bits always in PCbuild would work for me -- i.e. I *don't* build for multiple compilers/platforms in the same tree. Perhaps that is a better solution -- in the long run, anyway. Having the "bits" always in one dir for whatever the configuration is more akin to the Unix-y configure/make system. Is this something that could work for Python 2.5? Or just 2.6? Long term/aside: Moving to a configure/make build system on Windows, as you proposed in your first email, would be interesting. With MSYS though, not cygwin (a la bsmedberg's new MozillaBuild stuff). I just wish there were an autoconf alternative that wasn't as painful as autoconf. I have a few attempts for my purposes that are written in Python (an obvious bootstrapping problem for building Python itself :). Trent -- Trent Mick trentm at activestate.com
-----Original Message----- though, not cygwin (a la bsmedberg's new MozillaBuild stuff). I just wish there were an autoconf alternative that wasn't as painful as autoconf. I have a few attempts for my purposes that are written in Python (an obvious bootstrapping problem for building Python itself :).
Only for the theorist. As you, we use build tools for our stackless branch written in python. There exist successfully built python versions back from the nineties, which can be considered "external" tools for all practical purposes. Building python with python is really nifty. Kristjan
First of all, I have put some work into pcbuild8 recently and it works well. I am trying to drum up momentum for work on PCBuild8 next europython. See http://wiki.python.org/moin/EuroPython2007Sprints
-----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com@python.org
I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't.
No they don't, but that doesn't mean that you need different checkouts for python, only the others. Anyway, this is indeed something I'd like to see addressed. I don't think we should ditch cross-compilation. It should simplify a lot of stuff, including buildbot setup and so on (if we get the buildbot infrastructure to support it). It is also very cumbersome, if you are working on a project, to have the binaries all end up in the same place. Doing interactive work on python, I frequently compile both the 32 bit and 64 bit versions for testing and it would be downright silly to have to rebuild everything every time.
I would personally like to see Python "skip" VS 2005 altogether, as it will be soon superceded by Orcas. Unfortunately, it's unclear how long Microsoft will need to release Orcas (and also, when Python 2.6 will be released), so I would like to defer that question by a few months. I think this is a bit unrealistic. Here we are in the middle of 2007, VS2005 has just got SP1, and VS2003 is still the "official" compiler. PCBuild8 is ready, it just needs a little bit of extra love and buildbots to make us able to release PGO versions of x86 and x64.
Given the delay for getting even this far, waiting for Orcas and then someone to create PCBuild9, and then getting it up and running and so on will mean waiting another two years.
The MSI packaging would need to be changed, certainly. It currently detects the architecture it needs to package by looking at the file type of python.exe; that would have to be changed to give it an explicit parameter what architecture to package, or have it package all architectures it can find.
I am not familiar with the msi packaging process at all. But here is something we should start to consider: VISTA support. This could mean some of: 1) supplying python.dll as a Side By Side assembly 2) Changing python install locations 3) Supporting shadow libraries, where .pyc files end up in a different hierarchy from the .py files. (useful for many things beside VISTA) 4) Signing the python dlls and executables 5) Providing user level manifests. Vista adoption is going very fast. We see 10% of our users have moved to vista and rising.
I'd rather make it a platform-specific definition (for platform=Windows API). Correct me if I'm wrong, but isn't wchar_t also available in VS 2003 (and even in VC6?). And doesn't it have the "right" definition in all these compilers?
So +1 for setting Py_UNICODE to wchar_t on Windows.
Yes. Btw, in previous visual studio versions, wchar_t was not treated as a builtin type by default, but rather as synonymous with unsighed short. Now the default is that it is, and this causes some semantic differences and incompatibilities of the type seen. Kristjan
On 2007-05-21 12:30, Kristján Valur Jónsson wrote:
[Py_UNICODE being #defined as "unsigned short" on Windows]
I'd rather make it a platform-specific definition (for platform=Windows API). Correct me if I'm wrong, but isn't wchar_t also available in VS 2003 (and even in VC6?). And doesn't it have the "right" definition in all these compilers?
So +1 for setting Py_UNICODE to wchar_t on Windows.
Yes. Btw, in previous visual studio versions, wchar_t was not treated as a builtin type by default, but rather as synonymous with unsighed short. Now the default is that it is, and this causes some semantic differences and incompatibilities of the type seen.
+1 from me. If think this is simply a bug introduced with the UCS4 patches in Python 2.2. unicodeobject.h already has this code: #ifndef PY_UNICODE_TYPE /* Windows has a usable wchar_t type (unless we're using UCS-4) */ # if defined(MS_WIN32) && Py_UNICODE_SIZE == 2 # define HAVE_USABLE_WCHAR_T # define PY_UNICODE_TYPE wchar_t # endif # if defined(Py_UNICODE_WIDE) # define PY_UNICODE_TYPE Py_UCS4 # endif #endif But for some reason, pyconfig.h defines: /* Define as the integral type used for Unicode representation. */ #define PY_UNICODE_TYPE unsigned short /* Define as the size of the unicode type. */ #define Py_UNICODE_SIZE SIZEOF_SHORT /* Define if you have a useable wchar_t type defined in wchar.h; useable means wchar_t must be 16-bit unsigned type. (see Include/unicodeobject.h). */ #if Py_UNICODE_SIZE == 2 #define HAVE_USABLE_WCHAR_T #endif disabling the default settings in the unicodeobject.h. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
Hi Marc-Andre,
+1 from me.
If think this is simply a bug introduced with the UCS4 patches in Python 2.2.
unicodeobject.h already has this code:
#ifndef PY_UNICODE_TYPE
/* Windows has a usable wchar_t type (unless we're using UCS-4) */ # if defined(MS_WIN32) && Py_UNICODE_SIZE == 2 # define HAVE_USABLE_WCHAR_T # define PY_UNICODE_TYPE wchar_t # endif
# if defined(Py_UNICODE_WIDE) # define PY_UNICODE_TYPE Py_UCS4 # endif
#endif
But for some reason, pyconfig.h defines:
/* Define as the integral type used for Unicode representation. */ #define PY_UNICODE_TYPE unsigned short
/* Define as the size of the unicode type. */ #define Py_UNICODE_SIZE SIZEOF_SHORT
/* Define if you have a useable wchar_t type defined in wchar.h; useable means wchar_t must be 16-bit unsigned type. (see Include/unicodeobject.h). */ #if Py_UNICODE_SIZE == 2 #define HAVE_USABLE_WCHAR_T #endif
disabling the default settings in the unicodeobject.h.
Yes, that does appear strange. The following patch works for me, keeps Python building and appears to solve my problem. Any objections? Mark Index: pyconfig.h =================================================================== --- pyconfig.h (revision 55487) +++ pyconfig.h (working copy) @@ -491,22 +491,13 @@ /* Define if you want to have a Unicode type. */ #define Py_USING_UNICODE -/* Define as the integral type used for Unicode representation. */ -#define PY_UNICODE_TYPE unsigned short - /* Define as the size of the unicode type. */ -#define Py_UNICODE_SIZE SIZEOF_SHORT +/* This is enough for unicodeobject.h to do the "right thing" on Windows. */ +#define Py_UNICODE_SIZE 2 -/* Define if you have a useable wchar_t type defined in wchar.h; useable - means wchar_t must be 16-bit unsigned type. (see - Include/unicodeobject.h). */ -#if Py_UNICODE_SIZE == 2 -#define HAVE_USABLE_WCHAR_T - /* Define to indicate that the Python Unicode representation can be passed as-is to Win32 Wide API. */ #define Py_WIN_WIDE_FILENAMES -#endif /* Use Python's own small-block memory-allocator. */ #define WITH_PYMALLOC 1
Hi Mark,
+1 from me.
I think this is simply a bug introduced with the UCS4 patches in Python 2.2.
unicodeobject.h already has this code:
#ifndef PY_UNICODE_TYPE
/* Windows has a usable wchar_t type (unless we're using UCS-4) */ # if defined(MS_WIN32) && Py_UNICODE_SIZE == 2 # define HAVE_USABLE_WCHAR_T # define PY_UNICODE_TYPE wchar_t # endif
# if defined(Py_UNICODE_WIDE) # define PY_UNICODE_TYPE Py_UCS4 # endif
#endif
But for some reason, pyconfig.h defines:
/* Define as the integral type used for Unicode representation. */ #define PY_UNICODE_TYPE unsigned short
/* Define as the size of the unicode type. */ #define Py_UNICODE_SIZE SIZEOF_SHORT
/* Define if you have a useable wchar_t type defined in wchar.h; useable means wchar_t must be 16-bit unsigned type. (see Include/unicodeobject.h). */ #if Py_UNICODE_SIZE == 2 #define HAVE_USABLE_WCHAR_T #endif
disabling the default settings in the unicodeobject.h.
Yes, that does appear strange. The following patch works for me, keeps Python building and appears to solve my problem. Any objections?
Looks fine to me.
Mark
Index: pyconfig.h =================================================================== --- pyconfig.h (revision 55487) +++ pyconfig.h (working copy) @@ -491,22 +491,13 @@ /* Define if you want to have a Unicode type. */ #define Py_USING_UNICODE
-/* Define as the integral type used for Unicode representation. */ -#define PY_UNICODE_TYPE unsigned short - /* Define as the size of the unicode type. */ -#define Py_UNICODE_SIZE SIZEOF_SHORT +/* This is enough for unicodeobject.h to do the "right thing" on Windows. */ +#define Py_UNICODE_SIZE 2
-/* Define if you have a useable wchar_t type defined in wchar.h; useable - means wchar_t must be 16-bit unsigned type. (see - Include/unicodeobject.h). */ -#if Py_UNICODE_SIZE == 2 -#define HAVE_USABLE_WCHAR_T - /* Define to indicate that the Python Unicode representation can be passed as-is to Win32 Wide API. */ #define Py_WIN_WIDE_FILENAMES -#endif
/* Use Python's own small-block memory-allocator. */ #define WITH_PYMALLOC 1
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/mal%40egenix.com
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 22 2007)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
Hi Kristján,
First of all, I have put some work into pcbuild8 recently and it works well.
It does! There are a few issues though, notably with distutils (and as mentioned before, any other tools what may assume PCBuild - see below) You quoting Martin:
I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't.
No they don't, but that doesn't mean that you need different checkouts for python, only the others. Anyway, this is indeed something I'd like to see addressed. I don't think we should ditch cross-compilation.
While I agree with you, Martin's point about the dependant libraries and tools is valid, and may defeat this goal. In the short term, we should research how some of these other projects are approaching x64 - we may also find that this helps with any autoconf work we choose to do. Ultimately, the person releasing the official binaries gets to say how this works (based on how they actually get it to work)
I would personally like to see Python "skip" VS 2005 altogether, as it will be soon superceded by Orcas. Unfortunately, it's unclear how long Microsoft will need to release Orcas (and also, when Python 2.6 will be released), so I would like to defer that question by a few months. I think this is a bit unrealistic. Here we are in the middle of 2007, VS2005 has just got SP1, and VS2003 is still the "official" compiler. PCBuild8 is ready, it just needs a little bit of extra love and buildbots to make us able to release PGO versions of x86 and x64.
Given the delay for getting even this far, waiting for Orcas and then someone to create PCBuild9, and then getting it up and running and so on will mean waiting another two years.
I don't believe there was any suggestion that Python 2.6 would wait for a compiler release from Microsoft. Before we talk about Vista and while I have your attention <wink>, some final questions relating to PCBuild8. Regardless of the ultimate layout for x64, what do you think about having PCBuid8 put the binaries into the PCBuild directory, and thus (theoretically) letting such a directory work with distutils and otherwise as a fully functional Python installation?
I am not familiar with the msi packaging process at all. But here is something we should start to consider: VISTA support. This could mean some of: 1) supplying python.dll as a Side By Side assembly
Yes, this is something pywin32 is going to face. The hack of copying python*.dll into the 'system' directory - necessary for COM - is (sensibly!) no longer working. One thing at a time though...
2) Changing python install locations 3) Supporting shadow libraries, where .pyc files end up in a different hierarchy from the .py files. (useful for many things beside VISTA) 4) Signing the python dlls and executables 5) Providing user level manifests.
And dragging distutils back op topic, having bdist_wininst supply a manifest that indicates escalation is required appears necessary.
Vista adoption is going very fast. We see 10% of our users have moved to vista and rising.
ack - I'm yet to try a 32 bit version, but my Vista-x64 box isn't proving very reliable yet. It *is* very pretty and cute though :) I'm surprised to see 10% unless your users are skewed towards early-adopters though, but I'm in no position to refute it! Cheers, Mark
I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't.
No they don't, but that doesn't mean that you need different checkouts for python, only the others. Anyway, this is indeed something I'd like to see addressed. I don't think we should ditch cross-compilation.
Nobody proposed to ditch cross-compilation. I very much like cross-compilation, I do all Itanium and AMD64 in cross-compilation. I just want the *file structure* of the output to be the very same for all architectures, meaning that they can't coexist in a single source directory.
It should simplify a lot of stuff, including buildbot setup and so on (if we get the buildbot infrastructure to support it). It is also very cumbersome, if you are working on a project, to have the binaries all end up in the same place. Doing interactive work on python, I frequently compile both the 32 bit and 64 bit versions for testing and it would be downright silly to have to rebuild everything every time.
No, you use two checkouts, of course.
I think this is a bit unrealistic. Here we are in the middle of 2007, VS2005 has just got SP1, and VS2003 is still the "official" compiler. PCBuild8 is ready, it just needs a little bit of extra love and buildbots to make us able to release PGO versions of x86 and x64.
No matter what the next compiler is (VS 2005 or VS 2007/2008), it's still *a lot* of work until the VS 2005 build can be used for releasing Python. For example, there is no support for the SxS installation of msvcr8.dll, using the MSI merge module.
Given the delay for getting even this far, waiting for Orcas and then someone to create PCBuild9, and then getting it up and running and so on will mean waiting another two years.
No. I would expect that either the PCbuild or PCbuild8 project files can be used with little changes to build using VS9. I just tried, and it works reasonably well.
I am not familiar with the msi packaging process at all. But here is something we should start to consider: VISTA support. This could mean some of:
Not sure whether anything really is needed. Python works fine on Vista.
1) supplying python.dll as a Side By Side assembly
What would that improve?
2) Changing python install locations
Why?
3) Supporting shadow libraries, where .pyc files end up in a different hierarchy from the .py files. (useful for many things beside VISTA)
Yes, and people had written PEPs for it which failed due to lack of follow up.
4) Signing the python dlls and executables
Hmm.
5) Providing user level manifests.
Vista adoption is going very fast. We see 10% of our users have moved to vista and rising.
Sure, and have they reported problems with Python on Vista (problems specific to Vista?)
Yes. Btw, in previous visual studio versions, wchar_t was not treated as a builtin type by default, but rather as synonymous with unsighed short. Now the default is that it is, and this causes some semantic differences and incompatibilities of the type seen.
C or C++? According to the standards, it ought to be a builtin, primitive type in C++, and a typedef in C. Regards, Martin
-----Original Message----- Nobody proposed to ditch cross-compilation. I very much like cross-compilation, I do all Itanium and AMD64 in cross-compilation.
I just want the *file structure* of the output to be the very same for all architectures, meaning that they can't coexist in a single source directory.
Surely there are differences between architectures? PC uses MSI after all. Why can't linux be under trunk/linux and pc 86 under trunk/pcbuild8/win32PGO and 64 under trunk/pcbuild8/x64pgo? We shouldn't let bad tools keep us from new ways of doing things, rather we should fix and update the tools.
No, you use two checkouts, of course. That´s just silly. And two visual studios open, and edit the file in two places too? I say let's just admit that tools can compile for more than one target. Let's adapt to it and we will all be happier.
No matter what the next compiler is (VS 2005 or VS 2007/2008), it's still *a lot* of work until the VS 2005 build can be used for releasing Python. For example, there is no support for the SxS installation of msvcr8.dll, using the MSI merge module. Well, there is some work, which is why I am proposing the Europython sprint to do it. But we are almost there. And that work will be useful for Orcas, when that comes along. And btw, there is no need to install the msvcr8.dll. We can distribute them as a private assembly. then they (and the manifest) exist in the same directory as python2x.dll. This is a supported distribution mode and how we distribute the runtime with EVE.
Not sure whether anything really is needed. Python works fine on Vista. If you are an administrator. A limited user will have problems installing it and then running it.
1) supplying python.dll as a Side By Side assembly
What would that improve? Well, it should reduce dll-hell problems of applications that ship with python2x.dll. You ship with and link to your own and tested dll. We have some concerns here, for example, now that we are moving away from embedding python in our blue.dll and using python25.dll directly, that this exposes a vulnerability to the integrity of the software.
2) Changing python install locations To conform with Windows rules and get a "Vista approved" logo. Install in the ProgramFiles folder. Just as C does. Ah, and this also means that we could install both 32 bit and 64 bit versions, another plus.
3) Supporting shadow libraries, where .pyc files end up in a different hierarchy from the .py files. (useful for many things beside VISTA)
Yes, and people had written PEPs for it which failed due to lack of follow up. Interesting. We are definitely interested in that. You see, Someone installs a game or accounting software using vista. He then runs as a limited user. Python insists on saving its .pyc files in the installation folder, but this is not something that is permitted on Vista.
4) Signing the python dlls and executables Hmm.
Again, easing the installation experience for vista users. So that they don't get a red box about an unknown application requiring administrator privileges.
Sure, and have they reported problems with Python on Vista (problems specific to Vista?)
Certainly. We are working on them, of course. Chiefly, we have had to change where we save all kinds of temporary files. And we have to be careful that all our .py files ship as .pyc files. And we are versioning, and signing all executables, providing user level manifests and fixing up the install processes to be more compliant. I am just saying that this is something that a standard python distro might want to do too. Kristjan
Surely there are differences between architectures? PC uses MSI after all. Why can't linux be under trunk/linux and pc 86 under trunk/pcbuild8/win32PGO and 64 under trunk/pcbuild8/x64pgo?
That couldn't work for me. I try avoid building on a network drive, and with local drives, I just can't have a Windows build and a Linux build on the same checkout - they live on separate file systems, after all (Linux on ext3, Windows on NTFS, with multi-boot switching between them).
That´s just silly. And two visual studios open, and edit the file in two places too?
I have about 10 checkouts of Python, on different machines, with no problems. I don't feel silly doing so. I don't *use* them simultaneously, of course - I cannot work on two architectures simultaneously, anyway.
I say let's just admit that tools can compile for more than one target. Let's adapt to it and we will all be happier.
You might be; I will be sad. It comes for a price, and with little benefit. Disk space is cheaper than my time to fight build processes.
And btw, there is no need to install the msvcr8.dll. We can distribute them as a private assembly. then they (and the manifest) exist in the same directory as python2x.dll.
Yes, but then python2x.dll goes into system32, and so will msvcr8.dll, no?
Not sure whether anything really is needed. Python works fine on Vista. If you are an administrator. A limited user will have problems installing it and then running it.
Is there a bug report for that?
1) supplying python.dll as a Side By Side assembly What would that improve? Well, it should reduce dll-hell problems of applications that ship with python2x.dll. You ship with and link to your own and tested dll. We have some concerns here, for example, now that we are moving away from embedding python in our blue.dll and using python25.dll directly, that this exposes a vulnerability to the integrity of the software.
Why should there be versioning problems with python25.dll? Are there any past issues with incompatibilities with any python2x.dll release?
2) Changing python install locations To conform with Windows rules and get a "Vista approved" logo. Install in the ProgramFiles folder.
Only over my dead body. *This* is silly.
Just as C does. Ah, and this also means that we could install both 32 bit and 64 bit versions, another plus.
What about the registry?
Interesting. We are definitely interested in that. You see, Someone installs a game or accounting software using vista. He then runs as a limited user. Python insists on saving its .pyc files in the installation folder, but this is not something that is permitted on Vista.
But that's not a problem, is it? Writing silently "fails", i.e. it just won't save the pyc files. Happens all the time on Unix.
Sure, and have they reported problems with Python on Vista (problems specific to Vista?) Certainly. We are working on them, of course.
But, of course, they have not been reported. Regards, Martin
-----Original Message----- From: "Martin v. Löwis" [mailto:martin@v.loewis.de]
That couldn't work for me. I try avoid building on a network drive, and with local drives, I just can't have a Windows build and a Linux build on the same checkout - they live on separate file systems, after all (Linux on ext3, Windows on NTFS, with multi-boot switching between them). Very well, leaving linux aside, I don't see why this: /win32mount/trunk/PCbuild/ /x64mount/trunk/PCbuild/
Is any different from /winmount/trunk/PCBuild/win32 /winmount/trunk/PCBuild/x64 I don't understand this extraordinary reluctance to add a single extra directory. The windows build process is different from any other build process, so even If all the other platforms live one directory higher, why must windows?
I don't *use* them simultaneously, of course - I cannot work on two architectures simultaneously, anyway. I do so on a daily basis. I designed PCBuild8 to be easy for interactive work using VisualStudio, using property sheets and such to group common settings for easy editing. During the course of my work, I will edit a file (which is checked out from Perforce), compile debug for two platforms, test, and repeat.
I say let's just admit that tools can compile for more than one target. Let's adapt to it and we will all be happier.
You might be; I will be sad. It comes for a price,
And btw, there is no need to install the msvcr8.dll. We can distribute them as a private assembly. then they (and the manifest) exist in the same directory as python2x.dll.
Yes, but then python2x.dll goes into system32, and so will msvcr8.dll, no? Yes, that is correct. Well, there is a CRT .exe redist if you want to deploy
I well understand the benefits, I use it all the time, but the price still eludes me. how can a different name for the output folder for a different platform be such a big problem? this into the SxS cache, it just has to be run as part of the install process. But that may or may not be problematic, I don't know.
Not sure whether anything really is needed. Python works fine on Vista. If you are an administrator. A limited user will have problems installing it and then running it.
Is there a bug report for that?
I don't know. At any rate, I think any vista issues is a completely separate thing, something that needs to be handled as a whole, rather than responding to a particular problem reported in a bug report.
1) supplying python.dll as a Side By Side assembly What would that improve? Well, it should reduce dll-hell problems of applications that ship with python2x.dll. You ship with and link to your own and tested dll. We have some concerns here, for example, now that we are moving away from embedding python in our blue.dll and using python25.dll directly, that this exposes a vulnerability to the integrity of the software.
Why should there be versioning problems with python25.dll? Are there any past issues with incompatibilities with any python2x.dll release?
Someone could replace the python25.dll that we ship with their own patched version, thereby gaining backdoor access to the software. The way windows searches for old style dlls makes this easy. Using the SxS signed loading scheme, you can protect yourself up to a point from such attacks. Of course, this doesn't have to be a problem for vanilla python, we can do this on our own for the patched python25 that we employ, but it still might be something others could find useful.
Install in the ProgramFiles folder.
Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide. I mean we do want to play nice, don't we? Nothing installs itself in the root anymore, not since windows 3.1
Just as C does. Ah, and this also means that we could install both 32 bit and 64 bit versions, another plus.
What about the registry?
I don't know about the registry, what is it used for? 64 bit windows already ships with dual versions of some apps, notably explorer.exe so that shouldn't be a big problem.
Interesting. We are definitely interested in that. You see, Someone installs a game or accounting software using vista. He then runs as a limited user. Python insists on saving its .pyc files in the installation folder, but this is not something that is permitted on Vista.
But that's not a problem, is it? Writing silently "fails", i.e. it just won't save the pyc files. Happens all the time on Unix.
It may not silently fail, depending on your user status. An admin might get a confirmation window, for example.
Sure, and have they reported problems with Python on Vista (problems specific to Vista?) Certainly. We are working on them, of course.
But, of course, they have not been reported.
These are not python errors as such, but rather EVE errors. We ship the .py files precompiled and therefore avoid the aforementioned problems. But we have had to move all sorts of temporary files out of "program files" and into "documents and settings/user/local settings/Application Data/CCP/Eve Kristjan
On 5/23/07, Kristján Valur Jónsson <kristjan@ccpgames.com> wrote:
Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide. I mean we do want to play nice, don't we? Nothing installs itself in the root anymore, not since windows 3.1
Maybe installing in the root is not good, but installing to "Program Files" is just asking for trouble. All sorts of development tools might suddenly break because of that space in the middle of the path and requirement to use quotes around it. I thus usually install things to <drive>:\Programs. I'm not sure if any packages/programs will break because of that space, but what if some will?
-----Original Message----- From: Alexey Borzenkov [mailto:snaury@gmail.com] Sent: Wednesday, May 23, 2007 20:36 To: Kristján Valur Jónsson Cc: Martin v. Löwis; Mark Hammond; distutils-sig@python.org; python- dev@python.org Subject: Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
On 5/23/07, Kristján Valur Jónsson <kristjan@ccpgames.com> wrote:
Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide. I mean we do want to play nice, don't we? Nothing installs itself in the root anymore, not since windows 3.1
Maybe installing in the root is not good, but installing to "Program Files" is just asking for trouble. All sorts of development tools might suddenly break because of that space in the middle of the path and requirement to use quotes around it. I thus usually install things to <drive>:\Programs. I'm not sure if any packages/programs will break because of that space, but what if some will?
Development tools used on windows already have to cope with this. Spaces are not going away, so why not bite the bullet and deal with them? Moving forward sometimes means crossing rivers. As for Vista issues, I'll gather more data before making any more claims, but I think that it is important that we play by the rules here. Just imagine the a school teacher who in good faith wants to introduce his pupils to the wonderful programming language of Python, but when he installs it, all kinds of scary looking warnings drive him off. Vista is, like it or not, going to be very prevalent. If we want python to be easily accessible to the masses, we mustn't take an elitist attitude or else risk scaring people off. Finally, to add a (mis)quote from Mr. Gorbachev: "Wer zu spät kommt, den bestraft das Leben" Kristján
At 12:20 PM +0000 5/26/07, Kristján Valur Jónsson wrote:
-----Original Message----- From: Alexey Borzenkov [mailto:snaury@gmail.com] Sent: Wednesday, May 23, 2007 20:36 To: Kristján Valur Jónsson Cc: Martin v. Löwis; Mark Hammond; distutils-sig@python.org; python- dev@python.org Subject: Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
On 5/23/07, Kristján Valur Jónsson <kristjan@ccpgames.com> wrote:
Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide. I mean we do want to play nice, don't we? Nothing installs itself in the root anymore, not since windows 3.1
Maybe installing in the root is not good, but installing to "Program Files" is just asking for trouble. All sorts of development tools might suddenly break because of that space in the middle of the path and requirement to use quotes around it. I thus usually install things to <drive>:\Programs. I'm not sure if any packages/programs will break because of that space, but what if some will?
Development tools used on windows already have to cope with this. Spaces are not going away, so why not bite the bullet and deal with them? Moving forward sometimes means crossing rivers. ...
Microsoft's command line cannot cope with two pathnames that must be quoted, so if the command path itself must be quoted, then no argument to the command can be quoted. There are tricky hacks that can work around this mind-boggling stupidity, but life is simpler if Python itself doesn't use up the one quoted pathname. I don't know if Microsoft has had the good sense to fix this in Vista (which I probably will never use, since an alternative exists), but they didn't in XP. -- ____________________________________________________________________ TonyN.:' <mailto:tonynelson@georgeanelson.com> ' <http://www.georgeanelson.com/>
-----Original Message-----
Microsoft's command line cannot cope with two pathnames that must be quoted, so if the command path itself must be quoted, then no argument to the command can be quoted. There are tricky hacks that can work around this mind-boggling stupidity, but life is simpler if Python itself doesn't use up the one quoted pathname. I don't know if Microsoft has had the good sense to fix this in Vista (which I probably will never use, since an alternative exists), but they didn't in XP.
Do you have any references for this claim? In my command line on XP sp2, this works just fine: C:\Program Files\Microsoft Visual Studio 8\VC>"c:\Program Files\TextPad 4\TextPad.exe" "c:\tmp\f a.txt" "c:\tmp\f b.txt" Both the program, and the two file names are quoted and textpad.exe opens them both. Cheers, Kristján
At 1:14 PM +0000 5/29/07, Kristján Valur Jónsson wrote:
-----Original Message-----
Microsoft's command line cannot cope with two pathnames that must be quoted, so if the command path itself must be quoted, then no argument to the command can be quoted. There are tricky hacks that can work around this mind-boggling stupidity, but life is simpler if Python itself doesn't use up the one quoted pathname. I don't know if Microsoft has had the good sense to fix this in Vista (which I probably will never use, since an alternative exists), but they didn't in XP.
Do you have any references for this claim? In my command line on XP sp2, this works just fine:
C:\Program Files\Microsoft Visual Studio 8\VC>"c:\Program Files\TextPad 4\TextPad.exe" "c:\tmp\f a.txt" "c:\tmp\f b.txt"
Both the program, and the two file names are quoted and textpad.exe opens them both.
I pounded my head against this issue when working on a .bat file a few years back, until I read the help for cmd and saw the quote logic (and switched to VBScript). It's still there, in "help cmd". I had once found references to the same issue for the run command in Microsoft's online help. Perhaps it is fixed in SP2. If so, just change it and don't worry about users with earlier versions of Windows. -- ____________________________________________________________________ TonyN.:' <mailto:tonynelson@georgeanelson.com> ' <http://www.georgeanelson.com/>
Development tools used on windows already have to cope with this. Spaces are not going away, so why not bite the bullet and deal with them? Moving forward sometimes means crossing rivers.
But in a safe path, step by step. People continue to report problems with spaces in file names, even though many of them have been solved also.
As for Vista issues, I'll gather more data before making any more claims, but I think that it is important that we play by the rules here.
I completely disagree. It is important that it "works", not that we play by the rules.
Just imagine the a school teacher who in good faith wants to introduce his pupils to the wonderful programming language of Python, but when he installs it, all kinds of scary looking warnings drive him off. Vista is, like it or not, going to be very prevalent. If we want python to be easily accessible to the masses, we mustn't take an elitist attitude or else risk scaring people off.
I'm completely in favor of fixing actual bugs. However, I'm not aware of any (related to Vista). That it is not logo compliant is *not* a bug. Python hasn't been logo compliant for more than a decade now (the "install to Program Files" is not a new requirement, but existed since Win95). Regards, Martin
Just imagine the a school teacher who in good faith wants to introduce his pupils to the wonderful programming language of Python, but when he installs it, all kinds of scary looking warnings drive him off. Vista is, like it or not, going to be very prevalent. If we want
-----Original Message----- From: "Martin v. Löwis" [mailto:martin@v.loewis.de] python
to be easily accessible to the masses, we mustn't take an elitist attitude or else risk scaring people off.
I'm completely in favor of fixing actual bugs. However, I'm not aware of any (related to Vista). That it is not logo compliant is *not* a bug. Python hasn't been logo compliant for more than a decade now (the "install to Program Files" is not a new requirement, but existed since Win95).
I'm not saying that it is a bug, but since this is python-dev, I am discussing it as a desirable "feature". One feature that is easily addable and will certainly make installing python on vista nicer, is to add authenticode signing to the install. Currently the user is faced with a very nasty and off-putting message about an unidentified program requesting access to his computer. See http://msdn2.microsoft.com/en-us/library/bb172338.aspx . I think the PSF should be able to obtain a certificate from MS. cheers, Kristján
One feature that is easily addable and will certainly make installing python on vista nicer, is to add authenticode signing to the install.
This I question very much. I experimented with authenticode before 2.4, and found it an unacceptable experience. When the MSI file starts running, installer needs to verify the signature, for which it needs to compute a hash of the entire file. For the Python MSI, this takes many seconds on a slower Pentium 4 machine. During that time, there is no visual feedback, so users are uncertain whether they have actually invoked the MSI file at all.
Currently the user is faced with a very nasty and off-putting message about an unidentified program requesting access to his computer.
Certainly. However, telling them that they have to wait just so that Windows finds out what they know already (that this is the MSI file from the Python Software Foundation, or from Martin v. Löwis) is even more nasty. Regards, Martin
Very well, leaving linux aside, I don't see why this: /win32mount/trunk/PCbuild/ /x64mount/trunk/PCbuild/
Is any different from /winmount/trunk/PCBuild/win32 /winmount/trunk/PCBuild/x64
I don't understand this extraordinary reluctance to add a single extra directory. The windows build process is different from any other build process, so even If all the other platforms live one directory higher, why must windows?
It doesn't need to, and reluctance is not wrt. to the proposed new layout, but wrt. changing the current one. Tons of infrastructure depends on the files having exactly the names that they have now, and being located in exactly the locations where they are currently located. Any change to that, whatever minor, will cause problems to some people. So for the change to be made, the advantages must clearly outweigh the incompatibility of the change in the first place.
I say let's just admit that tools can compile for more than one target. Let's adapt to it and we will all be happier. You might be; I will be sad. It comes for a price, I well understand the benefits, I use it all the time, but the price still eludes me. how can a different name for the output folder for a different platform be such a big problem?
When running Python now, I type (after having changed to the source directory in the cmd.exe window) PCb<tab>pyt<tab><tab> or some such. To navigate to a subdirectory, I need many more keystrokes. I find it very painful to invoke python.exe from the PCbuild8 directory. I don't want that pain to be the default.
Yes, that is correct. Well, there is a CRT .exe redist if you want to deploy this into the SxS cache, it just has to be run as part of the install process. But that may or may not be problematic, I don't know.
Microsoft recommends to use the merge module (.msm), and I think this is what should be done (if feasible).
Why should there be versioning problems with python25.dll? Are there any past issues with incompatibilities with any python2x.dll release? Someone could replace the python25.dll that we ship with their own patched version, thereby gaining backdoor access to the software. The way windows searches for old style dlls makes this easy. Using the SxS signed loading scheme, you can protect yourself up to a point from such attacks. Of course, this doesn't have to be a problem for vanilla python, we can do this on our own for the patched python25 that we employ, but it still might be something others could find useful.
I personally think that if hostile users can replace DLLs on your system, you have bigger problems than SxS installation of pythhonxy.dll, but perhaps that's just me.
Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide.
He can decide not to give Python the "Vista ready" logo. If so, I don't want it. He cannot decide to make the Python installer to install into a directory with a space in its name by default.
I mean we do want to play nice, don't we?
Nice to users of Python, sure. I would not have said a word if the standard directory would have been, say "\usr\bin". However, they happened to chose "Program Files", making it language dependent, and putting a space in the name. That space has caused numerous problems to Python scripts, and I expect changing the default would cause a lot of problems to end users.
What about the registry? I don't know about the registry, what is it used for?
For two things, with different importance to different users: 1. File extensions are registered there, e.g. .py and .pyc. With two binaries installed, the will stomp over each other's file associations; only one of them can win. 2. Python installs keys under HL{LM|CU}\Software\Python\PythonCore\<version>, namely InstallPath InstallGroup PythonPath Documentation Modules For some of these, add-on libraries and applications may modify these keys, and the interpreter will pick up the changes. Again, there can be only one installation on the system "owning" these settings; two simultaneous installations will stomp on each other's settings.
64 bit windows already ships with dual versions of some apps, notably explorer.exe so that shouldn't be a big problem.
The two versions of MSIE actually *are* a big problem, that's why MS only runs the 32-bit IE, even on Win64 (otherwise, ActiveX controls downloaded from the net wouldn't work). Also, while they are both shipped, you can't the two versions of explorer.exe simultaneously (without trickery), so its far from simple.
But that's not a problem, is it? Writing silently "fails", i.e. it just won't save the pyc files. Happens all the time on Unix. It may not silently fail, depending on your user status. An admin might get a confirmation window, for example.
Can you describe the precise scenario which makes that happen? To my knowledge, Vista will *not* open a confirmation window when Python attempts to write a .pyc file. Regards, Martin
-----Original Message----- From: "Martin v. Löwis" [mailto:martin@v.loewis.de]
It doesn't need to, and reluctance is not wrt. to the proposed new layout, but wrt. changing the current one. Tons of infrastructure depends on the files having exactly the names that they have now, and being located in exactly the locations where they are currently located. Any change to that, whatever minor, will cause problems to some people.
Just to be absolutely clear: You are talking about the build environment, right? Because I am not proposing to change any layout of the installed Python (wherever that may be :) I am baffled about why the build environment's layout matters, but once an .msi install can place the binaries in any old place it wants. The build structure doesn't have to reflect the final installed structure at all. Kristján
It doesn't need to, and reluctance is not wrt. to the proposed new layout, but wrt. changing the current one. Tons of infrastructure depends on the files having exactly the names that they have now, and being located in exactly the locations where they are currently located. Any change to that, whatever minor, will cause problems to some people.
Just to be absolutely clear: You are talking about the build environment, right? Because I am not proposing to change any layout of the installed Python (wherever that may be :)
Correct.
I am baffled about why the build environment's layout matters, but once an .msi install can place the binaries in any old place it wants. The build structure doesn't have to reflect the final installed structure at all.
No. But still, people like to be able to "run" Python out of a source check-out. This has been supported for a long time, and more and more stuff was added to support it. For examples within Python itself, see the support in distutils, getpathp.c, PCbuild/rt.bat, and Tools/buildbot/*.bat. Reportedly (by Mark), building Mozilla (the web browser) also "knows" about PCbuild. Regards, Martin
-----Original Message-----
I personally think that if hostile users can replace DLLs on your system, you have bigger problems than SxS installation of pythhonxy.dll, but perhaps that's just me. An end user application on an end-user's machine is always voulnerable to reverse engineering. But it helps to make it more difficult. The old-style dll loader is a particular vulnerability which makes it easy to patch into an application at load time.
What about the registry? I don't know about the registry, what is it used for?
For two things, with different importance to different users: 1. File extensions are registered there, e.g. .py and .pyc. With two binaries installed, the will stomp over each other's file associations; only one of them can win. Sure. No argument about this. But as with the explorer and other apps, it is perfectly possible to manually start one or the other, autoclicking on .py files isn't the only option.
2. Python installs keys under HL{LM|CU}\Software\Python\PythonCore\<version>, namely InstallPath InstallGroup PythonPath Documentation Modules
Funnily enough, Bill has thought of this too. See http://support.microsoft.com/kb/305097/ for info. Co-existence of 32 and 64 bit apps is supported.
The two versions of MSIE actually *are* a big problem, that's why MS only runs the 32-bit IE, even on Win64 (otherwise, ActiveX controls downloaded from the net wouldn't work).
Also, while they are both shipped, you can't the two versions of explorer.exe simultaneously (without trickery), so its far from simple.
The two versions aren't the problem, it is the backward support for 32 bit active thingies that are, as you point out. There is confusion here: internet explorer shipped in both versions. The 32 bit version war default for the above reason. But explorer.exe (which I was talking about) also had two versions. The 64 bit version ran by default. You may recall that before tortoise shipped a 64 bit version, one had to kill explorer.exe and restart it (explorer32.exe IIRC) to get tortoise to work. Supporting both kinds (country and western) on the same machine might be helpful to people for this very reason. A lot of legacy modules are only avaible in 32 bit mode. But people may want to do contemporary development using the new 64 bit mode. Cheers, Kristján
Supporting both kinds (country and western) on the same machine might be helpful to people for this very reason. A lot of legacy modules are only avaible in 32 bit mode. But people may want to do contemporary development using the new 64 bit mode.
Of course, people who really want that can install both versions simultaneously today. Regards, Martin
Very well, leaving linux aside, I don't see why this: /win32mount/trunk/PCbuild/ /x64mount/trunk/PCbuild/
Is any different from /winmount/trunk/PCBuild/win32 /winmount/trunk/PCBuild/x64
In the former case, assuming python is running from the 'trunk' directory, all architectures know how to locate the binary directory - it is always named 'PCBuild'. In the latter case, the directory name depends on the architecture. A number of existing tools already know about the 'PCBuild' directory, so these tools will not need to be taught anything new for x64.
I don't understand this extraordinary reluctance to add a single extra directory.
I think that this thread has enumerated the concerns fairly well. You may not agree with them, but if you don't understand them it might be worth re-reading Martin's responses. Note that I also understand your concerns and goals - I certainly see where you are coming from - but we have 2 competing goals - "work with as many existing, external build tools as possible" versus "take this opportunity to create a new cross-compile-capable x64 build environment for Windows and let those external tools deal with the breakage". As I mentioned in a previous email, my personal opinion would be swayed by looking externally. Specifically, if we could determine the likelihood of external build processes (eg, mozilla) working unchanged if we stick with 'PCBuild', and if we could determine the cross-compilation strategy being adopted by the external libs we use (zlib, bsddb, etc), I think we could make an informed decision.
You might be; I will be sad. It comes for a price, I well understand the benefits, I use it all the time, but the price still eludes me. how can a different name for the output folder for a different platform be such a big problem?
Please see above - its not a problem if you think of the PCBuild8 process as the "last step" in a build process - but often it is not. External tools that use Python (ie, things you try and build after the Python build has completed) are impacted. I understand that you might not use such tools, but they do exist.
Why should there be versioning problems with python25.dll? Are there any past issues with incompatibilities with any python2x.dll release? Someone could replace the python25.dll that we ship with their own patched version, thereby gaining backdoor access to the software. The way windows searches for old style dlls makes this easy. Using the SxS signed loading scheme, you can protect yourself up to a point from such attacks.
I'm not sure I buy this. If someone has enough access to your machine to change pythonxx.dll, you are pretty screwed already.
What about the registry? I don't know about the registry, what is it used for?
Please see PC/getpathp.c in Python source tree. However, I agree that there are a number of things we could do to help Python play nicely on Vista. It might help if we can enumerate the specific problems and potential solutions in a more formal way (eg, a Python bug) Cheers, Mark
Kristján Valur Jónsson wrote:
First of all, I have put some work into pcbuild8 recently and it works well. I am trying to drum up momentum for work on PCBuild8 next europython. See http://wiki.python.org/moin/EuroPython2007Sprints
-----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com@python.org
I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't.
No they don't, but that doesn't mean that you need different checkouts for python, only the others. Anyway, this is indeed something I'd like to see addressed. I don't think we should ditch cross-compilation. It should simplify a lot of stuff, including buildbot setup and so on (if we get the buildbot infrastructure to support it). It is also very cumbersome, if you are working on a project, to have the binaries all end up in the same place. Doing interactive work on python, I frequently compile both the 32 bit and 64 bit versions for testing and it would be downright silly to have to rebuild everything every time.
I would personally like to see Python "skip" VS 2005 altogether, as it will be soon superceded by Orcas. Unfortunately, it's unclear how long Microsoft will need to release Orcas (and also, when Python 2.6 will be released), so I would like to defer that question by a few months. I think this is a bit unrealistic. Here we are in the middle of 2007, VS2005 has just got SP1, and VS2003 is still the "official" compiler. PCBuild8 is ready, it just needs a little bit of extra love and buildbots to make us able to release PGO versions of x86 and x64.
Given the delay for getting even this far, waiting for Orcas and then someone to create PCBuild9, and then getting it up and running and so on will mean waiting another two years.
Addressing only the issues of PCBuild8 and 64-bit architectures, I have tried to establish "free" buildbot support for the 64-bit architectures without any real success. Should the PSF be considering paying for infrastructure that will support 64-bit build reporting?
[...]
regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden ------------------ Asciimercial --------------------- Get on the web: Blog, lens and tag your way to fame!! holdenweb.blogspot.com squidoo.com/pythonology tagged items: del.icio.us/steve.holden/python All these services currently offer free registration! -------------- Thank You for Reading ----------------
Addressing only the issues of PCBuild8 and 64-bit architectures, I have tried to establish "free" buildbot support for the 64-bit architectures without any real success.
Should the PSF be considering paying for infrastructure that will support 64-bit build reporting?
You can bring it up to the board, of course. However, given that all other buildbot machines were contributed by volunteers, the fact that nobody volunteers to contribute buildbot machines for PCbuild8 indicates that there is not a lot of interest in that build infrastructure. Note that there *are* 64-bit buildbot slaves, e.g. for AMD64 (contributed by Neal Norwitz), Alpha (contributed by Neal Norwitz), Itanium (contributed by Matthias Klose, offering a machine from the Debian project), and PPC64 (likewise). These machines all run Linux or Tru64, and (to my understanding) serve other purposes as well, making the buildbot slave just a minor detail. It's an unfortunate fact of life that Microsoft Windows does not support multi-tasking multi-user workloads that well, so the Windows buildbot slave are (to my knowledge) typically dedicated machines (except for Tim's machine, which is just disconnected from the master when he doesn't feel like running buildbot jobs). Regards, Martin
On Tue, May 22, 2007 at 07:32:42AM +0200, "Martin v. L?wis" wrote: -> > Addressing only the issues of PCBuild8 and 64-bit architectures, I -> > have tried to establish "free" buildbot support for the 64-bit -> > architectures without any real success. -> > -> > Should the PSF be considering paying for infrastructure that will -> > support 64-bit build reporting? -> -> You can bring it up to the board, of course. However, given that -> all other buildbot machines were contributed by volunteers, the -> fact that nobody volunteers to contribute buildbot machines for -> PCbuild8 indicates that there is not a lot of interest in that -> build infrastructure. -> -> Note that there *are* 64-bit buildbot slaves, e.g. for AMD64 -> (contributed by Neal Norwitz), Alpha (contributed by Neal -> Norwitz), Itanium (contributed by Matthias Klose, offering -> a machine from the Debian project), and PPC64 (likewise). -> -> These machines all run Linux or Tru64, and (to my understanding) -> serve other purposes as well, making the buildbot slave -> just a minor detail. -> -> It's an unfortunate fact of life that Microsoft Windows does not -> support multi-tasking multi-user workloads that well, so the -> Windows buildbot slave are (to my knowledge) typically dedicated -> machines (except for Tim's machine, which is just disconnected -> from the master when he doesn't feel like running buildbot -> jobs). I haven't really been listening to this conversation, so forgive me if this isn't relevant, but: I'd be happy to install Windows and the latest VisualStudio on a 64-bit VMware image. I just can't be responsible for day-to-day administration of the buildslave; Windows requires too much attention for me to do that. cheers, --titus
I'd be happy to install Windows and the latest VisualStudio on a 64-bit VMware image. I just can't be responsible for day-to-day administration of the buildslave; Windows requires too much attention for me to do that.
Thanks for the offer. Perhaps Kristjan is interested in setting up a buildslave on such an installation. Regards, Martin
-----Original Message----- From: "Martin v. Löwis" [mailto:martin@v.loewis.de] Sent: Tuesday, May 22, 2007 05:33 To: Steve Holden Cc: Kristján Valur Jónsson; Mark Hammond; distutils-sig@python.org; python-dev@python.org Subject: Re: Adventures with x64, VS7 and VS8 on Windows
Addressing only the issues of PCBuild8 and 64-bit architectures, I have tried to establish "free" buildbot support for the 64-bit architectures without any real success.
Should the PSF be considering paying for infrastructure that will support 64-bit build reporting?
You can bring it up to the board, of course. However, given that all other buildbot machines were contributed by volunteers, the fact that nobody volunteers to contribute buildbot machines for PCbuild8 indicates that there is not a lot of interest in that build infrastructure. Most people just install whatever it is that they are offered to download. For me, the most compelling reason to provide the new builds is
the 15% performance benefit. If there are no technical and corporate difficulties, such as firewalls and security, I am sure that CCP can provide a VisualStudio2005 buildbot for our needs. Wasn't there some issue that each buildbot can only provide a single build? Here is a place where cross-compilation comes into its own again. Kristjan
If there are no technical and corporate difficulties, such as firewalls and security, I am sure that CCP can provide a VisualStudio2005 buildbot for our needs. Wasn't there some issue that each buildbot can only provide a single build?
Yes, but you can have multiple buildbot slaves on a single machine. They all create separate checkouts. Regards, Martin
Hi Martin, Way back on Monday, 21 May 2007, you wrote:
This is an issue to be discussed for Python 2.6. I'm personally hesitant to have the "official" build infrastructure deviate from the layout that has been in-use for so many years, as a lot of things depend on it.
I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't. ...
* Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to generate x64 binaries to their own directory - my expectation is that cross-compiling between platforms is a reasonable use-case, and we should support multiple achitectures for the same compiler version.
See above; I disagree.
[For additional context; the question was regarding where the output binaries were created for each platform, and specifically if x64 build should use something like 'PCBuild/x64'] I'm bringing this topic up again as I noticed the AMD64 builds for Python 2.6 create their output in the PCBuild/x64 directory, not directly into the PCBuild directory. When I raised this with Christian, he also expressed a desire to be able to cross-compile in the same directory structure and was unaware of this discussion (but I made him aware of it :) You made excellent points about how tools currently recognize the PCBuild directory, and we can't break them, and I agree. I'd like to propose a compromise that might be able to keep everyone happy. The main problem I see with all platforms using 'PCBuild' is that in a cross-compilation scenario, it is impossible for the distutils to determine the location of the correct Python libraries to link with (stuff in its own PYTHONHOME is no good; it's for the wrong platform). Obviously, we can change the distutils so that the location of the correct libraries can be specified, but that implies all other build systems that currently assume PCBuild must also change to work in a cross-compilation scenario. While those external tools *will* work today in a native build on x64, eventually they may need to be upgraded to deal with cross-compilation - and this means that all such tools will also be unable to automatically determine the location of the libraries. They will all need to take the library dir as a user-specified option, which seems a pain (eg, buildbots would seem to need site-specific configuration information to make this work). If eventually all such tools are likely to upgrade, it would be ideal if we can offer them a way to upgrade so the correct libraries can be determined automatically, as they can now for native builds. My compromise proposal is this: Each platform builds in its own directory (ie, PCBuild/x86_64, PCBuild/x86). Every single build configuration has a post-build step that copies the result of the build to the PCBuild directory. We then add an option to the distutils so that a cross-compile platform can be specified and when it is, to use 'PCBuild/{platform}" instead of PCBuild, and we encourage other tools to use the same directory should they want to support "configure-less" cross-compilation (if distutils on the trunk is not still trying to maintain b/w compat, it should just *always* look in PCBuild/{platform}, and I'd suggest py3k not bother with PCBuild at all; build systems will be touched to upgrade to py3k, so that change isn't painful. But I digress :) The main advantage of the compromise is that it allows for the status quo to remain in place - just continue to use separate source trees for each platform, and only ever build a single platform in each tree, and tools are free to have ways of specifying which directory should be used. The official build process need not change. However, it also supports a way to do cross-compilation in a way that build tools can *automatically* locate the correct platform's libraries, and this will be particularly useful for extension authors. Would something like that be acceptable? Thanks, Mark
I'm wondering if anyone has any comment on this issue? Its currently impossible to use distutils as documented to build x64 extensions, either natively or cross-compiled using the trunk and while I'm keen to help fix this, I'd like agreement on the direction. Can I assume silence means general agreement on the compromise proposal I outlined? Thanks, Mark
-----Original Message----- From: python-dev-bounces+mhammond=keypoint.com.au@python.org [mailto:python-dev-bounces+mhammond=keypoint.com.au@python.org] On Behalf Of Mark Hammond Sent: Tuesday, 22 January 2008 9:06 PM To: '"Martin v. Löwis"' Cc: distutils-sig@python.org; python-dev@python.org Subject: Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi Martin,
Way back on Monday, 21 May 2007, you wrote:
This is an issue to be discussed for Python 2.6. I'm personally hesitant to have the "official" build infrastructure deviate from the layout that has been in-use for so many years, as a lot of things depend on it.
I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't. ...
* Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to generate x64 binaries to their own directory - my expectation is that cross-compiling between platforms is a reasonable use-case, and we should support multiple achitectures for the same compiler version.
See above; I disagree.
[For additional context; the question was regarding where the output binaries were created for each platform, and specifically if x64 build should use something like 'PCBuild/x64']
I'm bringing this topic up again as I noticed the AMD64 builds for Python 2.6 create their output in the PCBuild/x64 directory, not directly into the PCBuild directory. When I raised this with Christian, he also expressed a desire to be able to cross-compile in the same directory structure and was unaware of this discussion (but I made him aware of it :)
You made excellent points about how tools currently recognize the PCBuild directory, and we can't break them, and I agree. I'd like to propose a compromise that might be able to keep everyone happy.
The main problem I see with all platforms using 'PCBuild' is that in a cross-compilation scenario, it is impossible for the distutils to determine the location of the correct Python libraries to link with (stuff in its own PYTHONHOME is no good; it's for the wrong platform). Obviously, we can change the distutils so that the location of the correct libraries can be specified, but that implies all other build systems that currently assume PCBuild must also change to work in a cross-compilation scenario. While those external tools *will* work today in a native build on x64, eventually they may need to be upgraded to deal with cross-compilation - and this means that all such tools will also be unable to automatically determine the location of the libraries. They will all need to take the library dir as a user-specified option, which seems a pain (eg, buildbots would seem to need site-specific configuration information to make this work). If eventually all such tools are likely to upgrade, it would be ideal if we can offer them a way to upgrade so the correct libraries can be determined automatically, as they can now for native builds.
My compromise proposal is this: Each platform builds in its own directory (ie, PCBuild/x86_64, PCBuild/x86). Every single build configuration has a post-build step that copies the result of the build to the PCBuild directory. We then add an option to the distutils so that a cross- compile platform can be specified and when it is, to use 'PCBuild/{platform}" instead of PCBuild, and we encourage other tools to use the same directory should they want to support "configure-less" cross-compilation (if distutils on the trunk is not still trying to maintain b/w compat, it should just *always* look in PCBuild/{platform}, and I'd suggest py3k not bother with PCBuild at all; build systems will be touched to upgrade to py3k, so that change isn't painful. But I digress :)
The main advantage of the compromise is that it allows for the status quo to remain in place - just continue to use separate source trees for each platform, and only ever build a single platform in each tree, and tools are free to have ways of specifying which directory should be used. The official build process need not change. However, it also supports a way to do cross-compilation in a way that build tools can *automatically* locate the correct platform's libraries, and this will be particularly useful for extension authors.
Would something like that be acceptable?
Thanks,
Mark
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python- dev/mhammond%40keypoint.com.au
I'm following up on this thread without checking if there were other following negating a need to respond... If so, ignore as needed. +1 from me. Always build on windows into an architecture specific PCBuild/XXX directory. A bonus if the directory name matches the return value of platform.machine() but I don't care too much about that part. (that'd mean 'i686' and 'x86_64' and who knows what for Itanic users) On Wed, Jan 30, 2008 at 11:25 PM, Mark Hammond <mhammond@skippinet.com.au> wrote:
I'm wondering if anyone has any comment on this issue? Its currently impossible to use distutils as documented to build x64 extensions, either natively or cross-compiled using the trunk and while I'm keen to help fix this, I'd like agreement on the direction.
Can I assume silence means general agreement on the compromise proposal I outlined?
Thanks,
Mark
-----Original Message----- From: python-dev-bounces+mhammond=keypoint.com.au@python.org [mailto:python-dev-bounces+mhammond=keypoint.com.au@python.org] On Behalf Of Mark Hammond Sent: Tuesday, 22 January 2008 9:06 PM To: '"Martin v. Löwis"' Cc: distutils-sig@python.org; python-dev@python.org Subject: Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi Martin,
Way back on Monday, 21 May 2007, you wrote:
This is an issue to be discussed for Python 2.6. I'm personally hesitant to have the "official" build infrastructure deviate from the layout that has been in-use for so many years, as a lot of things depend on it.
I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't. ...
* Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to generate x64 binaries to their own directory - my expectation is that cross-compiling between platforms is a reasonable use-case, and we should support multiple achitectures for the same compiler version.
See above; I disagree.
[For additional context; the question was regarding where the output binaries were created for each platform, and specifically if x64 build should use something like 'PCBuild/x64']
I'm bringing this topic up again as I noticed the AMD64 builds for Python 2.6 create their output in the PCBuild/x64 directory, not directly into the PCBuild directory. When I raised this with Christian, he also expressed a desire to be able to cross-compile in the same directory structure and was unaware of this discussion (but I made him aware of it :)
You made excellent points about how tools currently recognize the PCBuild directory, and we can't break them, and I agree. I'd like to propose a compromise that might be able to keep everyone happy.
The main problem I see with all platforms using 'PCBuild' is that in a cross-compilation scenario, it is impossible for the distutils to determine the location of the correct Python libraries to link with (stuff in its own PYTHONHOME is no good; it's for the wrong platform). Obviously, we can change the distutils so that the location of the correct libraries can be specified, but that implies all other build systems that currently assume PCBuild must also change to work in a cross-compilation scenario. While those external tools *will* work today in a native build on x64, eventually they may need to be upgraded to deal with cross-compilation - and this means that all such tools will also be unable to automatically determine the location of the libraries. They will all need to take the library dir as a user-specified option, which seems a pain (eg, buildbots would seem to need site-specific configuration information to make this work). If eventually all such tools are likely to upgrade, it would be ideal if we can offer them a way to upgrade so the correct libraries can be determined automatically, as they can now for native builds.
My compromise proposal is this: Each platform builds in its own directory (ie, PCBuild/x86_64, PCBuild/x86). Every single build configuration has a post-build step that copies the result of the build to the PCBuild directory. We then add an option to the distutils so that a cross- compile platform can be specified and when it is, to use 'PCBuild/{platform}" instead of PCBuild, and we encourage other tools to use the same directory should they want to support "configure-less" cross-compilation (if distutils on the trunk is not still trying to maintain b/w compat, it should just *always* look in PCBuild/{platform}, and I'd suggest py3k not bother with PCBuild at all; build systems will be touched to upgrade to py3k, so that change isn't painful. But I digress :)
The main advantage of the compromise is that it allows for the status quo to remain in place - just continue to use separate source trees for each platform, and only ever build a single platform in each tree, and tools are free to have ways of specifying which directory should be used. The official build process need not change. However, it also supports a way to do cross-compilation in a way that build tools can *automatically* locate the correct platform's libraries, and this will be particularly useful for extension authors.
Would something like that be acceptable?
Thanks,
Mark
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python- dev/mhammond%40keypoint.com.au
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
participants (12)
-
"Martin v. Löwis" -
Alexey Borzenkov -
Gregory P. Smith -
Jamie Kirkpatrick -
Kristján Valur Jónsson -
M.-A. Lemburg -
Mark Hammond -
Nick Coghlan -
Steve Holden -
Titus Brown -
Tony Nelson -
Trent Mick