Script(s) for building Python on Windows
The current scripts for building Python lack some things to be desired. The first thing I notice when I try to build Python on Windows is the scripts expect to be run inside of a Visual Studio environment, the environment of which is only defined inside of a cmd.exe context. This means the scripts can't be executed from within Powershell (my preferred shell on Windows). One must first shell out to cmd.exe, which disables any Powershell-specific features the developer might have installed (aliases, functions, etc). The second thing I notice is the scripts assume Visual Studio 2008. And while I recognize that Python is specifically built against Visual Studio 2008 for the official releases and that Visual Studio 2008 may be the only officially-supported build environment, later releases, such as Visual Studio 2010 are also adequate for testing purposes. I've been developing Python against Visual Studio 2010 for quite a while and it seems to be more than adequate. And while it's not the responsibility of the scripts to accommodate such environments, if the scripts could allow for such environments, that would be nice. Furthermore, having scripts that codify the process to upgrade will facilitate the migration should someone make the decision to officially upgrade to Visual Studio 2010. The third thing that I notice is that the command-line argument handling by the batch scripts is clumsy (compared to argparse, for example). This clumsiness is not a criticism of the authors, who have done well with the tools they had. However, batch programming is probably one of the least powerful ways to automate builds these days. So to ease my experience, I've developed my own library of functions and commands to facilitate building Python that aren't subject to the above limitations. Of course, I built these in Python, so they do require Python to build Python (not a huge burden, but worth mentioning). All of these modules are open-source and part of the jaraco.develop package <http://pypi.python.org/pypi/jaraco.develop> . The first of these modules is jaraco.develop.vstudio <https://bitbucket.org/jaraco/jaraco.develop/src/b7263c9d9c93/jaraco/develop /vstudio.py> . It exposes a class for locating Visual Studio in the usual locations, loading the environment for that instance of Visual Studio, and upgrading a project or solution file to that version. This class in particular enables running Visual Studio commands (including msbuild) from within a Visual Studio environment without actually requiring a cmd.exe context with that environment. Another module is jaraco.develop.python <https://bitbucket.org/jaraco/jaraco.develop/src/b7263c9d9c93/jaraco/develop /python.py> , which includes build_python, a function (and command) to build Python using whatever version of Visual Studio can be found (9 or 10 required). It has no environmental requirements except that Visual Studio be installed. Simply run build-python (part of jaraco.develop's console scripts) and it will build PCbuild.sln from the current directory to whatever targets are specified (or all of them if none are specified). The builder currently makes some assumptions (such as always building the 64-bit Release targets), but those could easily be customized using argparse parameters. This package and these modules have been tested and run on Python 2.7+. These tools solve the three shortcomings I mentioned above and make the development process so much smoother, IMO. If these modules were built into the repository, building Python could be as simple as "hg clone; cd cpython/pcbuild; ./build.py" (assuming only Visual Studio and Python available). I'd like to propose migrating this functionality (mainly these two modules) into the CPython heads for Python 2.7, 3.1, 3.2, and default as PCbuild/build.py (or similar). This functionality doesn't necessarily need to supersede the existing scripts (env, build_env, build), though it certainly could (and would as far as my usage is concerned). If there are no objections, I'll work to extract the aforementioned functionality from the jaraco.develop modules and into a portable script and put together a proof-of-concept in the default branch. The build script should not interfere with any build bots or other existing build processes, but should enable another more powerful technique for producing builds. I look forward to your comments and feedback. Regards, Jason
On Mon, Jan 16, 2012 at 14:00, Jason R. Coombs <jaraco@jaraco.com> wrote:
The second thing I notice is the scripts assume Visual Studio 2008. And while I recognize that Python is specifically built against Visual Studio 2008 for the official releases and that Visual Studio 2008 may be the only officially-supported build environment, later releases, such as Visual Studio 2010 are also adequate for testing purposes. I’ve been developing Python against Visual Studio 2010 for quite a while and it seems to be more than adequate. And while it’s not the responsibility of the scripts to accommodate such environments, if the scripts could allow for such environments, that would be nice.
2010 is adequate for limited use but the test suite doesn't pass, so I would be hesitant to add support and/or documentation for building with it until we actually support it the same as or in place of 2008.
From: Brian Curtin [mailto:brian@python.org] Sent: Monday, 16 January, 2012 15:20
2010 is adequate for limited use but the test suite doesn't pass, so I would be hesitant to add support and/or documentation for building with it until we actually support it the same as or in place of 2008.
Good point. The current tools don't automatically support 2010; an extra command is require to perform the conversion. I'll be cautious and not expose that functionality without some indication to the user of the limitations.
If there are no objections, I’ll work to extract the aforementioned functionality from the jaraco.develop modules and into a portable script and put together a proof-of-concept in the default branch. The build script should not interfere with any build bots or other existing build processes, but should enable another more powerful technique for producing builds.
I'd be hesitant to put too many specialized tools into the tree that will become unmaintained. Please take a look at the vs9to8 tool in PCbuild; if you could adjust that to support VS 10, it would be better IMO. As for completely automating the build: please take notice of Tools/buildbot/build.bat. It also fully automates the build, also doesn't require that the VS environment is already activated, and has the additional advantage of not requiring Python to be installed. Regards, Martin
From: "Martin v. Löwis" [mailto:martin@v.loewis.de] Sent: Monday, 16 January, 2012 16:25
I'd be hesitant to put too many specialized tools into the tree that will become unmaintained. Please take a look at the vs9to8 tool in PCbuild; if you could adjust that to support VS 10, it would be better IMO.
Are you suggesting creating vs10to9, which would be congruent to vs9to8, or vs9to10? I'm unsure if the conversion from 9 to 10 or 10 to 9 can be as simple as the vs9to8 suggests. When I run the upgrade using the Visual Studio tools, it does upgrade the .sln file [as so]( http://a.libpa.st/kB19G). But as you can see, it also converts all of the .vcproj to .vcxproj, which appears to be a very different schema. According to [this article]( http://social.msdn.microsoft.com/Forums/en/vsprereleaseannouncements/thread/ 4345a151-d288-48d6-b7c7-a7c598d0f85e) it should be trivial to downgrade by only updating the .sln file (perhaps Visual Studio 2008 is forward compatible with the .vcxproj format). I'll look into this more when I have a better idea what you had in mind. My goal in adding the upgrade code was to provide a one-step upgrade for developers with only VS 10 installed. That's what vs-upgrade in jaraco.develop does.
As for completely automating the build: please take notice of Tools/buildbot/build.bat. It also fully automates the build, also doesn't require that the VS environment is already activated, and has the additional advantage of not requiring Python to be installed.
That's interesting, but it still suffers from several shortcomings: 1) It still assumes Visual Studio 2008 and fails with an obscure error otherwise. 2) You can't use it to build different targets (only the whole solution). 3) It automatically downloads the external dependencies (it'd be nice to build without them on occasion). 4) It's still a batch file, so still gives the abominable "Terminate batch job (Y/N)?" when cancelling any operation via Ctrl+C. 5) This functionality isn't in PCBuild/*. Why not? 6) There's no good way to select which type to build (64-bit versus 32-bit, release versus debug). Adding these command-line options is clumsy in batch files. 7) Since it's written in batch script, Python programmers might be hesitant to work with it (improve it). For a buildbot, the batch file is perfectly adequate. It should do the same thing every time reliably. For anyone but a robot or seasoned CPython Windows developer, however, the build tools are not intuitive, and I find that I'm constantly tweaking the batch scripts and asking myself, "why couldn't this be in Python, which is a much more powerful language?" This is why I developed the scripts, and my thought is they could be useful to others as well. My hope is they might even supersede the existing scripts and become canonical, in which case there would be no possibility of them becoming unmaintained. If it turns out that they do become unused and unmaintained, they can be removed, but my feeling is since they're concise, documented, Python scripts, they'd be more likely to be maintained than their '.bat' counterparts.
On Mon, Jan 16, 2012 at 18:01, Jason R. Coombs
My goal in adding the upgrade code was to provide a one-step upgrade for developers with only VS 10 installed. That's what vs-upgrade in jaraco.develop does.
Upgrading to 2010 requires some code changes in addition to the conversion, so the process might not be as ripe for automation as the previous versions. For one, a lot of constants in errno had to be updated, then a few places that set certain errnos had to be updated.
From: python-dev-bounces+jaraco=jaraco.com@python.org [mailto:python- dev-bounces+jaraco=jaraco.com@python.org] On Behalf Of Jason R. Coombs Sent: Monday, 16 January, 2012 19:01
I'm unsure if the conversion from 9 to 10 or 10 to 9 can be as simple as the vs9to8 suggests. When I run the upgrade using the Visual Studio tools, it does upgrade the .sln file [as so]( http://a.libpa.st/kB19G). But as you can see, it also converts all of the .vcproj to .vcxproj, which appears to be a very different schema. According to [this article]( http://social.msdn.microsoft.com/Forums/en/vsprereleaseannouncements/thre ad/ 4345a151-d288-48d6-b7c7-a7c598d0f85e) it should be trivial to downgrade by only updating the .sln file (perhaps Visual Studio 2008 is forward compatible with the .vcxproj format).
I upgraded the solution file using Visual Studio, then followed those instructions suggested by the article, but the solution no longer builds under Visual Studio 2008, so apparently that answer is incorrect. Perhaps it's possible to upgrade the .sln in a less aggressive way than the Visual Studio tools do by default, but my initial experience suggests it won't be as easy to upgrade/downgrade the solution file as it was between VS8/VS9.
Are you suggesting creating vs10to9, which would be congruent to vs9to8, or vs9to10?
After reconsidering, I don't think I want anything like this in the tree at this point. The code will be outdated by the time Python 3.3 is released, as Python 3.3 will be built with a Visual Studio different from 2008. Regards, Martin P.S. Please shorten your messages. They contain too much text for me to grasp.
participants (3)
-
"Martin v. Löwis" -
Brian Curtin -
Jason R. Coombs