Step by step: Compiling extensions with MS Visual C++ Toolkit 2003 - msvccompiler-patch.txt (0/1)

Wed Dec 15 07:52:22 EST 2004

The good news:

Along with Python-2.4 comes really good news to Windows users. Yes,
you now CAN build extension modules yourself using the SAME C++
compiler and linker Python is built with itself. Everything you need
is available at no costs (except download hassle and installation
time). Once your system is set up properly its just a matter of
running 'python setup.py build'. No longer waiting for someone else to
build binaries and a relief to those who kindly used to provide those
binaries.

The bad news: 

It's not easy to get there! Gathering information and packages and
finding out about all the little steps and stumbling blocks may be
very time consuming.

Relax:

Here's (much of) what you need to know. With this document I'm trying
to provide the information I would have needed. It's sort of a step by
step instruction and contains some details. Goal: Compile Python-2.4
extensions with the freely available "Microsoft Visual C++ Toolkit
2003". This document DOES NOT address the issue of compiling Python
itself.

The example chosen to build is the "MySQLdb" module with its
_mysql.pyd extension.

Go and check out Mike Fletcher's page first at

   http://www.vrplumber.com/programming/mstoolkit/

It's his work (and the need to get a working MySQLdb for Python-2.4 of
course ;-) that motivated me to start and spend some hours. Without
Mike's page I wouldn't even have tried. Thank you so much! Now I'm
prepared.

Two things first - not to forget:

(1) In contrast to what Mike writes I had to use a different registry
key (see step 9)

(2) I don't now what people mean when talking about "msvcr71.lib".
There's no such file on my machine. BUT there IS a "msvcrt.lib" a
directory "C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib". I
guess the ".\Vc7\." indicates we are talking of the expected lib.
Correct? I don't know.

Attention - long lines:
Watch out for long lines that may have gotten split in this posting!

See if you've got something already. These are important directories
we're looking for. When "echo %ProgramFiles%" gives "C:\Programme"
there should be:

  C:\Programme\Microsoft Platform SDK for Windows XP SP2\Include
  C:\Programme\Microsoft Platform SDK for Windows XP SP2\Lib
  C:\Programme\Microsoft Visual C++ Toolkit 2003\bin
  C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib
  C:\Programme\Microsoft.NET\SDK\v1.1\include
  C:\Programme\Microsoft.NET\SDK\v1.1\Lib

Step By Step
============

1. Set up a Windows machine
1.1 Install Windows XP (Prof. or Home doesn't matter)
1.2 Apply Service Pack 2 (WinXP SP2)
1.3 Perform a Windows update at
   http://windowsupdate.microsoft.com
to get the latest patches.

2. Get some tools

This is optional - you don't have to. But it may be instructive and
helpful. And - I would say - fun.

2.1 Get "Windows Resource Kits"
Brings some useful tools like "CMD Prompt Here" to the windows
explorer context menu or "robocopy.exe" - a copy tools really worth to
look at. Seems MS developers need good tools too ...

   http://www.microsoft.com/windows/reskits/default.asp

The website says:
"""
Windows Server 2003 Resource Kit Tools
The Resource Kit Tools are a set of software tools for network
administrators, developers, and power users to manage TCP/IP,
networks, the registry, security, remote administration,
configuration, batch files, and many other areas of the Windows Server
2003 operating system. To install the tools, download and run
Rktools.exe.
"""
(rktools.exe, 12.337.752 bytes).

2.2 Add some Unix to Windows. Goto 

   http://unxutils.sourceforge.net/

and get UnxUtils.zip (4.125 kB)
"""
Here are some ports of common GNU utilities to native Win32. In this
context, native means the executables do only depend on the Microsoft
C-runtime (msvcrt.dll) and not an emulation layer like that provided
by Cygwin tools.
"""
For example, this gives you "diff.exe" and "patch.exe".

3. Get and install the the .NET Framework 1.1.

It's available at no costs from the MicroSoft website. Use Google to
find the download page.

3.1 Install the ".NET Framework 1.1 runtime" first (dotnetfx.exe,
24.277.024 bytes)

3.2 Install the .NET Framework 1.1 SDK" second. (setup(sdk1.1).exe,
117.062.664 bytes).

Attention, observation:

This step should indeed create the badly needed 

   C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib

folder. It didn't on my machine until I first completely deinstalled
on existing installation of .NET Framework 1.1 runtime AND .NET
Framework 1.1 SDK and then resumed with steps 3.1 and 3.2. Only
Redmond will know why - and I doubt it.

3.3 Perform a Windows update at
   http://windowsupdate.microsoft.com
to get the latest patches of the .NET Framework.

4. Platform SDK

Get and install the platform SDK "Windows XP SP2 SDK" (about
266.000.00 bytes) from
http://www.microsoft.com/msdownload/platformsdk/sdkupdate/XPSP2FULLInstall.htm

You better download the whole stuff (several .CAB-files) first instead
of trying to install online. This may be too risky.

5. Get and install free "Visual C++ Toolkit 2003"

This is the C++ compiler and linker. Use Google to find the "Microsoft
VisualC++ Toolkit 2003" download page and get "VCToolkitSetup.exe"
(32.948.392 bytes).

The website says:
"""
Overview
The Microsoft Visual C++ Toolkit 2003 includes the core tools
developers need to compile and link C++-based applications for Windows
and the .NET Common Language

Runtime:
Microsoft C/C++ Optimizing Compiler and Linker.  These are the same
compiler and linker that ship with Visual Studio .NET 2003
Professional!

C Runtime Library and the C++ Standard Library, including the Standard
Template Library.  These are the same static-link libraries included
with Visual Studio.

Microsoft .NET Framework Common Language Runtime. Visual C++ can
optionally build applications that target the Common LanguageRuntime
(CLR).

Sample code.  The toolkit includes four samples designed to showcase
the powerful new features of the 2003 version, including new
optimization capabilities, features to improve code-security and
robustness, enhanced ISO C++ standards support, and the ability to use
the .NET Framework library and target the CLR.
"""

6. Be sure that all components are up to date and do a Windows update
at
   http://windowsupdate.microsoft.com

7. Install Python-2.4 to C:\Python24

8. Modify the Python distutils

Make a backup of C:\Python24\Lib\distutils\msvccompiler.py and apply
the patches as described in

   http://www.vrplumber.com/programming/mstoolkit/

to this file. You can do this by hand or use "patch.exe" which you've
gotten if you did follow step 2.2. There should be a file
"msvccompiler-patch.txt" appended to this message.

   patch.exe  msvccompiler.py  msvccompiler-patch.txt

should succeed with no error.

9. Verify registry key of platform SDK

Start "regedit.exe" from the commandline and check for the registry
key of the platform SKD.

ON MY MACHINE and in my case it is different from the one listed on
   http://www.vrplumber.com/programming/mstoolkit/

It is
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05

and it is not
...\63DADB24-DC99-45EB-A748-EC93AB8A7497.

Find out about the correct key in your case and adjust the crucial
line 133 in msvccompiler.py: 

freeSDK=r"SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05"

If the key is wrong, you may encounter something like:

"""
C:\> python setup.py build
running build
running build_ext
error: Python was built with version 7.1 of Visual Studio, and
extensions need to be built with the same version of the ompiler, but
it isn't installed.
"""

10. Create a script file "pathsetup.bat" hat helps you set up the
necessary paths. We don't need to add the Python24 lib and include
paths as the distutils will take care for this automatically.

++++++++++++++++
REM START of pathsetup.bat, mb, 2004-12-12

REM add Python-2.4 to path
set Path=C:\Python24;%Path%

REM add the compiler path
set Path=C:\Programme\Microsoft Visual C++ Toolkit 2003\bin;%Path%

REM setup include paths
set Include=%ProgramFiles%\Microsoft Platform SDK for Windows XP
SP2\Include;%Include%
set Include=%ProgramFiles%\Microsoft Visual C++ Toolkit
2003\include;%Include%

REM setup Lib paths
set Lib=C:\Programme\Microsoft Platform SDK for Windows XP
SP2\Lib;%Lib%
set Lib=C:\Programme\Microsoft Visual C++ Toolkit 2003\lib;%Lib%
set Lib=C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib;%xxLib%

REM END of pathsetup.bat
++++++++++++++++

11. Source distribution of Python module

Get and unpack the source distribution of a module you want to build.
In our case get "MySQL-python-1.1.7.tar.gz" from 
   http://sourceforge.net/projects/mysql-python or

http://sourceforge.net/project/showfiles.php?group_id=22307&package_id=34790
und unpack to ...\somehwere\MySQL-python-1.1.7

11.1 Read README and setup.py

Is there a README in "MySQL-python-1.1.7\"? Yes, there is! It tells
you to get additional header and library files from the MySQL website.
Looking for version 4.x at

   http://www.mysql.com/downloads

we'll follow version 4.x and be taken to 

   http://dev.mysql.com/downloads/mysql/4.0.html

Locate "Windows downloads (platform notes)" on that page and download
the binary distribution ("Windows (x86) Without installer (unzip in
C:\) 4.0.22	20.8M"). The file is "mysql-4.0.22-win-noinstall.zip"
(21.346 kB).

Since we're not interested in running MySQL but only in building the
extension we unzip into ...\MySQL-python-1.1.7\mysql-4.0.22-win

11.2
Make sure the MySQL header and library files are accessible by the C++
compiler and linker. The location is hardcoded in 'setup.py' of the
MySQL-python-1.1.7 module.

Search for "win32" in 'setup.py' and change references to
'C:\myslq\...' to '.\mysql-4.0.22-win

### >>>>>> schnipp >>>>>>>
#  setup.py of MySQL-python-1.1.7

elif sys.platform == "win32": # Ugh
    include_dirs = [r'.\mysql-4.0.22-win\include'] 
    library_dirs = [r'.\mysql-4.0.22-win\lib\opt'] 
    extra_objects = [r'.\mysql-4.0.22-win\lib\opt\mysqlclient.lib']

### <<<<<< schnapp <<<<<<<

12.
Relax. Take a breath. We're ready to go :-)

12.1
Open a command window.

12.2
Run "pathsetup.bat" to setup the environment.

12.3 Change directory and step into ...\MySQL-python-1.1.7, that is,
into the root dir of the module you want to build - the dir that
contains the "setup.py" file.

12.4 Start building the module including the extension using distutils
with the command:

    python  setup.py  build

13.
There aren't fatal errors? Congratulations, you made it! Why not
install then:

    python  setup.py  install

should do the trick. (Yes, I had two or three warnings. But they don't
seem to be important).

14. Earn what you deserve

Start Python and import:

Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>                    <---------- imported! no error here!

Yes, it does import :-) And it does work :-) And _mysql.pyd is similar
in size to those versions I had before (about 185 kB) :-)

Life goes on. A different time another thread I'm going to ask about
the warnings I got with building MySQLdb ...

mb - Martin