On 30 November 1999, Ian Searle said:

We are developing a cross platform debug tool for embedded systems that uses Python as the central foundation. We distribute our product on both solaris and windows nt. Thus, distutils looked attractive for building our python extensions, which are mostly C++. This last weekend, I was working on an extension, and ended up punting distutils for two reasons:

[Please limit your messages to < 80 columns; I had to reformat it down so it quotes nicely.] First caveat: if building C++ extensions with Distutils actually works, it's a happy accident. I had nothing to do with it. ;-)

1) the documentation is not adequate for a moron like me. I needed to add a vendor supplied object file to the link command, and struck out on every attempt. The doc gave me some clues, but was not specific enough to really help. I realize that this is version 0.1.

I disagree -- the documentation isn't adequate for *anyone*, moron or not. You need to add an 'extra_objects' element to the dictionary that describes how to build your extension. Eg. if you now have: setup (... ext_modules = [('myext', {'sources': ['myext.c', 'support.c'] } )] ) then you will need to change this to setup (... ext_modules = [('myext', {'sources': ['myext.c', 'support.c'], 'extra_objects: ['vendor.o'] } )] ) In the code, see distutils.command.build_ext.BuildExt.build_extensions(). Doesn't look like extra_objects is mentioned in the USAGE file (oops). The usual disclaimer: this has not been used in practice, just tried out in artifical tests. Please let me know if it works or doesn't work.

2) the main/compelling reason is that I have to debug the python extensions. The extension implementation is normally tested, and debugged prior to integration into Python. But the extension layer often needs some debugging as we are interfacing C++ with STL containers to Python data types. I can debug just fine using distutils on Solaris/Linux, but cannot do the same on Windows. As far as I can tell, you _have_ to have a DevStudio project setup to debug anything under windows. Since our product will be delivered as a standalone tool, including python, I do not need the distribution facilities of distutils, only the development. But, once I have developed a DevStudio project/workspace for an extension, there is little motivation to use distutils.

Ooh, yuck. And here I thought that invoking the MSVC command-line compiler directly was a feature. David Ascher was the original volunteer to write the MSVC interface, and his plan was (I think) to generate a project file and then somehow invoke the build system on that. When Perry Stoll came along with his MSVCCompiler class, David appeared happy to bow out... but maybe his idea needs to be resuscitated? Yuck. You Windows/MSVC experts got any good ideas? Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

"Andrew M. Kuchling" wrote:

David Ascher writes:

...

The hard thing will probably be to support all of the things which Windows developers expect, such as the ability to have Debug and Release builds (and CE builds, and ...).

Is it really necessary for people to develop their code using Distutils, as opposed to developing in their preferred environment and then wrapping things up for distribution? If I was debugging a C module, I don't think I'd bother compiling it through the distutils; more likely I'd just write a Makefile.pre.in and Setup.in.

I am (among other things) a Windows developer, and I would never consider developing a C or C++ module without a project file in Visual C++. The project enables you to create Release and Debug versions. It has an excellent debugger and I even use it. Normally I do without debuggers. It has built-in Windows documentation which I can not do without. It helps navigate the dreadfully complex Windows compiler options. In my view, the program building part of distutils is only relevant to Unix. Which is fine with me, as I run Unix too. JimA

[Ian Searle]

... I can debug just fine using distutils on Solaris/Linux, but cannot do the same on Windows. As far as I can tell, you _have_ to have a DevStudio project setup to debug anything under windows.

[Greg Ward]

Ooh, yuck. And here I thought that invoking the MSVC command-line compiler directly was a feature. ...

It is. The DevStudio debugger doesn't care how an .exe got built (provided you passed the compiler the right switches to get debug info generated). The main Windows project I work on is built by a hand-written makefile run by a homegrown Win32 variant of GNU make; DevStudio plays no role in it unless & until I want to run under the debugger. Goodness, you can even use the DevStudio Build -> Start Debug -> Attach to Process ... menu item to start debugging an arbitrary process already running on the system, source code or not, originally in a DevStudio language or not. If it wasn't compiled with MS-style symbol info, though, you can't do much more than step thru the machine code. Notwithstanding, many hardcore Windows weenies *prefer* doing development under DevStudio. But I'd say that's a topic for the DevUtils SIG <wink>. ian-will-figure-out-how-to-use-devstudio-long-before-you'll-figure- out-how-to-automate-it-ly y'rs - tim

Tim Peters wrote:

It is. The DevStudio debugger doesn't care how an .exe got built (provided you passed the compiler the right switches to get debug info generated). The main Windows project I work on is built by a hand-written makefile run by a homegrown Win32 variant of GNU make; DevStudio plays no role in it unless & until I want to run under the debugger. Goodness, you can even use the DevStudio Build -> Start Debug -> Attach to Process ... menu item to start debugging an arbitrary process already running on the system, source code or not, originally in a DevStudio language or not.

Very interesting. I didn't know that. Probably neither does the average Windows developer.

If it wasn't compiled with MS-style symbol info, though, you can't do much more than step thru the machine code.

Ooops, not at all what people will want.

Notwithstanding, many hardcore Windows weenies *prefer* doing development under DevStudio. But I'd say that's a topic for the DevUtils SIG <wink>.

I think we may be kidding ourselves. It is not a matter of "prefer". Distutils is not at all compatible with DevStudio. Suppose you add a file to the project in DevStudio. You will have to make the parallel change by hand in Distutils. People use (or should use) _ASSERT(), so there must be Debug and Release versions no matter if the debugger is used or not. Built in to DevStudio, a file edit (at best) in Distutils. And to top it off, you need on-line documentation for Windows anyway, so you need DevStudio or equivalant even if you do learn Distutils. People in PythonDev[:] may put up with this, but if we go to main-stream Windows developers and say "do this our way" they will say "no thanks". Distutils makes sense on Unix where things are done with makefiles. Why do we even care if Windows developers make Python extensions in DevStudio? Jim Ahlstrom

Greg Ward:

The whole point of Distutils is to allow Joe Unix Geek to develop module distributions that will build and install painlessly on non-Unix systems, *and* Jane Windows Weenie to develop module distributions that will build and install painlessly on non-Windows systems. If Jane is writing Windows-only extensions, then nobody's forcing her to use Distutils and maintain a setup script (although it would be nice). Ditto for Joe writing Unix-only extensions.

That means that Unix geeks can't use makefiles to distribute their Python modules, and it means that Windows weenies can't use Microsoft's project files to distribute their Python modules. The bad news is that if you really really want to use the local convention for automated building (both of which have advantages), you have to maintain a separate setup.py script. The good news is that setup.py is portable and written in a real programming language.

Good summary of the distutil charter. I would add a third objective: *users* on all platforms are tired of having to understand the intricate details of building and installing Python modules, and want module distributions that "just work". This includes platform specific distributions. --Guido van Rossum (home page: http://www.python.org/~guido/)

Ahh - but we already have - David's compile.py knows how to use COM to automate DevStudio builds :-) In fact, with the COM extensions we could create and build a DevStudio project from scratch I believe. But I dont think we should.

Don't even need COM. The project files are just text after all. The wxWindows team uses tmake from Troll Tech to generate makefiles for all the various Windows compilers, Makefile.in for GNU configure, and VC++ workspace/project files. Since it's a perl based tool I couldn't stomach using it myself and since distutils wasn't usable yet I created my own script that can make makefiles for VC++ or Unix based systems and can also drive running make. Works pretty good. -- Robin Dunn Software Craftsman robin@AllDunn.com http://AllDunn.com/robin/ http://AllDunn.com/wxPython/ Check it out!

[Tim]

... Goodness, you can even use the DevStudio Build -> Start Debug -> Attach to Process ... menu item to start debugging an arbitrary process already running on the system, source code or not, originally in a DevStudio language or not.

[James C. Ahlstrom]

Very interesting. I didn't know that. Probably neither does the average Windows developer.

They don't need to know it. I was explaining an extreme. In normal usage, the worst that's required is that you open the .exe you want to debug from within DevStudio and click "Go".

...

If it wasn't compiled with MS-style symbol info, though, you can't do much more than step thru the machine code.

Ooops, not at all what people will want.

Again explaining an extreme. If you use MS's compiler to compile your stuff from a cmdline, you get MS-style symbol info, and debugging is the same as if you had compiled it using DevStudio (DevStudio compilation is just a massive GUI around a cmdline compiler & linker!). Of *course* MS's debugger doesn't understand non-MS symbol info, any more than gdb understands MS's style. So don't do that <wink>.

...

Notwithstanding, many hardcore Windows weenies *prefer* doing development under DevStudio. But I'd say that's a topic for the DevUtils SIG <wink>.

I think we may be kidding ourselves. It is not a matter of "prefer".

Sure it is. DevStudio also has a menu entry to generate a makefile (of MS's odd nmake flavor). I often set up personal projects using DevStudio, and then switch to a generated makefile so I can run overnight builds and tests from .bat scripts.

Distutils is not at all compatible with DevStudio. Suppose you add a file to the project in DevStudio. You will have to make the parallel change by hand in Distutils. People use (or should use) _ASSERT(), so there must be Debug and Release versions no matter if the debugger is used or not. Built in to DevStudio, a file edit (at best) in Distutils.

The difference between Debug and Release is entirely in switches passed to the compiler and linker; makefiles can handle that fine based on target name (or envars, or anything else a makefile can see; have DevStudio generate a makefile to see how MS does it). As I hope will become clearer soon, though, I don't grasp why any of this is a Distutils issue.

And to top it off, you need on-line documentation for Windows anyway, so you need DevStudio or equivalant even if you do learn Distutils.

Sorry, but I don't understand where this is coming from. As a Windows Python Guy, I have one of five scenarios in mind for "distribution": 1) The audience is purely Windows developers. Then I ship a zip file containing source and an MS project file, and leave Distutils out of it entirely. 2) The audience is purely Windows end users. Then I ship pre-compiled binaries, because they can't possibly deal with compilation themselves (they don't have compilers!). In this case I'm not sure whether Distutils can help (it would have to become an Installshield clone to meet expectations); but, ragardless, compilation is again not an issue for distribution. 3) The audience is Python programmers "in general", or the audience is me, and the product is a C extension module. This is where I expect to use Distutils, and hope that others do too. An MS project file doesn't do my Unix friends any good, any more than their fiddly platform-dependent gmake files help me, and binaries suck for similar reasons. Most interesting extension modules I've seen have been written by Unix weenies -- and *when you're lucky* their home page has a contributed Windows .pyd just two releases out of date . 4) The audience is Python end-users "in general", and the product is pure Python. I think this is the most important one for Distutils to address, and compilation isn't a part of it. So far, though, what Gordon is doing seems more appropriate than what Distutils has been up to. I hope his work gets folded into this. 5) The audience is Python end-users "in general", and the product is some mix of Python and C extension modules. This is hard. You have to ship binaries to meet Windows expectations; you have to ship source to meet Unixish ones; and the more popular Linux gets, the more the Unix dweebs will be left whining without a clue about the changing nature of their customer base <0.7 wink -- but over on Python-Help the sight of a Linux user who doesn't know how to compile is no longer rare>.

People in PythonDev[:] may put up with this, but if we go to main-stream Windows developers and say "do this our way" they will say "no thanks".

Main-stream Windows developers have no way at all to distribute a x-platform app now, so their choice is between "our way" and "no way". If they're not interested in x-platform distribution, I indeed don't see what Distutils *could* offer them in the way of helpful code (except in case #4 above -- where compilation (& so also DevStudio) isn't an issue).

Distutils makes sense on Unix where things are done with makefiles. Why do we even care if Windows developers make Python extensions in DevStudio?

I don't care at all, provided they have no interest in sharing their work with non-Windows people. If they do want to share, they need to do more work (their Windows binaries aren't going to run on a SPARC <wink>), and Distutils can help with that. life-will-be-simpler-when-windows-is-the-only-os-ly y'rs - tim

On 08 December 1999, Tim Peters said:

The difference between Debug and Release is entirely in switches passed to the compiler and linker; makefiles can handle that fine based on target name (or envars, or anything else a makefile can see; have DevStudio generate a makefile to see how MS does it). As I hope will become clearer soon, though, I don't grasp why any of this is a Distutils issue.

It could be a minor Distutils issue: I've toyed with the idea of "--debug" or "--optimize" flags (or maybe "--release"?) to the 'build_ext' command that would translate to appropriate compiler switches. I know how to do this for the "generic Unix C compiler" implemented by the UnixCCompiler class, but someone else would have to do it it for MSVCCompiler.

1) The audience is purely Windows developers. Then I ship a zip file containing source and an MS project file, and leave Distutils out of it entirely.

Probably appropriate, although in my more megalomaniacal moments I tend to think, "Who needs make? Who needs IDEs? You can do it ALL with Python, a C compiler, and Distutils!" (In an idle moment the other day, I started thinking about a Distutils command to build Perl extensions... I got over it pretty quickly though.)

2) The audience is purely Windows end users. Then I ship pre-compiled binaries, because they can't possibly deal with compilation themselves (they don't have compilers!). In this case I'm not sure whether Distutils can help (it would have to become an Installshield clone to meet expectations); but, ragardless, compilation is again not an issue for distribution.

Creating binary distributions is in the plans, although they'll be called "built distributions" because an RPM that contains only .py (and maybe .pyc) files isn't really "binary". The plan decided on at that fateful Developer's Day session *over a year ago* (wow!) was that we'd figure out something for Windows with WISE Installer (or whatever it's called). I think the theory was that somehow we could write Python code that would magically generate a Windows installer. I certainly haven't done anything about that, although at some point I expect I'll have to walk downstairs and grill Guido about Windows installers for Python until he gives up and tells me to bug someone else. FWIW, the interface I envision is *something* like this: $ ./setup.py bdist --format=rpm [...create a spec file, run rpm -B...] $ ./setup.py bdist --format=wise error: can't create Windows installer under Unix Oops! Let's try that second one again C:\> ./setup.py bdist --format=wise [...somehow magically create a WISE installer...] Hmmm, this is starting to look like yet another argument for system-wide and user-specific config files to supply default values for things like the bdist format. (And, for that matter, the dist format, which can be one of 'gztar', 'ztar', 'tar', or 'zip'.)

4) The audience is Python end-users "in general", and the product is pure Python. I think this is the most important one for Distutils to address, and compilation isn't a part of it. So far, though, what Gordon is doing seems more appropriate than what Distutils has been up to. I hope his work gets folded into this.

Here the distinction between "source distributions" and "build distributions" gets fuzzy. I actually think it's important and worthwhile to provide "built" distributions of pure Python distributions -- not for technical reasons, but for social/political/usability reasons. Eg. if I can install a mixed C/Python distribution from an RPM or a WISE installer, then I should be able to do the same for a pure Python distribution. You and I will know that it's a waste of disk space to offer both versions, and a waste of machine time to have dist-bots churning out pure Python built distributions from source distributions, but it's necessary for a consistent user interface on each platform supported by the "bdist" command. I assume that "what Gordon is doing" is the Python archive stuff? My attitude towards that is the same as it is towards all of these funky new things coming down the pike: interesting, keep an eye on it, maybe try it out if I get a few minutes... *but* Distutils support will have to wait until there is One Standard Way of doing things. The whole point of Distutils is to be the One Standard Way; I certainly don't claim that copying .py files into directories and compiling them to .pyc files is the only way to install Python module distributions, but until these issues (archive format, imputils, etc.) are worked out and finalized then Distutils won't (officially) support them. Of course, I think it would be really cool if competing archive formats were presented to the world with implementations in the form of Distutils command classes, but that's probably my megalomaniacal tendencies coming out again.

5) The audience is Python end-users "in general", and the product is some mix of Python and C extension modules. This is hard. You have to ship binaries to meet Windows expectations; you have to ship source to meet Unixish ones; and the more popular Linux gets, the more the Unix dweebs will be left whining without a clue about the changing nature of their customer base <0.7 wink -- but over on Python-Help the sight of a Linux user who doesn't know how to compile is no longer rare>.

As Guido said on python-dev, a Distutils "appdist" command is probably the answer. Or maybe "appbdist", I dunno. I'm not sure if separate commands are really needed, or if the "build", "install", "dist", and "bdist" commands will grow up to recognize Python applications rather than just module distributions. That's a bit of a ways off, though: I said at the beginning that I want to handle module distributions first, and then worry about applications. Maybe by that time, the "Python archive format", imputils, one-file-Python-distributions, etc. will be more stable. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

Mark Hammond wrote:

Just to echo the rest of the articles in this thread:

Just to echo your echo...

* If someone uses DevStudio to build their apps, it is likely they are targetting Windows only, so distutils is unlikely to offer great advantages.

Yes.

* If per chance they _are_ also targetting Unix, then they must also be maintaining seperate Unix build scripts - still no loss here - move the Unix stuff to distutils and either leave DevStudio alone, or move to make based builds in Windows too. Still the users choice, and nothing to lose from what they currently do - only possibly things to gain if they go that route.

Yes.

In a nutshell, I cant see _any_ good reasons to offer DevStudio support to distutils.

I guess you mean no reason for DevStudio support, but DistUtils still needs Windows compiler (command line) support. Then I agree with this last point. I don't see DistUtils being used except in cases where a Windows and Unix build is required. But that does happen. With wxWindows it happens Big Time. There is still an advantage to coding in DevStudio, makeing the DistUtils files, and pushing the DistUtils button to build multiple versions. JimA

On Thu, 2 Dec 1999, Greg Ward wrote:

...

2) the main/compelling reason is that I have to debug the python extensions. The extension implementation is normally tested, and debugged prior to integration into Python. But the extension layer often needs some debugging as we are interfacing C++ with STL containers to Python data types. I can debug just fine using distutils on Solaris/Linux, but cannot do the same on Windows. As far as I can tell, you _have_ to have a DevStudio project setup to debug anything under windows. Since our product will be delivered as a standalone tool, including python, I do not need the distribution facilities of distutils, only the development. But, once I have developed a DevStudio project/workspace for an extension, there is little motivation to use distutils.

Ooh, yuck. And here I thought that invoking the MSVC command-line compiler directly was a feature. David Ascher was the original volunteer to write the MSVC interface, and his plan was (I think) to generate a project file and then somehow invoke the build system on that. When Perry Stoll came along with his MSVCCompiler class, David appeared happy to bow out... but maybe his idea needs to be resuscitated? Yuck. You Windows/MSVC experts got any good ideas?

<ears tingle> Someone mentioned my name? Sigh. The project file vs. Makefile debate is as old as the world (well, at least as old as MPW =). I haven't looked carefully enough at the innards of distutils to be sure, but glancing at msvccompiler.py, it looks like Perry did the simple thing (and praise him). That said, I'd be surprised if making a project/workspace file would be that hard given Perry's groundwork (and stealing the templates from compile.py). Alas, I have *no time* in the foreseeable future to do this. If someone is interested in doing it, however, I can help them with whatever knowledge I gathered doing compile.py. The hard thing will probably be to support all of the things which Windows developers expect, such as the ability to have Debug and Release builds (and CE builds, and ...). --david

Hi, But is there a way to allow the win32 extensions to build *without* needing Visual Studio MFC code? I have a python script that converts dsw/dsp files into (semi) standard makefiles. These makefiles can be driven to work with MSDev, cygwin, ming or unix by changing a config option. I was hoping to compile the win32 extensions using ming and it worked - up to a point. It needed source from MS Dev which I no longer have :( Cheers, Anthony Pfrunder