Questions about distutils strategy

We are developing a cross platform debug tool for embedded systems that uses Python as the central foundation. We distribute our product on both solaris and windows nt. Thus, distutils looked attractive for building our python extensions, which are mostly C++. This last weekend, I was working on an extension, and ended up punting distutils for two reasons: 1) the documentation is not adequate for a moron like me. I needed to add a vendor supplied object file to the link command, and struck out on every attempt. The doc gave me some clues, but was not specific enough to really help. I realize that this is version 0.1. 2) the main/compelling reason is that I have to debug the python extensions. The extension implementation is normally tested, and debugged prior to integration into Python. But the extension layer often needs some debugging as we are interfacing C++ with STL containers to Python data types. I can debug just fine using distutils on Solaris/Linux, but cannot do the same on Windows. As far as I can tell, you _have_ to have a DevStudio project setup to debug anything under windows. Since our product will be delivered as a standalone tool, including python, I do not need the distribution facilities of distutils, only the development. But, once I have developed a DevStudio project/workspace for an extension, there is little motivation to use distutils. I am hopeful that distutils will evolve into a useful development and distribution tool for python. Though, I wonder how the "debugging extensions on Windows" problem will be approached. Cheers, -Ian Searle

On 30 November 1999, Ian Searle said:
[Please limit your messages to < 80 columns; I had to reformat it down so it quotes nicely.] First caveat: if building C++ extensions with Distutils actually works, it's a happy accident. I had nothing to do with it. ;-)
I disagree -- the documentation isn't adequate for *anyone*, moron or not. You need to add an 'extra_objects' element to the dictionary that describes how to build your extension. Eg. if you now have: setup (... ext_modules = [('myext', {'sources': ['myext.c', 'support.c'] } )] ) then you will need to change this to setup (... ext_modules = [('myext', {'sources': ['myext.c', 'support.c'], 'extra_objects: ['vendor.o'] } )] ) In the code, see distutils.command.build_ext.BuildExt.build_extensions(). Doesn't look like extra_objects is mentioned in the USAGE file (oops). The usual disclaimer: this has not been used in practice, just tried out in artifical tests. Please let me know if it works or doesn't work.
Ooh, yuck. And here I thought that invoking the MSVC command-line compiler directly was a feature. David Ascher was the original volunteer to write the MSVC interface, and his plan was (I think) to generate a project file and then somehow invoke the build system on that. When Perry Stoll came along with his MSVCCompiler class, David appeared happy to bow out... but maybe his idea needs to be resuscitated? Yuck. You Windows/MSVC experts got any good ideas? Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

[Ian Searle]
[Greg Ward]
Ooh, yuck. And here I thought that invoking the MSVC command-line compiler directly was a feature. ...
It is. The DevStudio debugger doesn't care how an .exe got built (provided you passed the compiler the right switches to get debug info generated). The main Windows project I work on is built by a hand-written makefile run by a homegrown Win32 variant of GNU make; DevStudio plays no role in it unless & until I want to run under the debugger. Goodness, you can even use the DevStudio Build -> Start Debug -> Attach to Process ... menu item to start debugging an arbitrary process already running on the system, source code or not, originally in a DevStudio language or not. If it wasn't compiled with MS-style symbol info, though, you can't do much more than step thru the machine code. Notwithstanding, many hardcore Windows weenies *prefer* doing development under DevStudio. But I'd say that's a topic for the DevUtils SIG <wink>. ian-will-figure-out-how-to-use-devstudio-long-before-you'll-figure- out-how-to-automate-it-ly y'rs - tim

Tim Peters wrote:
Very interesting. I didn't know that. Probably neither does the average Windows developer.
Ooops, not at all what people will want.
Notwithstanding, many hardcore Windows weenies *prefer* doing development under DevStudio. But I'd say that's a topic for the DevUtils SIG <wink>.
I think we may be kidding ourselves. It is not a matter of "prefer". Distutils is not at all compatible with DevStudio. Suppose you add a file to the project in DevStudio. You will have to make the parallel change by hand in Distutils. People use (or should use) _ASSERT(), so there must be Debug and Release versions no matter if the debugger is used or not. Built in to DevStudio, a file edit (at best) in Distutils. And to top it off, you need on-line documentation for Windows anyway, so you need DevStudio or equivalant even if you do learn Distutils. People in PythonDev[:] may put up with this, but if we go to main-stream Windows developers and say "do this our way" they will say "no thanks". Distutils makes sense on Unix where things are done with makefiles. Why do we even care if Windows developers make Python extensions in DevStudio? Jim Ahlstrom

On 07 December 1999, James C. Ahlstrom said:
The whole point of Distutils is to allow Joe Unix Geek to develop module distributions that will build and install painlessly on non-Unix systems, *and* Jane Windows Weenie to develop module distributions that will build and install painlessly on non-Windows systems. If Jane is writing Windows-only extensions, then nobody's forcing her to use Distutils and maintain a setup script (although it would be nice). Ditto for Joe writing Unix-only extensions. That means that Unix geeks can't use makefiles to distribute their Python modules, and it means that Windows weenies can't use Microsoft's project files to distribute their Python modules. The bad news is that if you really really want to use the local convention for automated building (both of which have advantages), you have to maintain a separate setup.py script. The good news is that setup.py is portable and written in a real programming language. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

Greg Ward:
Good summary of the distutil charter. I would add a third objective: *users* on all platforms are tired of having to understand the intricate details of building and installing Python modules, and want module distributions that "just work". This includes platform specific distributions. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Tim]
[James C. Ahlstrom]
Very interesting. I didn't know that. Probably neither does the average Windows developer.
They don't need to know it. I was explaining an extreme. In normal usage, the worst that's required is that you open the .exe you want to debug from within DevStudio and click "Go".
If it wasn't compiled with MS-style symbol info, though, you can't do much more than step thru the machine code.
Ooops, not at all what people will want.
Again explaining an extreme. If you use MS's compiler to compile your stuff from a cmdline, you get MS-style symbol info, and debugging is the same as if you had compiled it using DevStudio (DevStudio compilation is just a massive GUI around a cmdline compiler & linker!). Of *course* MS's debugger doesn't understand non-MS symbol info, any more than gdb understands MS's style. So don't do that <wink>.
I think we may be kidding ourselves. It is not a matter of "prefer".
Sure it is. DevStudio also has a menu entry to generate a makefile (of MS's odd nmake flavor). I often set up personal projects using DevStudio, and then switch to a generated makefile so I can run overnight builds and tests from .bat scripts.
The difference between Debug and Release is entirely in switches passed to the compiler and linker; makefiles can handle that fine based on target name (or envars, or anything else a makefile can see; have DevStudio generate a makefile to see how MS does it). As I hope will become clearer soon, though, I don't grasp why any of this is a Distutils issue.
Sorry, but I don't understand where this is coming from. As a Windows Python Guy, I have one of five scenarios in mind for "distribution": 1) The audience is purely Windows developers. Then I ship a zip file containing source and an MS project file, and leave Distutils out of it entirely. 2) The audience is purely Windows end users. Then I ship pre-compiled binaries, because they can't possibly deal with compilation themselves (they don't have compilers!). In this case I'm not sure whether Distutils can help (it would have to become an Installshield clone to meet expectations); but, ragardless, compilation is again not an issue for distribution. 3) The audience is Python programmers "in general", or the audience is me, and the product is a C extension module. This is where I expect to use Distutils, and hope that others do too. An MS project file doesn't do my Unix friends any good, any more than their fiddly platform-dependent gmake files help me, and binaries suck for similar reasons. Most interesting extension modules I've seen have been written by Unix weenies -- and *when you're lucky* their home page has a contributed Windows .pyd just two releases out of date <wink/frown>. 4) The audience is Python end-users "in general", and the product is pure Python. I think this is the most important one for Distutils to address, and compilation isn't a part of it. So far, though, what Gordon is doing seems more appropriate than what Distutils has been up to. I hope his work gets folded into this. 5) The audience is Python end-users "in general", and the product is some mix of Python and C extension modules. This is hard. You have to ship binaries to meet Windows expectations; you have to ship source to meet Unixish ones; and the more popular Linux gets, the more the Unix dweebs will be left whining without a clue about the changing nature of their customer base <0.7 wink -- but over on Python-Help the sight of a Linux user who doesn't know how to compile is no longer rare>.
Main-stream Windows developers have no way at all to distribute a x-platform app now, so their choice is between "our way" and "no way". If they're not interested in x-platform distribution, I indeed don't see what Distutils *could* offer them in the way of helpful code (except in case #4 above -- where compilation (& so also DevStudio) isn't an issue).
I don't care at all, provided they have no interest in sharing their work with non-Windows people. If they do want to share, they need to do more work (their Windows binaries aren't going to run on a SPARC <wink>), and Distutils can help with that. life-will-be-simpler-when-windows-is-the-only-os-ly y'rs - tim

Or who don't WANT to compile and would "much rather just use an RPM." I've had several of both types of people using wxPython. -- Robin Dunn Software Craftsman robin@AllDunn.com http://AllDunn.com/robin/ http://AllDunn.com/wxPython/ Check it out!

On 08 December 1999, Tim Peters said:
It could be a minor Distutils issue: I've toyed with the idea of "--debug" or "--optimize" flags (or maybe "--release"?) to the 'build_ext' command that would translate to appropriate compiler switches. I know how to do this for the "generic Unix C compiler" implemented by the UnixCCompiler class, but someone else would have to do it it for MSVCCompiler.
Probably appropriate, although in my more megalomaniacal moments I tend to think, "Who needs make? Who needs IDEs? You can do it ALL with Python, a C compiler, and Distutils!" (In an idle moment the other day, I started thinking about a Distutils command to build Perl extensions... I got over it pretty quickly though.)
Creating binary distributions is in the plans, although they'll be called "built distributions" because an RPM that contains only .py (and maybe .pyc) files isn't really "binary". The plan decided on at that fateful Developer's Day session *over a year ago* (wow!) was that we'd figure out something for Windows with WISE Installer (or whatever it's called). I think the theory was that somehow we could write Python code that would magically generate a Windows installer. I certainly haven't done anything about that, although at some point I expect I'll have to walk downstairs and grill Guido about Windows installers for Python until he gives up and tells me to bug someone else. FWIW, the interface I envision is *something* like this: $ ./setup.py bdist --format=rpm [...create a spec file, run rpm -B...] $ ./setup.py bdist --format=wise error: can't create Windows installer under Unix Oops! Let's try that second one again C:\> ./setup.py bdist --format=wise [...somehow magically create a WISE installer...] Hmmm, this is starting to look like yet another argument for system-wide and user-specific config files to supply default values for things like the bdist format. (And, for that matter, the dist format, which can be one of 'gztar', 'ztar', 'tar', or 'zip'.)
Here the distinction between "source distributions" and "build distributions" gets fuzzy. I actually think it's important and worthwhile to provide "built" distributions of pure Python distributions -- not for technical reasons, but for social/political/usability reasons. Eg. if I can install a mixed C/Python distribution from an RPM or a WISE installer, then I should be able to do the same for a pure Python distribution. You and I will know that it's a waste of disk space to offer both versions, and a waste of machine time to have dist-bots churning out pure Python built distributions from source distributions, but it's necessary for a consistent user interface on each platform supported by the "bdist" command. I assume that "what Gordon is doing" is the Python archive stuff? My attitude towards that is the same as it is towards all of these funky new things coming down the pike: interesting, keep an eye on it, maybe try it out if I get a few minutes... *but* Distutils support will have to wait until there is One Standard Way of doing things. The whole point of Distutils is to be the One Standard Way; I certainly don't claim that copying .py files into directories and compiling them to .pyc files is the only way to install Python module distributions, but until these issues (archive format, imputils, etc.) are worked out and finalized then Distutils won't (officially) support them. Of course, I think it would be really cool if competing archive formats were presented to the world with implementations in the form of Distutils command classes, but that's probably my megalomaniacal tendencies coming out again.
As Guido said on python-dev, a Distutils "appdist" command is probably the answer. Or maybe "appbdist", I dunno. I'm not sure if separate commands are really needed, or if the "build", "install", "dist", and "bdist" commands will grow up to recognize Python applications rather than just module distributions. That's a bit of a ways off, though: I said at the beginning that I want to handle module distributions first, and then worry about applications. Maybe by that time, the "Python archive format", imputils, one-file-Python-distributions, etc. will be more stable. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

Tim Peters wrote:
Yes, understood. Extremes are good for sharpening issues.
Yes, we write the makefiles by hand so we can build from Korn shell scripts. Microsoft's eccentric nmake format is a constant annoyance. But mostly Windows people stay in DevStudio.
It means that even if I need a Unix build too, I usually develop in DevStudio because of the on-line help. But for small projects, I develop on Linux because of the large setup annoyance of DevStudio. In other words, I have a poor memory and like built-in help.
As a WindowsPython Guy, I have one of five scenarios in mind for "distribution":
This is very useful because you have extended the discussion from just compiling to distributing.
Others have commented at length. Suffice it to say I agree.
Yes, this is the most common case where DistUtils can help in program development (as opposed to distribution). If only it were a more common case! JimA

Or use the NT task manager and select "Debug" when right-clicking on a process. Or, for Python programmers, add a quick "win32api.DebugBreak()" call.
ian-will-figure-out-how-to-use-devstudio-long-before-you'll-figure- out-how-to-automate-it-ly y'rs - tim
Ahh - but we already have - David's compile.py knows how to use COM to automate DevStudio builds :-) In fact, with the COM extensions we could create and build a DevStudio project from scratch I believe. But I dont think we should. Just to echo the rest of the articles in this thread: * If someone uses DevStudio to build their apps, it is likely they are targetting Windows only, so distutils is unlikely to offer great advantages. It will be a long time indeed before you can release source code for Windows apps under the assumption that all systems have a C compiler available to build it. * If per chance they _are_ also targetting Unix, then they must also be maintaining seperate Unix build scripts - still no loss here - move the Unix stuff to distutils and either leave DevStudio alone, or move to make based builds in Windows too. Still the users choice, and nothing to lose from what they currently do - only possibly things to gain if they go that route. * As Guido mentioned, on any platform, they should be able to get an extension building in almost no time (just like compile.py pretty-much does now). Windows users in this category are unlikely to be DevStudio wizards, so it is still a win for them. In a nutshell, I cant see _any_ good reasons to offer DevStudio support to distutils. Mark.

Don't even need COM. The project files are just text after all. The wxWindows team uses tmake from Troll Tech to generate makefiles for all the various Windows compilers, Makefile.in for GNU configure, and VC++ workspace/project files. Since it's a perl based tool I couldn't stomach using it myself and since distutils wasn't usable yet I created my own script that can make makefiles for VC++ or Unix based systems and can also drive running make. Works pretty good. -- Robin Dunn Software Craftsman robin@AllDunn.com http://AllDunn.com/robin/ http://AllDunn.com/wxPython/ Check it out!

Hi, I think all we Windows Zopisters have our own build systems ;) I call this good timing ;) Cheers, Anthony Pfrunder

Mark Hammond wrote:
Just to echo the rest of the articles in this thread:
Just to echo your echo...
Yes.
Yes.
In a nutshell, I cant see _any_ good reasons to offer DevStudio support to distutils.
I guess you mean no reason for DevStudio support, but DistUtils still needs Windows compiler (command line) support. Then I agree with this last point. I don't see DistUtils being used except in cases where a Windows and Unix build is required. But that does happen. With wxWindows it happens Big Time. There is still an advantage to coding in DevStudio, makeing the DistUtils files, and pushing the DistUtils button to build multiple versions. JimA

Hi, But is there a way to allow the win32 extensions to build *without* needing Visual Studio MFC code? I have a python script that converts dsw/dsp files into (semi) standard makefiles. These makefiles can be driven to work with MSDev, cygwin, ming or unix by changing a config option. I was hoping to compile the win32 extensions using ming and it worked - up to a point. It needed source from MS Dev which I no longer have :( Cheers, Anthony Pfrunder

But is there a way to allow the win32 extensions to build *without* needing Visual Studio MFC code?
Apart from Pythonwin, the win32 extensions should not require MFC at all.
What specifically were the problems? [Not really distutils specific - just mail me with the response] Mark.

On Thu, 2 Dec 1999, Greg Ward wrote:
<ears tingle> Someone mentioned my name? Sigh. The project file vs. Makefile debate is as old as the world (well, at least as old as MPW =). I haven't looked carefully enough at the innards of distutils to be sure, but glancing at msvccompiler.py, it looks like Perry did the simple thing (and praise him). That said, I'd be surprised if making a project/workspace file would be that hard given Perry's groundwork (and stealing the templates from compile.py). Alas, I have *no time* in the foreseeable future to do this. If someone is interested in doing it, however, I can help them with whatever knowledge I gathered doing compile.py. The hard thing will probably be to support all of the things which Windows developers expect, such as the ability to have Debug and Release builds (and CE builds, and ...). --david

David Ascher writes:
Is it really necessary for people to develop their code using Distutils, as opposed to developing in their preferred environment and then wrapping things up for distribution? If I was debugging a C module, I don't think I'd bother compiling it through the distutils; more likely I'd just write a Makefile.pre.in and Setup.in. -- A.M. Kuchling http://starship.python.net/crew/amk/ run-time. n. (QA testing) the moment when the programmer shouts "Must run!" and disappears. -- Stan Kelly-Bootle, _The Computer Contradictionary_

"Andrew M. Kuchling" wrote:
I am (among other things) a Windows developer, and I would never consider developing a C or C++ module without a project file in Visual C++. The project enables you to create Release and Debug versions. It has an excellent debugger and I even use it. Normally I do without debuggers. It has built-in Windows documentation which I can not do without. It helps navigate the dreadfully complex Windows compiler options. In my view, the program building part of distutils is only relevant to Unix. Which is fine with me, as I run Unix too. JimA

On Thu, 02 Dec 1999, Andrew M. Kuchling wrote:
Absolutely. Using Distutils we have been able to automate the making of our software. I have even just finished a little facility so that Pyfort can make a Fortran extension (although I am going to need community help getting data for the various Fortran compilers to flesh it out.) The important point here is that people who are not NORMALLY developers can BE developers. Despite not having memorized the location of Makefile.pre.in, and stie-packages, or even knowing that they exist, they can make and install extensions. For example, a user need only create a file like this: pyf('rs.pyf', 'rs', '.') name = 'eof_workbench' directory = '.' to make a Fortran extension using input file rs.pyf and loadling librs.a from directory ., picking up .py files to go with it from . and installing it as package eof_workbench. Assuming they call that file SETUP, they do pyfort -s SETUP python setup.py build_ext install and they can immediately use the package eof_workbench. Distutils doesn't have to be comprehensive. What Distutils does is lower the bar for learning how to make extensions.

On Thu, 2 Dec 1999, Andrew M. Kuchling wrote:
I doubt it's *necessary* but I agree with the sentiment that if you're going to be setting up the set of relationships between files, it'd be *nice* to be able to do it just once and have it work for both development & distribution. If you want to write code which is being developed on two operating systems at once, it can get tedious to maintain both the Makefile & project files. I do agree that this is fairly minor in the grand scheme of things, and if I were in charge, I'd put it off until 1.1 =). --david

On 30 November 1999, Ian Searle said:
[Please limit your messages to < 80 columns; I had to reformat it down so it quotes nicely.] First caveat: if building C++ extensions with Distutils actually works, it's a happy accident. I had nothing to do with it. ;-)
I disagree -- the documentation isn't adequate for *anyone*, moron or not. You need to add an 'extra_objects' element to the dictionary that describes how to build your extension. Eg. if you now have: setup (... ext_modules = [('myext', {'sources': ['myext.c', 'support.c'] } )] ) then you will need to change this to setup (... ext_modules = [('myext', {'sources': ['myext.c', 'support.c'], 'extra_objects: ['vendor.o'] } )] ) In the code, see distutils.command.build_ext.BuildExt.build_extensions(). Doesn't look like extra_objects is mentioned in the USAGE file (oops). The usual disclaimer: this has not been used in practice, just tried out in artifical tests. Please let me know if it works or doesn't work.
Ooh, yuck. And here I thought that invoking the MSVC command-line compiler directly was a feature. David Ascher was the original volunteer to write the MSVC interface, and his plan was (I think) to generate a project file and then somehow invoke the build system on that. When Perry Stoll came along with his MSVCCompiler class, David appeared happy to bow out... but maybe his idea needs to be resuscitated? Yuck. You Windows/MSVC experts got any good ideas? Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

[Ian Searle]
[Greg Ward]
Ooh, yuck. And here I thought that invoking the MSVC command-line compiler directly was a feature. ...
It is. The DevStudio debugger doesn't care how an .exe got built (provided you passed the compiler the right switches to get debug info generated). The main Windows project I work on is built by a hand-written makefile run by a homegrown Win32 variant of GNU make; DevStudio plays no role in it unless & until I want to run under the debugger. Goodness, you can even use the DevStudio Build -> Start Debug -> Attach to Process ... menu item to start debugging an arbitrary process already running on the system, source code or not, originally in a DevStudio language or not. If it wasn't compiled with MS-style symbol info, though, you can't do much more than step thru the machine code. Notwithstanding, many hardcore Windows weenies *prefer* doing development under DevStudio. But I'd say that's a topic for the DevUtils SIG <wink>. ian-will-figure-out-how-to-use-devstudio-long-before-you'll-figure- out-how-to-automate-it-ly y'rs - tim

Tim Peters wrote:
Very interesting. I didn't know that. Probably neither does the average Windows developer.
Ooops, not at all what people will want.
Notwithstanding, many hardcore Windows weenies *prefer* doing development under DevStudio. But I'd say that's a topic for the DevUtils SIG <wink>.
I think we may be kidding ourselves. It is not a matter of "prefer". Distutils is not at all compatible with DevStudio. Suppose you add a file to the project in DevStudio. You will have to make the parallel change by hand in Distutils. People use (or should use) _ASSERT(), so there must be Debug and Release versions no matter if the debugger is used or not. Built in to DevStudio, a file edit (at best) in Distutils. And to top it off, you need on-line documentation for Windows anyway, so you need DevStudio or equivalant even if you do learn Distutils. People in PythonDev[:] may put up with this, but if we go to main-stream Windows developers and say "do this our way" they will say "no thanks". Distutils makes sense on Unix where things are done with makefiles. Why do we even care if Windows developers make Python extensions in DevStudio? Jim Ahlstrom

On 07 December 1999, James C. Ahlstrom said:
The whole point of Distutils is to allow Joe Unix Geek to develop module distributions that will build and install painlessly on non-Unix systems, *and* Jane Windows Weenie to develop module distributions that will build and install painlessly on non-Windows systems. If Jane is writing Windows-only extensions, then nobody's forcing her to use Distutils and maintain a setup script (although it would be nice). Ditto for Joe writing Unix-only extensions. That means that Unix geeks can't use makefiles to distribute their Python modules, and it means that Windows weenies can't use Microsoft's project files to distribute their Python modules. The bad news is that if you really really want to use the local convention for automated building (both of which have advantages), you have to maintain a separate setup.py script. The good news is that setup.py is portable and written in a real programming language. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

Greg Ward:
Good summary of the distutil charter. I would add a third objective: *users* on all platforms are tired of having to understand the intricate details of building and installing Python modules, and want module distributions that "just work". This includes platform specific distributions. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Tim]
[James C. Ahlstrom]
Very interesting. I didn't know that. Probably neither does the average Windows developer.
They don't need to know it. I was explaining an extreme. In normal usage, the worst that's required is that you open the .exe you want to debug from within DevStudio and click "Go".
If it wasn't compiled with MS-style symbol info, though, you can't do much more than step thru the machine code.
Ooops, not at all what people will want.
Again explaining an extreme. If you use MS's compiler to compile your stuff from a cmdline, you get MS-style symbol info, and debugging is the same as if you had compiled it using DevStudio (DevStudio compilation is just a massive GUI around a cmdline compiler & linker!). Of *course* MS's debugger doesn't understand non-MS symbol info, any more than gdb understands MS's style. So don't do that <wink>.
I think we may be kidding ourselves. It is not a matter of "prefer".
Sure it is. DevStudio also has a menu entry to generate a makefile (of MS's odd nmake flavor). I often set up personal projects using DevStudio, and then switch to a generated makefile so I can run overnight builds and tests from .bat scripts.
The difference between Debug and Release is entirely in switches passed to the compiler and linker; makefiles can handle that fine based on target name (or envars, or anything else a makefile can see; have DevStudio generate a makefile to see how MS does it). As I hope will become clearer soon, though, I don't grasp why any of this is a Distutils issue.
Sorry, but I don't understand where this is coming from. As a Windows Python Guy, I have one of five scenarios in mind for "distribution": 1) The audience is purely Windows developers. Then I ship a zip file containing source and an MS project file, and leave Distutils out of it entirely. 2) The audience is purely Windows end users. Then I ship pre-compiled binaries, because they can't possibly deal with compilation themselves (they don't have compilers!). In this case I'm not sure whether Distutils can help (it would have to become an Installshield clone to meet expectations); but, ragardless, compilation is again not an issue for distribution. 3) The audience is Python programmers "in general", or the audience is me, and the product is a C extension module. This is where I expect to use Distutils, and hope that others do too. An MS project file doesn't do my Unix friends any good, any more than their fiddly platform-dependent gmake files help me, and binaries suck for similar reasons. Most interesting extension modules I've seen have been written by Unix weenies -- and *when you're lucky* their home page has a contributed Windows .pyd just two releases out of date <wink/frown>. 4) The audience is Python end-users "in general", and the product is pure Python. I think this is the most important one for Distutils to address, and compilation isn't a part of it. So far, though, what Gordon is doing seems more appropriate than what Distutils has been up to. I hope his work gets folded into this. 5) The audience is Python end-users "in general", and the product is some mix of Python and C extension modules. This is hard. You have to ship binaries to meet Windows expectations; you have to ship source to meet Unixish ones; and the more popular Linux gets, the more the Unix dweebs will be left whining without a clue about the changing nature of their customer base <0.7 wink -- but over on Python-Help the sight of a Linux user who doesn't know how to compile is no longer rare>.
Main-stream Windows developers have no way at all to distribute a x-platform app now, so their choice is between "our way" and "no way". If they're not interested in x-platform distribution, I indeed don't see what Distutils *could* offer them in the way of helpful code (except in case #4 above -- where compilation (& so also DevStudio) isn't an issue).
I don't care at all, provided they have no interest in sharing their work with non-Windows people. If they do want to share, they need to do more work (their Windows binaries aren't going to run on a SPARC <wink>), and Distutils can help with that. life-will-be-simpler-when-windows-is-the-only-os-ly y'rs - tim

Or who don't WANT to compile and would "much rather just use an RPM." I've had several of both types of people using wxPython. -- Robin Dunn Software Craftsman robin@AllDunn.com http://AllDunn.com/robin/ http://AllDunn.com/wxPython/ Check it out!

On 08 December 1999, Tim Peters said:
It could be a minor Distutils issue: I've toyed with the idea of "--debug" or "--optimize" flags (or maybe "--release"?) to the 'build_ext' command that would translate to appropriate compiler switches. I know how to do this for the "generic Unix C compiler" implemented by the UnixCCompiler class, but someone else would have to do it it for MSVCCompiler.
Probably appropriate, although in my more megalomaniacal moments I tend to think, "Who needs make? Who needs IDEs? You can do it ALL with Python, a C compiler, and Distutils!" (In an idle moment the other day, I started thinking about a Distutils command to build Perl extensions... I got over it pretty quickly though.)
Creating binary distributions is in the plans, although they'll be called "built distributions" because an RPM that contains only .py (and maybe .pyc) files isn't really "binary". The plan decided on at that fateful Developer's Day session *over a year ago* (wow!) was that we'd figure out something for Windows with WISE Installer (or whatever it's called). I think the theory was that somehow we could write Python code that would magically generate a Windows installer. I certainly haven't done anything about that, although at some point I expect I'll have to walk downstairs and grill Guido about Windows installers for Python until he gives up and tells me to bug someone else. FWIW, the interface I envision is *something* like this: $ ./setup.py bdist --format=rpm [...create a spec file, run rpm -B...] $ ./setup.py bdist --format=wise error: can't create Windows installer under Unix Oops! Let's try that second one again C:\> ./setup.py bdist --format=wise [...somehow magically create a WISE installer...] Hmmm, this is starting to look like yet another argument for system-wide and user-specific config files to supply default values for things like the bdist format. (And, for that matter, the dist format, which can be one of 'gztar', 'ztar', 'tar', or 'zip'.)
Here the distinction between "source distributions" and "build distributions" gets fuzzy. I actually think it's important and worthwhile to provide "built" distributions of pure Python distributions -- not for technical reasons, but for social/political/usability reasons. Eg. if I can install a mixed C/Python distribution from an RPM or a WISE installer, then I should be able to do the same for a pure Python distribution. You and I will know that it's a waste of disk space to offer both versions, and a waste of machine time to have dist-bots churning out pure Python built distributions from source distributions, but it's necessary for a consistent user interface on each platform supported by the "bdist" command. I assume that "what Gordon is doing" is the Python archive stuff? My attitude towards that is the same as it is towards all of these funky new things coming down the pike: interesting, keep an eye on it, maybe try it out if I get a few minutes... *but* Distutils support will have to wait until there is One Standard Way of doing things. The whole point of Distutils is to be the One Standard Way; I certainly don't claim that copying .py files into directories and compiling them to .pyc files is the only way to install Python module distributions, but until these issues (archive format, imputils, etc.) are worked out and finalized then Distutils won't (officially) support them. Of course, I think it would be really cool if competing archive formats were presented to the world with implementations in the form of Distutils command classes, but that's probably my megalomaniacal tendencies coming out again.
As Guido said on python-dev, a Distutils "appdist" command is probably the answer. Or maybe "appbdist", I dunno. I'm not sure if separate commands are really needed, or if the "build", "install", "dist", and "bdist" commands will grow up to recognize Python applications rather than just module distributions. That's a bit of a ways off, though: I said at the beginning that I want to handle module distributions first, and then worry about applications. Maybe by that time, the "Python archive format", imputils, one-file-Python-distributions, etc. will be more stable. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

Tim Peters wrote:
Yes, understood. Extremes are good for sharpening issues.
Yes, we write the makefiles by hand so we can build from Korn shell scripts. Microsoft's eccentric nmake format is a constant annoyance. But mostly Windows people stay in DevStudio.
It means that even if I need a Unix build too, I usually develop in DevStudio because of the on-line help. But for small projects, I develop on Linux because of the large setup annoyance of DevStudio. In other words, I have a poor memory and like built-in help.
As a WindowsPython Guy, I have one of five scenarios in mind for "distribution":
This is very useful because you have extended the discussion from just compiling to distributing.
Others have commented at length. Suffice it to say I agree.
Yes, this is the most common case where DistUtils can help in program development (as opposed to distribution). If only it were a more common case! JimA

Or use the NT task manager and select "Debug" when right-clicking on a process. Or, for Python programmers, add a quick "win32api.DebugBreak()" call.
ian-will-figure-out-how-to-use-devstudio-long-before-you'll-figure- out-how-to-automate-it-ly y'rs - tim
Ahh - but we already have - David's compile.py knows how to use COM to automate DevStudio builds :-) In fact, with the COM extensions we could create and build a DevStudio project from scratch I believe. But I dont think we should. Just to echo the rest of the articles in this thread: * If someone uses DevStudio to build their apps, it is likely they are targetting Windows only, so distutils is unlikely to offer great advantages. It will be a long time indeed before you can release source code for Windows apps under the assumption that all systems have a C compiler available to build it. * If per chance they _are_ also targetting Unix, then they must also be maintaining seperate Unix build scripts - still no loss here - move the Unix stuff to distutils and either leave DevStudio alone, or move to make based builds in Windows too. Still the users choice, and nothing to lose from what they currently do - only possibly things to gain if they go that route. * As Guido mentioned, on any platform, they should be able to get an extension building in almost no time (just like compile.py pretty-much does now). Windows users in this category are unlikely to be DevStudio wizards, so it is still a win for them. In a nutshell, I cant see _any_ good reasons to offer DevStudio support to distutils. Mark.

Don't even need COM. The project files are just text after all. The wxWindows team uses tmake from Troll Tech to generate makefiles for all the various Windows compilers, Makefile.in for GNU configure, and VC++ workspace/project files. Since it's a perl based tool I couldn't stomach using it myself and since distutils wasn't usable yet I created my own script that can make makefiles for VC++ or Unix based systems and can also drive running make. Works pretty good. -- Robin Dunn Software Craftsman robin@AllDunn.com http://AllDunn.com/robin/ http://AllDunn.com/wxPython/ Check it out!

Hi, I think all we Windows Zopisters have our own build systems ;) I call this good timing ;) Cheers, Anthony Pfrunder

Mark Hammond wrote:
Just to echo the rest of the articles in this thread:
Just to echo your echo...
Yes.
Yes.
In a nutshell, I cant see _any_ good reasons to offer DevStudio support to distutils.
I guess you mean no reason for DevStudio support, but DistUtils still needs Windows compiler (command line) support. Then I agree with this last point. I don't see DistUtils being used except in cases where a Windows and Unix build is required. But that does happen. With wxWindows it happens Big Time. There is still an advantage to coding in DevStudio, makeing the DistUtils files, and pushing the DistUtils button to build multiple versions. JimA

Hi, But is there a way to allow the win32 extensions to build *without* needing Visual Studio MFC code? I have a python script that converts dsw/dsp files into (semi) standard makefiles. These makefiles can be driven to work with MSDev, cygwin, ming or unix by changing a config option. I was hoping to compile the win32 extensions using ming and it worked - up to a point. It needed source from MS Dev which I no longer have :( Cheers, Anthony Pfrunder

But is there a way to allow the win32 extensions to build *without* needing Visual Studio MFC code?
Apart from Pythonwin, the win32 extensions should not require MFC at all.
What specifically were the problems? [Not really distutils specific - just mail me with the response] Mark.

On Thu, 2 Dec 1999, Greg Ward wrote:
<ears tingle> Someone mentioned my name? Sigh. The project file vs. Makefile debate is as old as the world (well, at least as old as MPW =). I haven't looked carefully enough at the innards of distutils to be sure, but glancing at msvccompiler.py, it looks like Perry did the simple thing (and praise him). That said, I'd be surprised if making a project/workspace file would be that hard given Perry's groundwork (and stealing the templates from compile.py). Alas, I have *no time* in the foreseeable future to do this. If someone is interested in doing it, however, I can help them with whatever knowledge I gathered doing compile.py. The hard thing will probably be to support all of the things which Windows developers expect, such as the ability to have Debug and Release builds (and CE builds, and ...). --david

David Ascher writes:
Is it really necessary for people to develop their code using Distutils, as opposed to developing in their preferred environment and then wrapping things up for distribution? If I was debugging a C module, I don't think I'd bother compiling it through the distutils; more likely I'd just write a Makefile.pre.in and Setup.in. -- A.M. Kuchling http://starship.python.net/crew/amk/ run-time. n. (QA testing) the moment when the programmer shouts "Must run!" and disappears. -- Stan Kelly-Bootle, _The Computer Contradictionary_

"Andrew M. Kuchling" wrote:
I am (among other things) a Windows developer, and I would never consider developing a C or C++ module without a project file in Visual C++. The project enables you to create Release and Debug versions. It has an excellent debugger and I even use it. Normally I do without debuggers. It has built-in Windows documentation which I can not do without. It helps navigate the dreadfully complex Windows compiler options. In my view, the program building part of distutils is only relevant to Unix. Which is fine with me, as I run Unix too. JimA

On Thu, 02 Dec 1999, Andrew M. Kuchling wrote:
Absolutely. Using Distutils we have been able to automate the making of our software. I have even just finished a little facility so that Pyfort can make a Fortran extension (although I am going to need community help getting data for the various Fortran compilers to flesh it out.) The important point here is that people who are not NORMALLY developers can BE developers. Despite not having memorized the location of Makefile.pre.in, and stie-packages, or even knowing that they exist, they can make and install extensions. For example, a user need only create a file like this: pyf('rs.pyf', 'rs', '.') name = 'eof_workbench' directory = '.' to make a Fortran extension using input file rs.pyf and loadling librs.a from directory ., picking up .py files to go with it from . and installing it as package eof_workbench. Assuming they call that file SETUP, they do pyfort -s SETUP python setup.py build_ext install and they can immediately use the package eof_workbench. Distutils doesn't have to be comprehensive. What Distutils does is lower the bar for learning how to make extensions.

On Thu, 2 Dec 1999, Andrew M. Kuchling wrote:
I doubt it's *necessary* but I agree with the sentiment that if you're going to be setting up the set of relationships between files, it'd be *nice* to be able to do it just once and have it work for both development & distribution. If you want to write code which is being developed on two operating systems at once, it can get tedious to maintain both the Makefile & project files. I do agree that this is fairly minor in the grand scheme of things, and if I were in charge, I'd put it off until 1.1 =). --david
participants (11)
-
Andrew M. Kuchling
-
Anthony Pfrunder
-
David Ascher
-
Greg Ward
-
Guido van Rossum
-
Ian Searle
-
James C. Ahlstrom
-
Mark Hammond
-
Paul F. Dubois
-
Robin Dunn
-
Tim Peters