I'm currently working at the Cygwin-Compiler class on Win32, there I found a problem with "bdist". When I have build all extensions with my compiler class and then try to use "bdist", the msvc compiler class starts and fails (I don't have it on my machine.) I think there are two problems. First the system seems to check the filedates with the objectfile-extensions of the standard compiler ( msvc ".obj" , cygwin ".o" ). So it doesn't find my object files, and tries to recreate them, but uses a wrong compiler. I think it should be possible to configure the compiler also for "bdist". (If this is already possible? Please explain how.) Also I think "bdist" does a "build" if there aren't already the right files. In this case "bdist" should accept *all* parameters which "build" accepts. The second problem. Why "bdist" doesn't check only the source files against the finished extension modules? Currently it seems to check the source files against the compiled object files. Then there is something else concerning the compiler option. If the user uses help he get the information that he can use the compiler option, but how can he know which compilers(classes) are available. There should be a possibility to get a list. (In ccompiler.py we have such list, we only had print it.) kind regards Rene Liebscher
I'm currently working at the Cygwin-Compiler class on Win32, there I found a problem with "bdist".
When I have build all extensions with my compiler class and then try to use "bdist", the msvc compiler class starts and fails (I don't have it on my machine.)
I think there are two problems.
First the system seems to check the filedates with the objectfile-extensions of the standard compiler ( msvc ".obj" , cygwin ".o" ). So it doesn't find my object files, and tries to recreate them, but uses a wrong compiler. I think it should be possible to configure the compiler also for "bdist". (If this is already possible? Please explain how.) Also I think "bdist" does a "build" if there aren't already the right files. In this case "bdist" should accept *all* parameters which "build" accepts.
You could use: setup.py bdist build_ext --compiler Cygwin In this way you can use all the options build_ext accepts. You have to be carefull with the short options: setup.py bdist build_ext -f is NOT interpreted as build_ext --force but as bdist --format Thomas
On 23 May 2000, Thomas Heller said:
You could use: setup.py bdist build_ext --compiler Cygwin
Careful -- if you're using code earlier than last night, I think "build_ext" should come first. That particular bit of bogosity should be fixed with the config file changes, though. One more reason to be following the CVS tree. (Whaddya mean, time for a new release? ;-)
You have to be carefull with the short options: setup.py bdist build_ext -f is NOT interpreted as build_ext --force but as bdist --format
That's a feature. I think. ;-) Greg -- Greg Ward - "always the quiet one" gward@python.net http://starship.python.net/~gward/ Could I have a drug overdose?
On 23 May 2000, Rene Liebscher said:
First the system seems to check the filedates with the objectfile-extensions of the standard compiler ( msvc ".obj" , cygwin ".o" ). So it doesn't find my object files, and tries to recreate them, but uses a wrong compiler. I think it should be possible to configure the compiler also for "bdist".
See Thomas' reply, and my reply to him, for one way of doing it.
(If this is already possible? Please explain how.) Also I think "bdist" does a "build" if there aren't already the right files. In this case "bdist" should accept *all* parameters which "build" accepts.
The trouble with that is that lots of commands run other commands. Currently there's very little formal notion of "sub-commands" -- if command X wants to replicate an option from command Y, it has to be done manually. This is deliberate design minimalism; the idea is to keep each command's scope under control. Of course, that leads to awkward command lines. Hence, config files. If you're running last night's code ("cvs update" -- it's your friend!), try this: create "<prefix>/pydistutils.cfg" (on Windows and Mac OS -- it's different for Unix) and put the following in it [build_ext] compiler=cygwin and then you never have to worry about specifying the compiler again: it's a system default. Of course, be sure to read all the debugging output that is deliberately still in the code: I want everyone to share the pain!
The second problem. Why "bdist" doesn't check only the source files against the finished extension modules? Currently it seems to check the source files against the compiled object files.
Good point, I think. I'm trying *not* to emulate everything you'd do in a real Makefile, since Distutils != make. But this could be a useful addition, if I understand what you're saying -- I'll have to think about it. (Hint: working patches are an excellent way to prod me into thinking about something. They also clarify wonderfully what exactly you're talking about.)
Then there is something else concerning the compiler option. If the user uses help he get the information that he can use the compiler option, but how can he know which compilers(classes) are available. There should be a possibility to get a list. (In ccompiler.py we have such list, we only had print it.)
That's definitely a good idea. Wasn't such a big deal in the "old days", when we only had UnixCCompiler and MSVCCompiler. Those days appear numbered, though... How does a "--help-compilers" global option sound? (Kind of on the same level as "--help-commands".) It's a bit klugey, but I can't think of a better option offhand. Greg -- Greg Ward - programmer-at-large gward@python.net http://starship.python.net/~gward/ Heisenberg may have slept here.
On Tue, 23 May 2000, Greg Ward wrote:
try this: create "<prefix>/pydistutils.cfg" (on Windows and Mac OS -- it's different for Unix) and put the following in it
[build_ext] compiler=cygwin
As Python grows support for additional compilers, the Python installation should set this up with the proper compiler information. The admin can change the file after installation if necessary. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
Greg Ward wrote:
The second problem. Why "bdist" doesn't check only the source files against the finished extension modules? Currently it seems to check the source files against the compiled object files.
Good point, I think. I'm trying *not* to emulate everything you'd do in a real Makefile, since Distutils != make. But this could be a useful addition, if I understand what you're saying -- I'll have to think about it. (Hint: working patches are an excellent way to prod me into thinking about something. They also clarify wonderfully what exactly you're talking about.)
You have the following files: XXXX.c => XXXX.o => XXXX.pyd (on Windows) If you remove the object files or use a different compiler, which wants to see XXXX.obj instead XXXX.o (msvc <=> cygwin), it would try to rebuild all, even if the XXXX.pyd-file is up-to-date. I have the place in the build_ext.py file to change this, the patch is attached. I simple let check it before I start the compiler (which only can check XXXX.c<=>XXXX.o and XXXX.o<=>XXXX.pyd)
How does a "--help-compilers" global option sound? (Kind of on the same level as "--help-commands".) It's a bit klugey, but I can't think of a better option offhand.
There are more options which could need such a list (bdist format), perhaps we should introduce something like --help-option XXXXX ( XXXXX = compiler,format,...) It could scan all commands for a option with this name and a more extensive description (string or Callable, which generates a string on the fly.) Kind regards Rene Liebscher diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/build_ext.py distutils.patched/distutils/command/build_ext.py --- distutils.orig/distutils/command/build_ext.py Thu May 25 03:10:04 2000 +++ distutils.patched/distutils/command/build_ext.py Thu May 25 18:47:24 2000 @@ -12,7 +12,7 @@ from types import * from distutils.core import Command from distutils.errors import * - +from distutils.dep_util import newer_group # An extension name is just a dot-separated list of Python NAMEs (ie. # the same as a fully-qualified module name). @@ -285,7 +285,28 @@ "a list of source filenames") % extension_name sources = list (sources) - self.announce ("building '%s' extension" % extension_name) + fullname = self.get_ext_fullname (extension_name) + if self.inplace: + # ignore build-lib -- put the compiled extension into + # the source tree along with pure Python modules + + modpath = string.split (fullname, '.') + package = string.join (modpath[0:-1], '.') + base = modpath[-1] + + build_py = self.find_peer ('build_py') + package_dir = build_py.get_package_dir (package) + ext_filename = os.path.join (package_dir, + self.get_ext_filename(base)) + else: + ext_filename = os.path.join (self.build_lib, + self.get_ext_filename(fullname)) + + if not newer_group(sources,ext_filename,'newer'): + self.announce ("skipping '%s' extension (up-to-date)" % extension_name) + continue # 'for' loop over all extensions + else: + self.announce ("building '%s' extension" % extension_name) # First step: compile the source code to object files. This # drops the object files in the current directory, regardless @@ -356,23 +377,6 @@ extra_args.append ('/IMPLIB:' + implib_file) self.mkpath (os.path.dirname (implib_file)) # if MSVC - - fullname = self.get_ext_fullname (extension_name) - if self.inplace: - # ignore build-lib -- put the compiled extension into - # the source tree along with pure Python modules - - modpath = string.split (fullname, '.') - package = string.join (modpath[0:-1], '.') - base = modpath[-1] - - build_py = self.find_peer ('build_py') - package_dir = build_py.get_package_dir (package) - ext_filename = os.path.join (package_dir, - self.get_ext_filename(base)) - else: - ext_filename = os.path.join (self.build_lib, - self.get_ext_filename(fullname)) self.compiler.link_shared_object (objects, ext_filename, libraries=libraries,
On 25 May 2000, Rene Liebscher said:
You have the following files:
XXXX.c => XXXX.o => XXXX.pyd
(on Windows)
If you remove the object files or use a different compiler, which wants to see XXXX.obj instead XXXX.o (msvc <=> cygwin), it would try to rebuild all, even if the XXXX.pyd-file is up-to-date.
Yup, good explanation. Now that you put it this way, this has bugged me in the past. The problem, of course, is that Distutils != make: we don't have a dependency graph, we're not doing topological sorts, etc. We just compare this timestamp to that timestamp. So we end up doing stupid stuff occasionally.
I have the place in the build_ext.py file to change this, the patch is attached. I simple let check it before I start the compiler (which only can check XXXX.c<=>XXXX.o and XXXX.o<=>XXXX.pyd)
...and this is a lovely hack: fixes the problem at hand without going overboard. Checked in -- thanks!
There are more options which could need such a list (bdist format), perhaps we should introduce something like --help-option XXXXX ( XXXXX = compiler,format,...) It could scan all commands for a option with this name and a more extensive description (string or Callable, which generates a string on the fly.)
Good point. This should be some sort of "extra interface" that certain commands will implement: they need to announce which of their options have a help option, what that help option should be called (default: "--help-" + option_name), and what the values are. If you want to give this a shot, go for it -- I'm kept busy enough just processing incoming patches and trying to write the docs! Greg -- Greg Ward - Linux geek gward@python.net http://starship.python.net/~gward/ A man wrapped up in himself makes a very small package.
Greg Ward wrote:
On 25 May 2000, Rene Liebscher said:
There are more options which could need such a list (bdist format), perhaps we should introduce something like --help-option XXXXX ( XXXXX = compiler,format,...) It could scan all commands for a option with this name and a more extensive description (string or Callable, which generates a string on the fly.)
Good point. This should be some sort of "extra interface" that certain commands will implement: they need to announce which of their options have a help option, what that help option should be called (default: "--help-" + option_name), and what the values are. If you want to give this a shot, go for it -- I'm kept busy enough just processing incoming patches and trying to write the docs!
Perhaps, someone else has a better overview and wants to start it. At least it would be helpful to know all options which could need more documentation. So please, extend this list: * build_ext, compiler : list of available compilers (and their long names like 'Microsoft Visual C++ 5.X and higher' or so) * bdist, format : list + explanation (gztar : creates XXXX.tar.gz) Kind regards Rene Liebscher
On 26 May 2000, Rene Liebscher said:
Perhaps, someone else has a better overview and wants to start it. At least it would be helpful to know all options which could need more documentation.
So please, extend this list:
* build_ext, compiler : list of available compilers (and their long names like 'Microsoft Visual C++ 5.X and higher' or so)
* bdist, format : list + explanation (gztar : creates XXXX.tar.gz)
That's all I can think of right now. Here's a Stupid Unix Trick (TM) to get help for all Distutils commands: ./setup.py --help `./setup.py --help-commands | awk '/^ [a-z]/ {print $1}'` (err, assuming setup.py is executable and has a proper #! line). Scanning the output of this, I see: build, build_ext, build_clib --compiler sdist --formats bdist --formats written that way because the three commands with --compiler have the same list of compilers, but the two options with --format have different lists of formats. So yes, you got them all. But there could well be more in future, so I think it's worth adding a smidge of bureaucracy to grease the wheels here. Here's a strawman proposal for "custom help option" interface. The above five command classes (and any future commands with similar requirements) would define a class attribute 'help_options': class build (Command): # ... help_options = [('compiler', show_compilers, "list available compilers (for --compiler option)")] When 'parse_command_line()' (in Distribution) sees this list, it would mangle each entry as appropriate to feed into the FancyGetopt parser, so that the following would appear in the help for "build": [...] --compiler (-c) specify the compiler type --help-compiler list available compilers (for --compiler option) [...] When --help-compiler is seen on the command line, the function 'show_compilers()' would be called. (Presumably this would really be provided by the ccompiler module, since the same function would be needed in three commands.) This has to be a function object, not a method (or method name), since the command class hasn't been instantiated when we're just parsing the command line. Presence of any --help option should terminate processing -- see the code in 'parse_command_line()'. Rene, can you whip up a patch for this? Thanks! Greg -- Greg Ward - nerd gward@python.net http://starship.python.net/~gward/ I guess it was all a DREAM ... or an episode of HAWAII FIVE-O ...
Greg Ward wrote:
Scanning the output of this, I see:
build, build_ext, build_clib --compiler sdist --formats bdist --formats
written that way because the three commands with --compiler have the same list of compilers, but the two options with --format have different lists of formats. So yes, you got them all.
If I try 'help' to sdist and bdist, I get 'formats' and 'format'. I think we should change this so both are called 'format'.
Rene, can you whip up a patch for this? Thanks!
I will try to track down all your options processing and then send you a patch, but it will take a while. Kind regards Rene Liebscher
On 29 May 2000, Rene Liebscher said:
If I try 'help' to sdist and bdist, I get 'formats' and 'format'. I think we should change this so both are called 'format'.
Not quite. First, we have to change it so both of them can take multiple formats. For obscure, annoying reasons (which I have since forgotten and which may have changed), "bdist" can only take one format -- ie. the syntax is inconsistent because the semantics are inconsistent. Fix the semantics, and we can fix the syntax. Oh, now I remember: because many formats map to the "bdist_dumb" command, and because the command life-cycle used to be strictly linear. That has been fixed, so it should be possible to run "bdist_dumb" multiple times with multiple formats. (The option is to complicate "bdist_dumb" so it can generate multiple formats in one go; I dislike this because it means duplicating the "multiple formats" logic in "bdist" and "bdist_dumb".) I've added this to my todo list -- have to think about the consequences a bit, but offhand it should work. Greg -- Greg Ward - Linux nerd gward@python.net http://starship.python.net/~gward/ Never try to outstubborn a cat.
participants (4)
-
Fred L. Drake -
Greg Ward -
Rene Liebscher -
Thomas Heller