[Numpy-discussion] numpy build system questions for use in another project (fwrap)

[Kurt Smith]

On Fri, Apr 9, 2010 at 2:25 AM, David <david at silveregg.co.jp> wrote:
> On 04/07/2010 11:52 AM, Kurt Smith wrote:
>> Briefly, I'm encountering difficulties getting things working in numpy
>> distutils for fwrap's build system.
>>
>> Here are the steps I want the build system to accomplish:
>>
>> 1) Compile a directory of Fortran 90 source code -- this works.
>>      - The .mod files generated by this compilation step are put in the
>> build directory.
>
> This is difficult - fortran modules are a PITA from a build perspective.
> Many compilers don't seem to have a way to control exactly where to put
> the generated .mod, so the only way I am aware of to control this is to
> cwd the process into the build directory...

>From what I can tell, numpy's distutils already takes care of putting
the .mod files in the build directory (at least for gfortran and
ifort) by manually moving them there -- see
numpy.distutils.command.build_ext.build_ext.build_extension, after the
line "if fmodule_source".  This is fine for my purposes.  All I need
is the project's .mod files put in one place before the next step.

>
> This was also a problem when I worked on fortran support for waf (see
> http://groups.google.com/group/waf-users/browse_thread/thread/889e2a5e5256e420/84ee939e93c9e30f?lnk=gst&q=fortran+modules#84ee939e93c9e30f
> )

Yes, I saw that thread and it dissuaded me from pursuing waf for now.
I'd like to take a stab at it sometime down the road, though.

>
>>
>> My problem is in instantiating numpy.distutils.config such that it is
>> appropriately configured with command line flags.
>>
>> I've tried the following with no success:
>>
>> ('self' is a build_ext instance)
>> cfg = self.distribution.get_command_obj('config')
>> cfg.initialize_options()
>> cfg.finalize_options()  # doesn't do what I hoped it would do.
>>
>> This creates a config object, but it doesn't use the command line
>> flags (e.g. --fcompiler=gfortran doesn't affect the fortran compiler
>> used).
>
> Why don't you do the testing in config ? That's how things are done

Fortran's .mod files are essentially compiler-generated header files;
fwrap needs to use these 'headers' to get type information so it can
figure out how the C types and Fortran types match up.  Fwrap then
generates the config files with this information and compiles the
wrappers with the other source files.

I could make it a requirement that the user supply the .mod files to
fwrap before calling 'python setup.py build_ext', but that might be a
big mess when you take into account compiler flags that modify default
type sizes, etc.  Maybe it would work fine; I'm not sure whether
different compiler flags used by distutils would change the .mod files
and break the type info, though. And the source code would be compiled
twice -- once to get the .mod files, and again when distutils compiles
everything with the right flags which would be suboptimal for large
projects.

(Note: I've looked into this a bit, and it seems like it might work
alright -- you can use '-fsyntax-only' in gfortran and g95, and
'-syntax-only' in ifort, for example, and that generates the .mod
files.  I'll look into it some more...)

It would be a PITA to test for every kind-type-parameter that someone
might use in the config stage.  As you probably know, Fortran 9x
allows types like this:

real(kind=selected_real_kind(10,100)) :: ffed_real

Which means I'd have to do exhaustive testing for every combination of
arguments to selected_real_kind() (up to some limit).  This would be
*extremely* slow.  I could do a binary search, but it's not something
I'd like to pursue if it's avoidable.  There are some other issues
that I'm not mentioning that make this approach a pain, too.

> normally, unless you have a reason to do otherwise. Concerning the
> --fcompiler option, how do you pass --fcompiler (what is the exact list
> of commands you used to call distutils here) ? Generally, given the
> general command/option screw up, the only real solution really is to
> pass the options to each command, and hope each one is the same.
>

The commandline is 'python setup.py build_ext --inplace
--fcompiler=gfortran' (or --fcompiler=intelem).  When I pass
--fcompiler=gfortran, the try_compile() command ignores it, searches
for a fortran compiler and finds ifort instead and uses it, which
breaks everything.

Is there an example in numpy.distutils that shows how to pass the
commandline options to each command?

I'm thinking of rolling my own 'try_compile()' function as part of a
custom build_ext command class that suits my purposes.  We'll see how
far that gets me.

>> Any pointers?  More generally -- seeing the above, any ideas on how to
>> go about doing what I'm trying to do better?
>
> Not really, that's how you are supposed to do things with distutils,
>

Then I can see why you'd like to use scons & waf instead.  Thanks for
the comments,

Kurt