[Numpy-discussion] numpy build system questions for use in another project (fwrap)

[David]

On 04/10/2010 03:02 AM, Kurt Smith wrote:
> On Fri, Apr 9, 2010 at 2:25 AM, David<david at silveregg.co.jp>  wrote:
>> On 04/07/2010 11:52 AM, Kurt Smith wrote:
>>> Briefly, I'm encountering difficulties getting things working in numpy
>>> distutils for fwrap's build system.
>>>
>>> Here are the steps I want the build system to accomplish:
>>>
>>> 1) Compile a directory of Fortran 90 source code -- this works.
>>>       - The .mod files generated by this compilation step are put in the
>>> build directory.
>>
>> This is difficult - fortran modules are a PITA from a build perspective.
>> Many compilers don't seem to have a way to control exactly where to put
>> the generated .mod, so the only way I am aware of to control this is to
>> cwd the process into the build directory...
>
>> From what I can tell, numpy's distutils already takes care of putting
> the .mod files in the build directory (at least for gfortran and
> ifort) by manually moving them there -- see
> numpy.distutils.command.build_ext.build_ext.build_extension, after the
> line "if fmodule_source".  This is fine for my purposes.  All I need
> is the project's .mod files put in one place before the next step.

Moving files is not good for various reasons - I think it contribute to 
making the build more fragile, and it may cause race conditions for // 
builds (I am not sure about that last point, I think it is also OS 
dependent). I agree that it is not the main worry at that point, though.

>
> I could make it a requirement that the user supply the .mod files to
> fwrap before calling 'python setup.py build_ext', but that might be a
> big mess when you take into account compiler flags that modify default
> type sizes, etc.  Maybe it would work fine; I'm not sure whether
> different compiler flags used by distutils would change the .mod files
> and break the type info, though. And the source code would be compiled
> twice -- once to get the .mod files, and again when distutils compiles
> everything with the right flags which would be suboptimal for large
> projects.
>
> (Note: I've looked into this a bit, and it seems like it might work
> alright -- you can use '-fsyntax-only' in gfortran and g95, and
> '-syntax-only' in ifort, for example, and that generates the .mod
> files.  I'll look into it some more...)
>
> It would be a PITA to test for every kind-type-parameter that someone
> might use in the config stage.  As you probably know, Fortran 9x
> allows types like this:
>
> real(kind=selected_real_kind(10,100)) :: ffed_real
>
> Which means I'd have to do exhaustive testing for every combination of
> arguments to selected_real_kind() (up to some limit).  This would be
> *extremely* slow.

I did not know about this feature, but I think it can be solved 
relatively easily with type checking, and it would not be that slow if 
you allow "expected size". Also, the code for this is already there in 
numpy/distutils/commands/config.py, and is relatively independent of the 
rest of distutils (you only need to give _compile).

>
>> normally, unless you have a reason to do otherwise. Concerning the
>> --fcompiler option, how do you pass --fcompiler (what is the exact list
>> of commands you used to call distutils here) ? Generally, given the
>> general command/option screw up, the only real solution really is to
>> pass the options to each command, and hope each one is the same.
>>
>
> The commandline is 'python setup.py build_ext --inplace
> --fcompiler=gfortran' (or --fcompiler=intelem).  When I pass
> --fcompiler=gfortran, the try_compile() command ignores it, searches
> for a fortran compiler and finds ifort instead and uses it, which
> breaks everything.

Yes, only build_ext knows about --fcompiler, and try_compile comes from 
configure command. That's one aspect of the utterly broken command 
design of distutils, that's why you need to say something like python 
setup.py config --fcompiler=gfortran build_ext --fcompiler=gfortran.

>
> Is there an example in numpy.distutils that shows how to pass the
> commandline options to each command?

It is not possible AFAIK - you only know about --fcompiler in 
build_ext.finalize_options call, and at that point, config may already 
have run.

>
> I'm thinking of rolling my own 'try_compile()' function as part of a
> custom build_ext command class that suits my purposes.  We'll see how
> far that gets me.

That's another solution.

cheers,

David