f2py: sharing F90 module data between modules
I have a suite of fortran modules that I want to wrap with f2py
independently (so they appear to python as seperate imports) but
where each module has access to another fortran module (which
contains global data that is shared between the suite of fortran
modules). I currently compile all the fortran modules, including the
global data, in a single f2py statement so all the fortran code gets
imported in a single statement
import tsim
where tsim is the python name of the aggregated modules. What I would
like to do is as below, where fm1, fm2 are two example modules that
will be called from python, and fm3 has the global data shared
between fm1, fm2.
On Tue, February 12, 2008 7:52 am, Garry Willgoose wrote:
I have a suite of fortran modules that I want to wrap with f2py independently (so they appear to python as seperate imports) but where each module has access to another fortran module (which contains global data that is shared between the suite of fortran modules). I currently compile all the fortran modules, including the global data, in a single f2py statement so all the fortran code gets imported in a single statement
The source of this issue boils down to http://bugs.python.org/issue521854 according to which makes your goal unachivable because of how Python loads shared libraries *by default*, see below.
I guess the question is there any way that I can get fm3 to be shared between fm1 and fm2? The reasons for wanting to do this are because I'm developing a plug-in like architecture for environmental modelling where the user can develop new fortran modules (suitably f2py'ed) that can just be dropped into the module search path but still have access to the global data (subject to fortran module interfaces, etc).
The link above also gives an hint how to resolve this issue. Try to use sys.setdlopenflags(...) before importing f2py generated extension modules and then reset the state using sys.setdlopenflags(0). See http://docs.python.org/lib/module-sys.html for more information how to find proper value for ... HTH, Pearu
On 2/12/08, Pearu Peterson
according to which makes your goal unachivable because of how Python loads shared libraries *by default*, see below.
Try to use sys.setdlopenflags(...) before importing f2py generated extension modules and then reset the state using sys.setdlopenflags(0).
I also had to do something similar for solving a different problem, feel free to reuse the code here. This way, you have chances to make it working in a many platforms. You can put this in a __init__.py, and next import all your extensions inside the last try/finally block. http://projects.scipy.org/mpi4py/browser/mpi4py/trunk/src/_rtld.py -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
Thanks for that. The docs suggest library dl is Unix only. Does that mean this solution will not work on Windows? Windows is on my implementation roadmap but I'm not quite there yet to test it. I guess I am now thinking maybe I can assemble (using f2py) an (aggregated) shared library on the fly from the individual module shared libraries when I open my modelling environment. I could check the aggregated library mod dates against all the .so files of the components and only rebuild the aggregated library if there was a newer component than the aggregated library. That appears to work (and is fast) except for one behaviour of f2py. If I give f2py a list of files that are ALL .so (i.e. no fortran files) then f2py quits without actually doing anything, even if all the component shared libraries all have perfectly fine pythons interfaces from previous f2py builds. I can give it a trivial fortran file (module .. end module) and it works fine. f2py -c --fcompiler=g95 --verbose -m stuff -L/Library/Frameworks/ Python.framework/Versions/Current/lib/python2.5/config -lpython2.5 aread8.so fglobal_data.so # doesn't work f2py -c --fcompiler=g95 --verbose -m stuff -L/Library/Frameworks/ Python.framework/Versions/Current/lib/python2.5/config -lpython2.5 aread8.so fglobal_data.so junk.f90 # does work Why is that problem? I can envisage a user that just wants to use the environment without writing any additional fortran modules (and thus may not even have an installed fortran compiler) and if they screw up mod dates on the files (by say a ftp from one machine to another ... for instance on our cluster the compiler is only installed on one machine and only binaries are moved around the cluster) then the environment might want to reassemble (with f2py) the aggregated library because it (erroneously) thinks there is a newer component shared library. This will fail because f2py quits when asked to process ONLY .so files. If I have a trivial fortran file to force f2py then this forces users to have a fortran compiler on their machine, even if they do not want to actually compile a new fortran module component, simply because f2py will not operate unless it is offered at least one fortran file. Does this make sense or am I just being thick about this? Is there a way of making f2py merge a number of existing shared libraries into a single library without having to compile any fortran. I guess I could just invoke the linker directly in the case where there are no fortran files to compile but is nice being able to use distutils to get away from platform dependencies.
according to which makes your goal unachivable because of how Python loads shared libraries *by default*, see below.
Try to use sys.setdlopenflags(...) before importing f2py generated extension modules and then reset the state using sys.setdlopenflags (0).
I also had to do something similar for solving a different problem, feel free to reuse the code here. This way, you have chances to make it working in a many platforms. You can put this in a __init__.py, and next import all your extensions inside the last try/finally block.
http://projects.scipy.org/mpi4py/browser/mpi4py/trunk/src/_rtld.py
-- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
==================================================================== Prof Garry Willgoose, Australian Professorial Fellow in Environmental Engineering, Director, Centre for Climate Impact Management (C2IM), School of Engineering, The University of Newcastle, Callaghan, 2308 Australia. Centre webpage: www.c2im.org.au Phone: (International) +61 2 4921 6050 (Tues-Fri AM); +61 2 6545 9574 (Fri PM-Mon) FAX: (International) +61 2 4921 6991 (Uni); +61 2 6545 9574 (personal and Telluric) Env. Engg. Secretary: (International) +61 2 4921 6042 email: garry.willgoose@newcastle.edu.au; g.willgoose@telluricresearch.com email-for-life: garry.willgoose@alum.mit.edu personal webpage: www.telluricresearch.com/garry ==================================================================== "Do not go where the path may lead, go instead where there is no path and leave a trail" Ralph Waldo Emerson ====================================================================
On Thu, February 14, 2008 8:24 am, Garry Willgoose wrote:
Thanks for that. The docs suggest library dl is Unix only. Does that mean this solution will not work on Windows? Windows is on my implementation roadmap but I'm not quite there yet to test it.
I have no idea whether it will work on Windows or not. I would try though as there seems to be other ways than dl to find the needed flags as Lisandro pointed out.
I guess I am now thinking maybe I can assemble (using f2py) an (aggregated) shared library on the fly from the individual module shared libraries when I open my modelling environment. I could check the aggregated library mod dates against all the .so files of the components and only rebuild the aggregated library if there was a newer component than the aggregated library. That appears to work (and is fast) except for one behaviour of f2py. If I give f2py a list of files that are ALL .so (i.e. no fortran files) then f2py quits without actually doing anything, even if all the component shared libraries all have perfectly fine pythons interfaces from previous f2py builds. I can give it a trivial fortran file (module .. end module) and it works fine.
Note that f2py job is not really perform linking tasks. It is a useful feature that simplifies creating extensions modules, but please, don't complain if it cannot be used as a general linker:)
Why is that problem? I can envisage a user that just wants to use the environment without writing any additional fortran modules (and thus may not even have an installed fortran compiler) and if they screw up mod dates on the files (by say a ftp from one machine to another ... for instance on our cluster the compiler is only installed on one machine and only binaries are moved around the cluster) then the environment might want to reassemble (with f2py) the aggregated library because it (erroneously) thinks there is a newer component shared library. This will fail because f2py quits when asked to process ONLY .so files. If I have a trivial fortran file to force f2py then this forces users to have a fortran compiler on their machine, even if they do not want to actually compile a new fortran module component, simply because f2py will not operate unless it is offered at least one fortran file.
This is not a typical task for f2py. f2py is not a general purpose linker. It's amazing that f2py could even be used for such a task, so I don't think that the above demonstrates any bug of f2py. However, if you are worried about whether users have fortran compilers installed then can you assume that they have a C compiler installed? If so, then instead of trivial Fortran file try using the following trivial .pyf file: python module dummy interface subroutine dummyfunc() fortranname callstatement ; end subroutine dummyfunc end interface end python module dummy that should force f2py to build a shared library dummy.so with no Fortran dependencies.
Does this make sense or am I just being thick about this? Is there a way of making f2py merge a number of existing shared libraries into a single library without having to compile any fortran. I guess I could just invoke the linker directly in the case where there are no fortran files to compile but is nice being able to use distutils to get away from platform dependencies.
Hopefully the hint above works for you. Pearu
Pearu, Ohh Pearu I'm not complaining about deficiencies in f2py ... its a great piece of work that makes what I'm doing possible at all. Just like most open source software (including my own ;-) there may be ways to tweak it to do things that are undocumented.
Why is that problem? I can envisage a user that just wants to use the environment without writing any additional fortran modules (and thus may not even have an installed fortran compiler) and if they screw up mod dates on the files (by say a ftp from one machine to another ... for instance on our cluster the compiler is only installed on one machine and only binaries are moved around the cluster) then the environment might want to reassemble (with f2py) the aggregated library because it (erroneously) thinks there is a newer component shared library. This will fail because f2py quits when asked to process ONLY .so files. If I have a trivial fortran file to force f2py then this forces users to have a fortran compiler on their machine, even if they do not want to actually compile a new fortran module component, simply because f2py will not operate unless it is offered at least one fortran file.
This is not a typical task for f2py. f2py is not a general purpose linker. It's amazing that f2py could even be used for such a task, so I don't think that the above demonstrates any bug of f2py.
Indeed not typical ... as I recognise ... which is why I wondered if there was an undocumented way to tweak to do what I want to do (I get requests like this on my own software all the time;-).
However, if you are worried about whether users have fortran compilers installed then can you assume that they have a C compiler installed? If so, then instead of trivial Fortran file try using the following trivial .pyf file:
python module dummy interface subroutine dummyfunc() fortranname callstatement ; end subroutine dummyfunc end interface end python module dummy
that should force f2py to build a shared library dummy.so with no Fortran dependencies.
Perfect ...
==================================================================== Prof Garry Willgoose, Australian Professorial Fellow in Environmental Engineering, Director, Centre for Climate Impact Management (C2IM), School of Engineering, The University of Newcastle, Callaghan, 2308 Australia. Centre webpage: www.c3im.org.au Phone: (International) +61 2 4921 6050 (Tues-Fri AM); +61 2 6545 9574 (Fri PM-Mon) FAX: (International) +61 2 4921 6991 (Uni); +61 2 6545 9574 (personal and Telluric) Env. Engg. Secretary: (International) +61 2 4921 6042 email: garry.willgoose@newcastle.edu.au; g.willgoose@telluricresearch.com email-for-life: garry.willgoose@alum.mit.edu personal webpage: www.telluricresearch.com/garry ==================================================================== "Do not go where the path may lead, go instead where there is no path and leave a trail" Ralph Waldo Emerson ====================================================================
participants (4)
-
Garry Willgoose -
Garry Willgoose
-
Lisandro Dalcin -
Pearu Peterson