Re: [Numpy-discussion] RPMs out of date, have problems
python setup.py bdist_rpm needs only write access to the user's home directory.
Everywhere? It didn't when I first tried (that was an early distutils release under RedHat 6.2).
Can you check which version of Python/distutils. If it is a version python that does not come with distutils, we can count it out.
Of course, adding a generic spec file is the easy 2 minutes solution, but I have choosen not to do that because it will invite bad policy (providing binary RPMs is asking for trouble).
And yet many people will not accept anything but binary RPMs, out of ignorance or fear. We won't make everyone happy...
Yes, and they will complain to JH if they are failing to install JH's binary RPMs. The best we can do is explain the motivation of NOT providing binary RPMs. If that is not sufficient for some, we can only advise that they switch to Windows or pester their distro providers for RPMs of the latest NumPy.
I admit that I never looked into fully automatic RPM generation by distutils as I usually need to add some manual steps to the build/install process. For example, distutils doesn't let me specify compiler options, but I want time-critical code to be compiled with the highest optimization level. Hopefully this will be addressed in distutils one day.
There are two ways to do this: (1) Extension( ..., extra_compile_args = [ "-O17" ], ...) This will append the optimization level 17 to the compiler flag (the last -O option counts). (2) export CFLAGS=-O3; python setup.py build This will append -O3 The result of (1) and (2) is gcc .... -017 -03 Gerard PS: distutils may look intimidating at first, but it is really simple compared to automake/autoconf. The disadvantage is that it is not very well tested. --------------------------------------------- This message was sent using Endymion MailMan. http://www.endymion.com/products/mailman/
Can you check which version of Python/distutils. If it is a version python that does not come with distutils, we can count it out.
I don't remember, and I don't have any of that stuff left.
Yes, and they will complain to JH if they are failing to install JH's binary RPMs. The best we can do is explain the motivation of NOT providing binary RPMs. If that is not sufficient for some, we can
Somebody else will provide them, and you will still get complaints. I also get them. Fortunately there is a standard answer to send out.
There are two ways to do this:
(1) Extension( ..., extra_compile_args = [ "-O17" ], ...) This will append the optimization level 17 to the compiler flag (the last -O option counts).
But that will add the argument for all platforms. I need to specify compilation flags separately for each platform/compiler. I might even want to compile some modules with the new Intel compiler for Linux.
(2) export CFLAGS=-O3; python setup.py build This will append -O3
For all extension modules - I need to specify different flags for each extension module, in one case even for a single source code file. My current solution is to build everything with distutils and then recompile the critical modules manually in the spec file. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
(1) Extension( ..., extra_compile_args = [ "-O17" ], ...) This will append the optimization level 17 to the compiler flag (the last -O option counts).
But that will add the argument for all platforms. I need to specify compilation flags separately for each platform/compiler. I might even want to compile some modules with the new Intel compiler for Linux.
(2) export CFLAGS=-O3; python setup.py build This will append -O3
For all extension modules - I need to specify different flags for each extension module, in one case even for a single source code file.
This seems to be one of the big complaints about distutils for scientist (and maybe others). I think it is also one of Paul Dubois' major reasons for not liking distutils for Fortran files. I've run into this occasionally too and wished for a dictionary argument, maybe file_special_instructions, that had file names as keys and a dictionary of options for that file as the value. I think this might go some distance to solving the problem. file_special_instructions = { 'really_fast.c': { compiler = 'icc', override_compiler_flags = ['O2', 'tpp7'] } 'sorta_fast.c': { extra_compile_args = ['-O2']} } So an extension module that used this and had 'slow.c', 'really_fast.c', and 'kinda_fast.c' would use the standard compile tools for 'slow.c', use the specified compiler with the specified flags for 'really_fast.c', and add the specified compiler flags to the standard flags for 'sorta_fast.c'. This approach would provide quite a bit of flexiblity. To handle platform specific versions of file_special_instructions, I think you'd have to use if/thens within the setup.py file to build a separate set of instructions for each platform. There is one other major issue I've run into with distutils that I haven't found a way around. Sometimes you need compiler/linker flags inter-mingled with source or library files in a certain way. This can occur on SunOS when you want to link statically to some libraries and dynamically to others. distutils just doesn't provide a way of doing this that I have found. SciPy now has a package of extensions/changes to distutils called scipy_distutils helpful for building fortran based extension modules and other things needed by SciPy. We could experiment with adding something like the file_special_instructions flag if others thought it useful. Later, it could be folded back into distutils if the rest of the community wanted it. Thoughts? eric
About doing different things in distutils for different platforms, that is easy. It is a Python script so you can set things depending on sys.platform. E.g. from Numerical, # You might need to add a case here for your system if sys.platform == 'win32': mathlibs = [] define_macros = [] undef_macros = ['HAVE_INVERSE_HYPERBOLIC'] elif sys.platform == 'beos5': mathlibs = [] Later the setup call uses these variables in its argument list. Adding Fortran support is quite a bit more difficult because the whole idea of distutils is to piggy-back off the Python configure, which doesn't configure Fortran compiler options or paths. I don't think distutils ought to try, really. You just compile the Fortran into a library and then use that in your setup.py. I think a possible way out is scons, but that is just a preliminary impression. It bears a strong resemblance to the system I was working on in 1998 and abandoned when I changed jobs. My theory was that the build should be rule-based, with finer and finer rules for special cases or platforms. The highest priority rule that governs a particular file, wins. E.g., Rule 1: To compile a .c file, do cc -O ... Rule 2: To compile foo.c, do cc -O3 ... Rule 3: To compile foo.c on platform win32, do cc -O1 ... Rule 4: compile bar.c but only on platform win32 There was more to it, because one of the tricky points is that nowadays files produce multiple outputs per execution and may need to be processed in more than one way. Note that if you add a new .c file or a new platform, you're covered unless it needs special treatment. Anyway, this is actually not a design area the numerical community ought to try to get into; there are people who have spent a lot of time on this already. Given that sentence, I can't explain why I was doing it, other than that I hate make so badly I was driven to it. The project was called MMD: Make Must Die.
On Sun, 6 Jan 2002, Paul F. Dubois wrote:
Adding Fortran support is quite a bit more difficult because the whole idea of distutils is to piggy-back off the Python configure, which doesn't configure Fortran compiler options or paths. I don't think distutils ought to try, really. You just compile the Fortran into a library and then use that in your setup.py.
Saying that "just compile the Fortran into .." sounds like a common step for using Fortran, then I don't see why distutils shouldn't try to support this. As Eric mentioned in the previous message, scipy_distutils already does this step for an intermediate user. We have tested this to work for different compilers like Gnu,Intel F90,VAST,NAG F95,Absoft,etc and it works like a charm. Regards, Pearu
Please when writing these utilities keep us BSD users in mind. I had to severely hack SciPy to get it to compile on FreeBSD, part of which was the Fortran utility. Rob. Pearu Peterson wrote:
On Sun, 6 Jan 2002, Paul F. Dubois wrote:
Adding Fortran support is quite a bit more difficult because the whole idea of distutils is to piggy-back off the Python configure, which doesn't configure Fortran compiler options or paths. I don't think distutils ought to try, really. You just compile the Fortran into a library and then use that in your setup.py.
Saying that "just compile the Fortran into .." sounds like a common step for using Fortran, then I don't see why distutils shouldn't try to support this. As Eric mentioned in the previous message, scipy_distutils already does this step for an intermediate user. We have tested this to work for different compilers like Gnu,Intel F90,VAST,NAG F95,Absoft,etc and it works like a charm.
Regards, Pearu
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- The Numeric Python EM Project www.pythonemproject.com
participants (6)
-
eric -
gvermeul＠labs.polycnrs-gre.fr -
Konrad Hinsen -
Paul F. Dubois -
Pearu Peterson -
Rob