Evaluated cmake as an autoconf replacement
I've heard some good things about cmake — LLVM, googletest, and Boost are all looking at switching to it — so I wanted to see if we could simplify our autoconf+makefile system by using it. The biggest wins I see from going to cmake are: 1. It can autogenerate the Visual Studio project files instead of needing them to be maintained separately 2. It lets you write functions and modules without understanding autoconf's mix of shell and M4. 3. Its generated Makefiles track header dependencies accurately so we might be able to add private headers efficiently. However, after trying it out during PyCon and looking over the previous attempt at <http://mail.python.org/pipermail/python-dev/2007-July/073912.html>, I can't recommend switching to it. A. It has no equivalent of autoheader, so we'd have to maintain pyconfig.h.in by hand. B. It does not allow the CMakeLists.txt file control the --help output. This appears to be an intentional decision (http://www.cmake.org/pipermail/cmake-promote/2006-May/000095.html). To replace it, they have an interactive mode (which asks you about each possible option) and a -LH flag (which runs the whole configure process and then tells you about the flags you could have set if you knew about them). C. It does not allow the CMakeLists.txt file to see the command line, so we can't stay backward compatible with the configure script, and we'd have to replace flags like --with-pydebug with -Dpydebug:bool=true. We could write a separate configure script to mitigate this and the previous problem, but we'd have to keep that in sync with CMakeLists.txt manually. D. It doesn't have an expression language, so to replace ac_md_system=`echo $ac_sys_system | tr -d '[/ ]' | tr '[[A-Z]]' '[[a-z]]'` you have to write: string(REGEX REPLACE "[/ ]" "" ac_md_system ${ac_sys_system}) string(TOLOWER ${ac_md_system} ac_md_system) So, at the very least, it doesn't look like a big enough win to justify switching, and may not be a win at all. The other popular configure+make replacement is scons. The biggest objection to scons before even looking at it is that it requires a working Python interpreter in order to build Python, causing a bootstrap problem. However, Brett Cannon and I talked, and we think this is surmountable. First, nearly every desktop system comes with a Python interpreter, which would avoid the bootstrap for ordinary development. Second, we need a cross compilation story anyway for embedded systems, and the same mechanism could be used to get Python onto a new desktop platform. Third, Jython and IronPython are pretty mature and either can run scons now or should be able to run it after some relatively easy tweaks. They could be used to build CPython on a new platform. I don't intend to look into scons myself any time soon, but I'd be interested in anyone's experiences who does try it. Jeffrey
Nice report! Thanks! -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian W. Kernighan
Jeffrey Yasskin <jyasskin <at> gmail.com> writes:
The other popular configure+make replacement is scons.
I can only give uninformed information (!) here, but in one company I worked with, the main project decided to switch from scons to cmake due to some huge performance problems in scons. This was in 2005-2006, though, and I don't know whether things have changed. If you want to investigate Python-based build systems, there is waf (*), which apparently started out as a fork of scons (precisely due to the aforementioned performance problems). Again, I have never tried it. (*) http://code.google.com/p/waf/ Regards Antoine.
On Mon, Mar 30, 2009 at 2:59 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Jeffrey Yasskin <jyasskin <at> gmail.com> writes:
The other popular configure+make replacement is scons.
I can only give uninformed information (!) here, but in one company I worked with, the main project decided to switch from scons to cmake due to some huge performance problems in scons. This was in 2005-2006, though, and I don't know whether things have changed.
They haven't - scons is still slow. Python is not that big, though (from a build POV) ? I would think the bootstrap problem to be much more significant. I don't find the argument "many desktop have already python" very convincing - what if you can't install it, for example ? AFAIK, scons does not run on jython or ironpython.
If you want to investigate Python-based build systems, there is waf (*), which apparently started out as a fork of scons (precisely due to the aforementioned performance problems). Again, I have never tried it.
Waf is definitely faster than scons - something like one order of magnitude. I am yet very familiar with waf, but I like what I saw - the architecture is much nicer than scons (waf core amount of code is almost ten times smaller than scons core), but I would not call it a mature project yet. About cmake: I haven't looked at it recently, but I have a bit of hard time believing python requires more from a build system than KDE. The lack of autoheader is not accurate, if only because kde projects have it: http://www.cmake.org/Wiki/CMake_HowToDoPlatformChecks Whether using it compared to the current system is really a win for python, I have no idea. David
David Cournapeau <cournape <at> gmail.com> writes:
I would think the bootstrap problem to be much more significant. I don't find the argument "many desktop have already python" very convincing - what if you can't install it, for example ?
I agree. I had to build Python once on a corporate AIX box without any modern facilities. If it had needed anything else than a standard C compiler, I couldn't have done it.
About cmake: I haven't looked at it recently, but I have a bit of hard time believing python requires more from a build system than KDE.
What are the compilation requirements for cmake itself? Does it only need a standard C compiler and library, or are there other dependencies?
On Mon, Mar 30, 2009 at 3:18 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
What are the compilation requirements for cmake itself? Does it only need a standard C compiler and library, or are there other dependencies?
CMake is written in C++. IIRC, that's the only dependency. cheers, David
On Sun, Mar 29, 2009 at 1:14 PM, David Cournapeau <cournape@gmail.com> wrote:
About cmake: I haven't looked at it recently, but I have a bit of hard time believing python requires more from a build system than KDE. The lack of autoheader is not accurate, if only because kde projects have it:
That page says, "... So we write a source file named config.h.in...". The purpose of autoheader is to write that file automatically. I might have missed something, but you'll have to provide a more precise link. The problems I found were enough to convince me that it wasn't worth continuing to a full translation of configure.in and Makefile.pre.in. If you or someone else wants to do that translation, I'd be happy to look at the result in case it turns out not to be as inconvenient as I currently expect. Jeffrey
David Cournapeau wrote:
On Mon, Mar 30, 2009 at 2:59 AM, Antoine Pitrou <solipsis@pitrou.net> wrote: ...
Waf is definitely faster than scons - something like one order of magnitude. I am yet very familiar with waf, but I like what I saw - the architecture is much nicer than scons (waf core amount of code is almost ten times smaller than scons core), but I would not call it a mature project yet.
I haven't tried waf, but IIUC it _solves_ the bootstrap issue.
Jeffrey Yasskin:
1. It can autogenerate the Visual Studio project files instead of needing them to be maintained separately
I have looked at a couple of build tools (scons was probably one) that generate Visual Studio project files in the past and they produced fairly poor project files, which would compile the code but wouldn't be as capable as project files created by hand. Its been a while so I can't remember the details. The current Python project files are hierarchical, building several DLLs and an EXE and I think this was outside the scope of the tools I looked at. Neil
On Mon, Mar 30, 2009 at 12:09 AM, Neil Hodgson <nyamatongwe@gmail.com> wrote: ...
while so I can't remember the details. The current Python project files are hierarchical, building several DLLs and an EXE and I think this was outside the scope of the tools I looked at.
Not sure I understand. Having a project which builds (shared) libraries and executables which use them (and which maybe have to be executed later on during the build) is no problem for CMake, also with the VisualStudio projects.
From what I remember when I wrote the CMake files for python it was quite straight forward.
Alex
On Tue, Mar 31, 2009 at 2:37 AM, Alexander Neundorf <alex.neundorf@kitware.com> wrote:
On Mon, Mar 30, 2009 at 12:09 AM, Neil Hodgson <nyamatongwe@gmail.com> wrote: ...
while so I can't remember the details. The current Python project files are hierarchical, building several DLLs and an EXE and I think this was outside the scope of the tools I looked at.
Not sure I understand. Having a project which builds (shared) libraries and executables which use them (and which maybe have to be executed later on during the build) is no problem for CMake, also with the VisualStudio projects.
From what I remember when I wrote the CMake files for python it was quite straight forward.
I think Christian meant that since on windows, those are built with visual studio project files, but everywhere else, it is built with distutils, you can't use a common system without first converting everything to cmake for all the other platforms. Also, when converting a project from one build system to another, doing the 80 % takes 20 % in my experience. The most time consuming part is all small the details on not so common platforms. David
On Mon, Mar 30, 2009 at 8:04 PM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Mar 31, 2009 at 2:37 AM, Alexander Neundorf <alex.neundorf@kitware.com> wrote: ...
Not sure I understand. Having a project which builds (shared) libraries and executables which use them (and which maybe have to be executed later on during the build) is no problem for CMake, also with the VisualStudio projects.
From what I remember when I wrote the CMake files for python it was quite straight forward.
I think Christian meant that since on windows, those are built with visual studio project files, but everywhere else, it is built with distutils, you can't use a common system without first converting everything to cmake for all the other platforms.
Can you please explain ? What is "those" ?
Also, when converting a project from one build system to another, doing the 80 % takes 20 % in my experience.
Getting it working took me like 2 days, if that's 20% it's not too bad ;-)
The most time consuming part is all small the details on not so common platforms.
Yes. Alex
On Tue, Mar 31, 2009 at 3:16 AM, Alexander Neundorf <alex.neundorf@kitware.com> wrote:
Can you please explain ? What is "those" ?
Everything in Lib. On windows, I believe this is done through project files, but on linux at least, and I guess on most other OS, those are handled by distutils. I guess the lack of autoconf on windows is one reason for this difference ?
Also, when converting a project from one build system to another, doing the 80 % takes 20 % in my experience.
Getting it working took me like 2 days, if that's 20% it's not too bad ;-)
So it means ten days of work to convert to a new system that maybe most python maintainers do not know. What does it bring ? I think supporting cross compilation would be more worthwhile, for example, in the build department. cheers, David
David Cournapeau wrote:
On Tue, Mar 31, 2009 at 3:16 AM, Alexander Neundorf <alex.neundorf@kitware.com> wrote:
Can you please explain ? What is "those" ?
Everything in Lib. On windows, I believe this is done through project files, but on linux at least, and I guess on most other OS, those are handled by distutils. I guess the lack of autoconf on windows is one reason for this difference ?
All modules under Modules/ and PC/ are build inside the VS project. Some dependencies like gzip, bzip2, openssl etc. are build inside the project, too. Other dependencies are using shell scripts and nmake. On Unix the builtin modules are compiled with a custom build system (Modules/Setup.* and friends). The shared libraries are build by a distutils script (setup.py) in the root folder of the Python distribution. I already explained the reasons for the Windows build in a previous mail about 15 minutes ago. Christian
On Mon, Mar 30, 2009 at 8:49 PM, Christian Heimes <lists@cheimes.de> wrote:
David Cournapeau wrote:
On Tue, Mar 31, 2009 at 3:16 AM, Alexander Neundorf <alex.neundorf@kitware.com> wrote:
Can you please explain ? What is "those" ?
Everything in Lib. On windows, I believe this is done through project files, but on linux at least, and I guess on most other OS, those are handled by distutils. I guess the lack of autoconf on windows is one reason for this difference ?
All modules under Modules/ and PC/ are build inside the VS project. Some dependencies like gzip, bzip2, openssl etc. are build inside the project, too. Other dependencies are using shell scripts and nmake.
AFAIK we also have cmake-based builds for gzip, bzip2 and other dependencies KDE needs somewhere.
On Unix the builtin modules are compiled with a custom build system (Modules/Setup.* and friends). The shared libraries are build by a distutils script (setup.py) in the root folder of the Python distribution.
With CMake I was also building the Modules directly with CMake, i.e. not using distutils, so these were already taken care of. In Lib/ there are only python files, and they should be compiled to pyc files, right ? I didn't do this back then. We are right now getting better support for this in KDE, which would probably help here: http://lists.kde.org/?l=kde-buildsystem&m=123795136609498&w=2 Alex
Jeffrey Yasskin wrote:
1. It can autogenerate the Visual Studio project files instead of needing them to be maintained separately
I'm familiar with the Unix and the Windows build system. More than a year ago I went to a great deal of work to migrate the Windows builds from VS 7.1 to VS 9.0. I'm in doubt that any automatic tool can create configuration files that are as good as our hand made files. The VS project files support debug, non debug and profile guided optimization builds for X86 and AMD64 including cross compilation of AMD64 binaries. The project files are using multiple inheritance to avoid duplication of options. The differences between Windows and Unix builds are fairly large, too. On Windows lots of modules are built in and the remaining Python extensions are build with VS directly. On Unix most modules are build as shared libraries using distutils and setup.py. In my opinion any change to an automated system is a waste of precious developer times and makes our Windows support worse. Christian
Hi, On Mon, Mar 30, 2009 at 1:22 AM, Christian Heimes <lists@cheimes.de> wrote:
Jeffrey Yasskin wrote:
1. It can autogenerate the Visual Studio project files instead of needing them to be maintained separately
I'm familiar with the Unix and the Windows build system. More than a year ago I went to a great deal of work to migrate the Windows builds from VS 7.1 to VS 9.0. I'm in doubt that any automatic tool can create configuration files that are as good as our hand made files.
This of course depends on the definition of "as good as" ;-) Well, I have met Windows-only developers which use CMake because it is able to generate project files for different versions of Visual Studio, and praise it for that.
The VS project files support debug, non debug and profile guided optimization builds for X86 and AMD64 including cross compilation of AMD64 binaries.
CMake supports different build configurations.
The project files are using multiple inheritance to avoid duplication of options.
No idea. Maybe it wouldn't be necessary ? With CMake you can just generate different buildtrees with different options, so you can get different behaviour in these trees.
The differences between Windows and Unix builds are fairly large, too. On Windows lots of modules are built in and the remaining Python extensions are build with VS directly. On Unix most modules are build as shared libraries using distutils and setup.py.
That's right. Is there actually a real reason for this ? When I posted the cmake files for python in 2007 I think they also worked for Windows,, but I didn't test this that much.
In my opinion any change to an automated system is a waste of precious developer times and makes our Windows support worse.
Seriously, I don't think so. In KDE, our (small group of) Windows developers are the ones which appreciate CMake most. At Kitware, support for Windows and MSVC is a primary feature of CMake, not an afterthought. Alex
Hallo Alexander! Alexander Neundorf wrote:
This of course depends on the definition of "as good as" ;-) Well, I have met Windows-only developers which use CMake because it is able to generate project files for different versions of Visual Studio, and praise it for that.
So far I haven't heard any complains about or feature requests for the project files. ;) Most Windows related issue where about the SxS assembly issues with the new MSVCRT library. The multitude of project files formats are an issue for most open source libraries. Each minor version of Python has only one supported version of Visual Studio due to possible issues with mixed MSVCRTs. 99% of all Windows users are using the official MSI package provided by Martin. We also have project files for VS 6.0, 7.1 and 8.0 in the PC/ subdirectory. The VS 8.0 are created from the VS 9.0 project files. The remaining projects are maintained manually.
The VS project files support debug, non debug and profile guided optimization builds for X86 and AMD64 including cross compilation of AMD64 binaries.
CMake supports different build configurations.
Does it support all eight build configurations? (Debug, Release, PGO instrument, PGO update for X86 and X64)
No idea. Maybe it wouldn't be necessary ? With CMake you can just generate different buildtrees with different options, so you can get different behaviour in these trees.
I don't know, too.
That's right. Is there actually a real reason for this ? When I posted the cmake files for python in 2007 I think they also worked for Windows,, but I didn't test this that much.
As far as I remember DLL take considerable more time to load than shared libraries on Unix OSes. The VS project driven build also makes it possible to cross compile builds for Itanium and AMD64 platforms on a X86 Windows. Distutils doesn't support cross compilation so far. You have to ask Martin von Löwis and Mark Hammond for detailed information.
In my opinion any change to an automated system is a waste of precious developer times and makes our Windows support worse.
Seriously, I don't think so. In KDE, our (small group of) Windows developers are the ones which appreciate CMake most. At Kitware, support for Windows and MSVC is a primary feature of CMake, not an afterthought.
Please understand my previous mail in the context of Python core development. For KDE it makes perfectly sense to use cmake. For Python on Windows I just don't see any relevant benefits. I'm not against a CMake approach if somebody else is able to pull it off. So +0 from me. Christian
On Mon, 30 Mar 2009 20:34:21 +0200, Christian Heimes wrote:
Hallo Alexander!
Alexander Neundorf wrote:
This of course depends on the definition of "as good as" ;-) Well, I have met Windows-only developers which use CMake because it is able to generate project files for different versions of Visual Studio, and praise it for that.
So far I haven't heard any complains about or feature requests for the project files. ;)
In fact, I have had one. I asked to put all those big CJK codecs outside of python2x.dll because they were too big and create far larger self-contained distributions (aka: py2exe/pyinstaller) as would normally be required. I was replied that it would be unconvienent to do so because of the fact that the build system is made by hand and it's hard to generate project files for each third party module. Were those project files generated automatically, changing between external modules within or outside python2x dll would be a one-line switch in CMakeLists.txt (or similar). -- Giovanni Bajo Develer S.r.l. http://www.develer.com
-On [20090329 19:21], Jeffrey Yasskin (jyasskin@gmail.com) wrote:
However, Brett Cannon and I talked, and we think this is surmountable. First, nearly every desktop system comes with a Python interpreter, which would avoid the bootstrap for ordinary development.
This is quite a major assumption. Most FreeBSD, NetBSD, and OpenBSD users tend to install a minimal binary OS (kernel plus system tools) and then proceed to install third party applications via either ports or pkgsrc. This means that Python gets built from scratch. So depending on Python to build Python seems a bad decision if you care for my opinion. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B I must be cruel, only to be kind...
Hi, On Sun, Mar 29, 2009 at 10:21 AM, Jeffrey Yasskin <jyasskin@gmail.com> wrote:
I've heard some good things about cmake — LLVM, googletest, and Boost are all looking at switching to it — so I wanted to see if we could simplify our autoconf+makefile system by using it. The biggest wins I see from going to cmake are: 1. It can autogenerate the Visual Studio project files instead of needing them to be maintained separately 2. It lets you write functions and modules without understanding autoconf's mix of shell and M4. 3. Its generated Makefiles track header dependencies accurately so we might be able to add private headers efficiently.
I am switching to cmake with all my python projects, as it is rock solid, supports building in parallel (if I have some C++ and Cython extensions), and the configure part works well. The only disadvantage that I can see is that one has to learn a new syntax, which is not Python. But on the other hand, at least it forces one to really just use cmake to write build scripts in a standard way, while scons and other Python solutions imho encourage to write full Python programs, which imho is a disadvantage for the build system, as then every build system is nonstandard. Ondrej
Ondrej Certik wrote:
Hi,
On Sun, Mar 29, 2009 at 10:21 AM, Jeffrey Yasskin <jyasskin@gmail.com> wrote:
I've heard some good things about cmake — LLVM, googletest, and Boost are all looking at switching to it — so I wanted to see if we could simplify our autoconf+makefile system by using it. The biggest wins I see from going to cmake are: 1. It can autogenerate the Visual Studio project files instead of needing them to be maintained separately 2. It lets you write functions and modules without understanding autoconf's mix of shell and M4. 3. Its generated Makefiles track header dependencies accurately so we might be able to add private headers efficiently.
I am switching to cmake with all my python projects, as it is rock solid, supports building in parallel (if I have some C++ and Cython extensions), and the configure part works well.
The only disadvantage that I can see is that one has to learn a new syntax, which is not Python. But on the other hand, at least it forces one to really just use cmake to write build scripts in a standard way, while scons and other Python solutions imho encourage to write full Python programs, which imho is a disadvantage for the build system, as then every build system is nonstandard.
[obirrelevance] Isn't it strange how nobody every complained about the significance of whitespace in makefiles: only the fact that leading tabs were required rather than just-any-old whitespace. I guess some people just home in on things to complain about. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/
Steve Holden wrote:
Isn't it strange how nobody every complained about the significance of whitespace in makefiles: only the fact that leading tabs were required rather than just-any-old whitespace.
Make doesn't care how *much* whitespace there is, though, only whether it's there or not. If it accepted anything that looks like whitespace, there would be no cause for complaint. -- Greg
Greg Ewing wrote:
Steve Holden wrote:
Isn't it strange how nobody every complained about the significance of whitespace in makefiles: only the fact that leading tabs were required rather than just-any-old whitespace.
Make doesn't care how *much* whitespace there is, though, only whether it's there or not. If it accepted anything that looks like whitespace, there would be no cause for complaint.
Make and the *roff formats had the nasty feature that they treated homographs differently. That is, you could print two sources that placed all the same ink on the paper at the same places, but they would perform differently. For make it was tabs. For the *roff files, the periods ending sentences and the periods for abbreviations (such as honorifics) were distinguished by following end-of-sentence periods with two spaces. This left any line ending in a period ambiguous, and tools to strip whitespace off the end of lines as information-destroying. --Scott David Daniels Scott.Daniels@Acm.Org
Ondrej> ... while scons and other Python solutions imho encourage to Ondrej> write full Python programs, which imho is a disadvantage for the Ondrej> build system, as then every build system is nonstandard. Hmmm... Like distutils setup scripts? I don't know thing one about cmake, but if it's good for the goose (building Python proper) would it be good for the gander (building extensions)? -- Skip Montanaro - skip@pobox.com - http://www.smontanaro.net/ "XML sucks, dictionaries rock" - Dave Beazley
>> I don't know thing one about cmake, but if it's good for the goose >> (building Python proper) would it be good for the gander (building >> extensions)? Antoine> African or European? I was thinking Canadian... Skip
On Tue, Apr 7, 2009 at 9:14 PM, <skip@pobox.com> wrote:
Ondrej> ... while scons and other Python solutions imho encourage to Ondrej> write full Python programs, which imho is a disadvantage for the Ondrej> build system, as then every build system is nonstandard.
Hmmm... Like distutils setup scripts?
fortunately, waf and scons are much better than distutils, at least for the build part :) I think it is hard to overestimate the importance of a python solution for python softwares (python itself is different). Having a full fledged language for complex builds is nice, I think most familiar with complex makefiles would agree with this.
I don't know thing one about cmake, but if it's good for the goose (building Python proper) would it be good for the gander (building extensions)?
For complex softwares, specially ones relying on lot of C and platform idiosyncrasies, distutils is just too cumbersome and limited. Both Ondrej and me use python for scientific usage, and I think it is no hazard that we both look for something else. In those cases, scons - and cmake it seems - are very nice; build tools are incredibly hard to get right once you want to manage dependencies automatically. For simple python projects (pure python, a few .c source files without much dependencies), I think it is just overkill. cheers, David
-- Skip Montanaro - skip@pobox.com - http://www.smontanaro.net/ "XML sucks, dictionaries rock" - Dave Beazley _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/cournape%40gmail.com
David Cournapeau wrote:
Having a full fledged language for complex builds is nice, I think most familiar with complex makefiles would agree with this.
Yes, people will still need general computation in their build process from time to time whether the build tool they're using supports it or not. And if it doesn't, they'll resort to some ungodly mash such as Makefile+ shell+m4. Python has got to be a better choice than that. -- Greg
On Wed, Apr 8, 2009 at 12:43 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
David Cournapeau wrote:
Having a full fledged language for complex builds is nice, I think most familiar with complex makefiles would agree with this.
Yes, people will still need general computation in their build process from time to time whether the build tool they're using supports it or not.
From what I saw when I was building Python I didn't actually see too complicated things. In KDE4 we are not only building and installing
I'm maintaining the CMake-based buildsystem for KDE4 since 3 years now in my sparetime, millions lines of code, multiple code generators, all major operating systems. My experience is that people don't need general computation in their build process. CMake supports now more general purpose programming features than it did 2 years ago, e.g. it has now functions with local variables, it can do simple math, regexps and other things. If we get to the point where this is not enough, it usually means a real program which does real work is required. In this case it's actually a good thing to have this as a separate tool, and not mixed into the buildsystem. Having a not very powerful, but therefor domain specific language for the buildsystem is really a feature :-) (even if it sounds wrong in the first moment). programs, but we are also installing and shipping a development platform. This includes CMake files which contain functionality which helps in developing KDE software, i.e. variables and a bunch of KDE-specific macros. They are documented here: http://api.kde.org/cmake/modules.html#module_FindKDE4Internal (this is generated automatically from the cmake file we ship). I guess something similar could be useful for Python, maybe this is what distutils actually do ? I.e. they help with developing python-standard-conformant software ? This could be solved easily if python would install a cmake file which provides the necessary utility functions/macros. Alex
Alexander Neundorf wrote:
My experience is that people don't need general computation in their build process. ... CMake supports now more general purpose programming features than it did 2 years ago, e.g. it has now functions with local variables, it can do simple math, regexps and other things.
In other words, it's growing towards being able to do general computation. Why is it doing that, if people don't need general computation in their build process?
If we get to the point where this is not enough, it usually means a real program which does real work is required. In this case it's actually a good thing to have this as a separate tool, and not mixed into the buildsystem.
There's some merit in that idea, but the build tool and the program need to work together smoothly somehow. If the build tool is implemented in Python, there's more chance of that happening (e.g. the Python code can import parts of the build system and call them directly, rather than having to generate a file in some other language). -- Greg
On Wed, Apr 8, 2009 at 7:54 AM, Alexander Neundorf <alex.neundorf@kitware.com> wrote:
On Wed, Apr 8, 2009 at 12:43 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
David Cournapeau wrote:
Having a full fledged language for complex builds is nice, I think most familiar with complex makefiles would agree with this.
Yes, people will still need general computation in their build process from time to time whether the build tool they're using supports it or not.
I'm maintaining the CMake-based buildsystem for KDE4 since 3 years now in my sparetime, millions lines of code, multiple code generators, all major operating systems. My experience is that people don't need general computation in their build process. CMake supports now more general purpose programming features than it did 2 years ago, e.g. it has now functions with local variables, it can do simple math, regexps and other things. If we get to the point where this is not enough, it usually means a real program which does real work is required. In this case it's actually a good thing to have this as a separate tool, and not mixed into the buildsystem. Having a not very powerful, but therefor domain specific language for the buildsystem is really a feature :-) (even if it sounds wrong in the first moment).
Yes, there are some advantages to that. The point of python is to have the same language for the build specification and the extensions, in my mind. For extensions, you really need a full language - for example, if you want to add support for tools which generate unknown files in advance, and handle this correctly from a build POV, a macro-like language is not sufficient.
From what I saw when I was building Python I didn't actually see too complicated things. In KDE4 we are not only building and installing programs, but we are also installing and shipping a development platform. This includes CMake files which contain functionality which helps in developing KDE software, i.e. variables and a bunch of KDE-specific macros. They are documented here: http://api.kde.org/cmake/modules.html#module_FindKDE4Internal (this is generated automatically from the cmake file we ship). I guess something similar could be useful for Python, maybe this is what distutils actually do ?
distutils does roughly everything that autotools does, and more: - configuration: not often used in extensions, we (numpy) are the exception I would guess - build - installation - tarball generation - bdist_ installers (msi, .exe on windows, .pkg/.mpkg on mac os x, rpm/deb on Linux) - registration to pypi - more things which just ellude me at the moment cheers, David
On Wed, Apr 8, 2009 at 4:18 AM, David Cournapeau <cournape@gmail.com> wrote: ...
I guess something similar could be useful for Python, maybe this is what distutils actually do ?
distutils does roughly everything that autotools does, and more: - configuration: not often used in extensions, we (numpy) are the exception I would guess - build - installation - tarball generation - bdist_ installers (msi, .exe on windows, .pkg/.mpkg on mac os x, rpm/deb on Linux)
I think cmake can do all of the above (cpack supports creating packages).
- registration to pypi
No idea what this is . Alex
>> - registration to pypi Alex> No idea what this is . http://pypi.python.org/ It is, in some ways, a CPAN-like system for Python. Skip
On Thu, Apr 9, 2009 at 4:45 AM, Alexander Neundorf <alex.neundorf@kitware.com> wrote:
I think cmake can do all of the above (cpack supports creating packages).
I am sure it is - it is just a lot of work, specially if you want to stay compatible with distutils-built extensions :) cheers, David
cmake does not produce relative paths in its generated make and project files. There is an option CMAKE_USE_RELATIVE_PATHS which appears to do this but the documentation says: """This option does not work for more complicated projects, and relative paths are used when possible. In general, it is not possible to move CMake generated makefiles to a different location regardless of the value of this variable.""" This means that generated Visual Studio project files will not work for other people unless a particular absolute build location is specified for everyone which will not suit most. Each person that wants to build Python will have to run cmake before starting Visual Studio thus increasing the prerequisites. Neil
Neil Hodgson wrote:
cmake does not produce relative paths in its generated make and project files. There is an option CMAKE_USE_RELATIVE_PATHS which appears to do this but the documentation says:
"""This option does not work for more complicated projects, and relative paths are used when possible. In general, it is not possible to move CMake generated makefiles to a different location regardless of the value of this variable."""
This means that generated Visual Studio project files will not work for other people unless a particular absolute build location is specified for everyone which will not suit most. Each person that wants to build Python will have to run cmake before starting Visual Studio thus increasing the prerequisites.
This is true. CMake does not generate stand alone transferable projects. CMake must be installed on the machine where the compilation is done. CMake will automatically re-run if any of the inputs are changed, and have visual studio re-load the project, and CMake can be used for simple cross platform commands like file copy and and other operations so that the build files do not depend on shell commands or anything system specific. -Bill
On Fri, 10 Apr 2009 11:49:04 +1000, Neil Hodgson wrote:
This means that generated Visual Studio project files will not work for other people unless a particular absolute build location is specified for everyone which will not suit most. Each person that wants to build Python will have to run cmake before starting Visual Studio thus increasing the prerequisites.
Given that we're now stuck with using whatever Visual Studio version the Python maintainers decided to use, I don't see this as a problem. As in: there is already a far larger and invasive dependency. CMake is readily available on all platforms, and it can be installed in a couple of seconds. -- Giovanni Bajo Develer S.r.l. http://www.develer.com
On Tue, Apr 7, 2009 at 2:14 PM, <skip@pobox.com> wrote:
Ondrej> ... while scons and other Python solutions imho encourage to Ondrej> write full Python programs, which imho is a disadvantage for the Ondrej> build system, as then every build system is nonstandard.
I fully agree here.
Hmmm... Like distutils setup scripts?
I don't know thing one about cmake, but if it's good for the goose (building Python proper) would it be good for the gander (building extensions)?
What is involved in building python extensions ? Can you please explain ? Alex
On Tue, Apr 7, 2009 at 10:08 PM, Alexander Neundorf <alex.neundorf@kitware.com> wrote:
What is involved in building python extensions ? Can you please explain ?
Not much: at the core, a python extension is nothing more than a dynamically loaded library + a couple of options. One choice is whether to take options from distutils or to set them up independently. In my own scons tool to build python extensions, both are possible. The hard (or rather time consuming) work is to do everything else that distutils does related to the packaging. That's where scons/waf are more interesting than cmake IMO, because you can "easily" give up this task back to distutils, whereas it is inherently more difficult with cmake. cheers, David
On Tue, Apr 7, 2009 at 3:23 PM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Apr 7, 2009 at 10:08 PM, Alexander Neundorf <alex.neundorf@kitware.com> wrote:
What is involved in building python extensions ? Can you please explain ?
Not much: at the core, a python extension is nothing more than a dynamically loaded library + a couple of options.
CMake has support (slightly but intentionally undocumented) for this, from FindPythonLibs.cmake: # PYTHON_ADD_MODULE(<name> src1 src2 ... srcN) is used to build modules for python. # PYTHON_WRITE_MODULES_HEADER(<filename>) writes a header file you can include # in your sources to initialize the static python modules Using python_add_module(name file1.c file2.c) you can build python modules, and decide at cmake time whether it should be a dynamically loaded module (default) or whether it should be built as a static library (useful for platforms without shared libs). Installation then happens simply via install(TARGETS ...)
One choice is whether to take options from distutils or to set them up
What options ?
independently. In my own scons tool to build python extensions, both are possible.
The hard (or rather time consuming) work is to do everything else that distutils does related to the packaging. That's where scons/waf are more interesting than cmake IMO, because you can "easily" give up this task back to distutils, whereas it is inherently more difficult with cmake.
Can you please explain ? It is easy to run external tools with cmake at cmake time and at build time, and it is also possible to run them at install time. Alex
On Wed, Apr 8, 2009 at 6:42 AM, Alexander Neundorf <alex.neundorf@kitware.com> wrote:
What options ?
Compilation options. If you build an extension with distutils, the extension is built with the same flags as the ones used by python, the options are taken from distutils.sysconfig (except for MS compilers, which has its own options, which is one of the big pain in distutils).
Can you please explain ?
If you want to stay compatible with distutils, you have to do quite a lot of things. Cmake (and scons, and waf) only handle the build, but they can't handle all the packaging done by distutils (tarballs generation, binaries generation, in place build, develop mode of setuptools, eggs, .pyc and .pyo generation, etc...), so you have two choices: add support for this in the build tool (lot of work) or just use distutils once everything is built with your tool of choice.
It is easy to run external tools with cmake at cmake time and at build time, and it is also possible to run them at install time.
Sure, what can of build tool could not do that :) But given the design of distutils, if you want to keep all its packaging features, you can't just launch a few commands, you have to integrate them somewhat. Everytime you need something from distutils, you would need to launch python for cmake, whether with scons/waf, you can just use it as you would use any python library. That's just inherent to the fact that waf/scons are in the same language as distutils; if we were doing ocaml builds, having a build tool in ocaml would have been easier, etc... David
participants (16)
-
Aahz -
Alexander Neundorf -
Antoine Pitrou -
Bill Hoffman -
Christian Heimes -
David Cournapeau -
Giovanni Bajo -
Greg Ewing -
Jeffrey Yasskin -
Jeroen Ruigrok van der Werven -
Neal Becker -
Neil Hodgson -
Ondrej Certik -
Scott David Daniels -
skip＠pobox.com -
Steve Holden