On Mon, Mar 30, 2009 at 2:59 AM, Antoine Pitrou wrote:

Jeffrey Yasskin writes:

...

The other popular configure+make replacement is scons.

I can only give uninformed information (!) here, but in one company I worked with, the main project decided to switch from scons to cmake due to some huge performance problems in scons. This was in 2005-2006, though, and I don't know whether things have changed.

They haven't - scons is still slow. Python is not that big, though (from a build POV) ? I would think the bootstrap problem to be much more significant. I don't find the argument "many desktop have already python" very convincing - what if you can't install it, for example ? AFAIK, scons does not run on jython or ironpython.

If you want to investigate Python-based build systems, there is waf (*), which apparently started out as a fork of scons (precisely due to the aforementioned performance problems). Again, I have never tried it.

Waf is definitely faster than scons - something like one order of magnitude. I am yet very familiar with waf, but I like what I saw - the architecture is much nicer than scons (waf core amount of code is almost ten times smaller than scons core), but I would not call it a mature project yet. About cmake: I haven't looked at it recently, but I have a bit of hard time believing python requires more from a build system than KDE. The lack of autoheader is not accurate, if only because kde projects have it: http://www.cmake.org/Wiki/CMake_HowToDoPlatformChecks Whether using it compared to the current system is really a win for python, I have no idea. David

On Mon, Mar 30, 2009 at 3:18 AM, Antoine Pitrou wrote:

What are the compilation requirements for cmake itself? Does it only need a standard C compiler and library, or are there other dependencies?

CMake is written in C++. IIRC, that's the only dependency. cheers, David

David Cournapeau wrote:

On Mon, Mar 30, 2009 at 2:59 AM, Antoine Pitrou wrote: ...

Waf is definitely faster than scons - something like one order of magnitude. I am yet very familiar with waf, but I like what I saw - the architecture is much nicer than scons (waf core amount of code is almost ten times smaller than scons core), but I would not call it a mature project yet.

I haven't tried waf, but IIUC it _solves_ the bootstrap issue.

On Mon, Mar 30, 2009 at 12:09 AM, Neil Hodgson wrote: ...

while so I can't remember the details. The current Python project files are hierarchical, building several DLLs and an EXE and I think this was outside the scope of the tools I looked at.

Not sure I understand. Having a project which builds (shared) libraries and executables which use them (and which maybe have to be executed later on during the build) is no problem for CMake, also with the VisualStudio projects.

From what I remember when I wrote the CMake files for python it was quite straight forward.

Alex

On Mon, Mar 30, 2009 at 8:04 PM, David Cournapeau wrote:

On Tue, Mar 31, 2009 at 2:37 AM, Alexander Neundorf wrote: ...

...

Not sure I understand. Having a project which builds (shared) libraries and executables which use them (and which maybe have to be executed later on during the build) is no problem for CMake, also with the VisualStudio projects.

...

From what I remember when I wrote the CMake files for python it was quite straight forward.

I think Christian meant that since on windows, those are built with visual studio project files, but everywhere else, it is built with distutils, you can't use a common system without first converting everything to cmake for all the other platforms.

Can you please explain ? What is "those" ?

Also, when converting a project from one build system to another, doing the 80 % takes 20 % in my experience.

Getting it working took me like 2 days, if that's 20% it's not too bad ;-)

The most time consuming part is all small the details on not so common platforms.

Yes. Alex

David Cournapeau wrote:

On Tue, Mar 31, 2009 at 3:16 AM, Alexander Neundorf wrote:

...

Can you please explain ? What is "those" ?

Everything in Lib. On windows, I believe this is done through project files, but on linux at least, and I guess on most other OS, those are handled by distutils. I guess the lack of autoconf on windows is one reason for this difference ?

All modules under Modules/ and PC/ are build inside the VS project. Some dependencies like gzip, bzip2, openssl etc. are build inside the project, too. Other dependencies are using shell scripts and nmake. On Unix the builtin modules are compiled with a custom build system (Modules/Setup.* and friends). The shared libraries are build by a distutils script (setup.py) in the root folder of the Python distribution. I already explained the reasons for the Windows build in a previous mail about 15 minutes ago. Christian

Hallo Alexander! Alexander Neundorf wrote:

This of course depends on the definition of "as good as" ;-) Well, I have met Windows-only developers which use CMake because it is able to generate project files for different versions of Visual Studio, and praise it for that.

So far I haven't heard any complains about or feature requests for the project files. ;) Most Windows related issue where about the SxS assembly issues with the new MSVCRT library. The multitude of project files formats are an issue for most open source libraries. Each minor version of Python has only one supported version of Visual Studio due to possible issues with mixed MSVCRTs. 99% of all Windows users are using the official MSI package provided by Martin. We also have project files for VS 6.0, 7.1 and 8.0 in the PC/ subdirectory. The VS 8.0 are created from the VS 9.0 project files. The remaining projects are maintained manually.

...

The VS project files support debug, non debug and profile guided optimization builds for X86 and AMD64 including cross compilation of AMD64 binaries.

CMake supports different build configurations.

Does it support all eight build configurations? (Debug, Release, PGO instrument, PGO update for X86 and X64)

No idea. Maybe it wouldn't be necessary ? With CMake you can just generate different buildtrees with different options, so you can get different behaviour in these trees.

I don't know, too.

That's right. Is there actually a real reason for this ? When I posted the cmake files for python in 2007 I think they also worked for Windows,, but I didn't test this that much.

As far as I remember DLL take considerable more time to load than shared libraries on Unix OSes. The VS project driven build also makes it possible to cross compile builds for Itanium and AMD64 platforms on a X86 Windows. Distutils doesn't support cross compilation so far. You have to ask Martin von Löwis and Mark Hammond for detailed information.

...

In my opinion any change to an automated system is a waste of precious developer times and makes our Windows support worse.

Seriously, I don't think so. In KDE, our (small group of) Windows developers are the ones which appreciate CMake most. At Kitware, support for Windows and MSVC is a primary feature of CMake, not an afterthought.

Please understand my previous mail in the context of Python core development. For KDE it makes perfectly sense to use cmake. For Python on Windows I just don't see any relevant benefits. I'm not against a CMake approach if somebody else is able to pull it off. So +0 from me. Christian

Ondrej Certik wrote:

Hi,

On Sun, Mar 29, 2009 at 10:21 AM, Jeffrey Yasskin wrote:

...

I've heard some good things about cmake — LLVM, googletest, and Boost are all looking at switching to it — so I wanted to see if we could simplify our autoconf+makefile system by using it. The biggest wins I see from going to cmake are: 1. It can autogenerate the Visual Studio project files instead of needing them to be maintained separately 2. It lets you write functions and modules without understanding autoconf's mix of shell and M4. 3. Its generated Makefiles track header dependencies accurately so we might be able to add private headers efficiently.

I am switching to cmake with all my python projects, as it is rock solid, supports building in parallel (if I have some C++ and Cython extensions), and the configure part works well.

The only disadvantage that I can see is that one has to learn a new syntax, which is not Python. But on the other hand, at least it forces one to really just use cmake to write build scripts in a standard way, while scons and other Python solutions imho encourage to write full Python programs, which imho is a disadvantage for the build system, as then every build system is nonstandard.

[obirrelevance] Isn't it strange how nobody every complained about the significance of whitespace in makefiles: only the fact that leading tabs were required rather than just-any-old whitespace. I guess some people just home in on things to complain about. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/

Greg Ewing wrote:

Steve Holden wrote:

...

Isn't it strange how nobody every complained about the significance of whitespace in makefiles: only the fact that leading tabs were required rather than just-any-old whitespace.

Make doesn't care how *much* whitespace there is, though, only whether it's there or not. If it accepted anything that looks like whitespace, there would be no cause for complaint.

Make and the *roff formats had the nasty feature that they treated homographs differently. That is, you could print two sources that placed all the same ink on the paper at the same places, but they would perform differently. For make it was tabs. For the *roff files, the periods ending sentences and the periods for abbreviations (such as honorifics) were distinguished by following end-of-sentence periods with two spaces. This left any line ending in a period ambiguous, and tools to strip whitespace off the end of lines as information-destroying. --Scott David Daniels Scott.Daniels@Acm.Org

writes:

I don't know thing one about cmake, but if it's good for the goose (building Python proper) would it be good for the gander (building extensions)?

African or European?

On Tue, Apr 7, 2009 at 9:14 PM, wrote:

   Ondrej> ... while scons and other Python solutions imho encourage to    Ondrej> write full Python programs, which imho is a disadvantage for the    Ondrej> build system, as then every build system is nonstandard.

Hmmm...  Like distutils setup scripts?

fortunately, waf and scons are much better than distutils, at least for the build part :) I think it is hard to overestimate the importance of a python solution for python softwares (python itself is different). Having a full fledged language for complex builds is nice, I think most familiar with complex makefiles would agree with this.

I don't know thing one about cmake, but if it's good for the goose (building Python proper) would it be good for the gander (building extensions)?

For complex softwares, specially ones relying on lot of C and platform idiosyncrasies, distutils is just too cumbersome and limited. Both Ondrej and me use python for scientific usage, and I think it is no hazard that we both look for something else. In those cases, scons - and cmake it seems - are very nice; build tools are incredibly hard to get right once you want to manage dependencies automatically. For simple python projects (pure python, a few .c source files without much dependencies), I think it is just overkill. cheers, David

-- Skip Montanaro - skip@pobox.com - http://www.smontanaro.net/        "XML sucks, dictionaries rock" - Dave Beazley _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/cournape%40gmail.com

On Wed, Apr 8, 2009 at 12:43 AM, Greg Ewing wrote:

David Cournapeau wrote:

...

Having a full fledged language for complex builds is nice, I think most familiar with complex makefiles would agree with this.

Yes, people will still need general computation in their build process from time to time whether the build tool they're using supports it or not.

From what I saw when I was building Python I didn't actually see too complicated things. In KDE4 we are not only building and installing

I'm maintaining the CMake-based buildsystem for KDE4 since 3 years now in my sparetime, millions lines of code, multiple code generators, all major operating systems. My experience is that people don't need general computation in their build process. CMake supports now more general purpose programming features than it did 2 years ago, e.g. it has now functions with local variables, it can do simple math, regexps and other things. If we get to the point where this is not enough, it usually means a real program which does real work is required. In this case it's actually a good thing to have this as a separate tool, and not mixed into the buildsystem. Having a not very powerful, but therefor domain specific language for the buildsystem is really a feature :-) (even if it sounds wrong in the first moment). programs, but we are also installing and shipping a development platform. This includes CMake files which contain functionality which helps in developing KDE software, i.e. variables and a bunch of KDE-specific macros. They are documented here: http://api.kde.org/cmake/modules.html#module_FindKDE4Internal (this is generated automatically from the cmake file we ship). I guess something similar could be useful for Python, maybe this is what distutils actually do ? I.e. they help with developing python-standard-conformant software ? This could be solved easily if python would install a cmake file which provides the necessary utility functions/macros. Alex

On Wed, Apr 8, 2009 at 7:54 AM, Alexander Neundorf wrote:

On Wed, Apr 8, 2009 at 12:43 AM, Greg Ewing wrote:

...

David Cournapeau wrote:

...

Having a full fledged language for complex builds is nice, I think most familiar with complex makefiles would agree with this.

Yes, people will still need general computation in their build process from time to time whether the build tool they're using supports it or not.

I'm maintaining the CMake-based buildsystem for KDE4 since 3 years now in my sparetime, millions lines of code, multiple code generators, all major operating systems. My experience is that people don't need general computation in their build process. CMake supports now more general purpose programming features than it did 2 years ago, e.g. it has now functions with local variables, it can do simple math, regexps and other things. If we get to the point where this is not enough, it usually means a real program which does real work is required. In this case it's actually a good thing to have this as a separate tool, and not mixed into the buildsystem. Having a not very powerful, but therefor domain specific language for the buildsystem is really a feature :-) (even if it sounds wrong in the first moment).

Yes, there are some advantages to that. The point of python is to have the same language for the build specification and the extensions, in my mind. For extensions, you really need a full language - for example, if you want to add support for tools which generate unknown files in advance, and handle this correctly from a build POV, a macro-like language is not sufficient.

...

From what I saw when I was building Python I didn't actually see too complicated things. In KDE4 we are not only building and installing programs, but we are also installing and shipping a development platform. This includes CMake files which contain functionality which helps in developing KDE software, i.e. variables and a bunch of KDE-specific macros. They are documented here: http://api.kde.org/cmake/modules.html#module_FindKDE4Internal (this is generated automatically from the cmake file we ship). I guess something similar could be useful for Python, maybe this is what distutils actually do ?

distutils does roughly everything that autotools does, and more: - configuration: not often used in extensions, we (numpy) are the exception I would guess - build - installation - tarball generation - bdist_ installers (msi, .exe on windows, .pkg/.mpkg on mac os x, rpm/deb on Linux) - registration to pypi - more things which just ellude me at the moment cheers, David

>> - registration to pypi Alex> No idea what this is . http://pypi.python.org/ It is, in some ways, a CPAN-like system for Python. Skip

cmake does not produce relative paths in its generated make and project files. There is an option CMAKE_USE_RELATIVE_PATHS which appears to do this but the documentation says: """This option does not work for more complicated projects, and relative paths are used when possible. In general, it is not possible to move CMake generated makefiles to a different location regardless of the value of this variable.""" This means that generated Visual Studio project files will not work for other people unless a particular absolute build location is specified for everyone which will not suit most. Each person that wants to build Python will have to run cmake before starting Visual Studio thus increasing the prerequisites. Neil

On Fri, 10 Apr 2009 11:49:04 +1000, Neil Hodgson wrote:

This means that generated Visual Studio project files will not work for other people unless a particular absolute build location is specified for everyone which will not suit most. Each person that wants to build Python will have to run cmake before starting Visual Studio thus increasing the prerequisites.

Given that we're now stuck with using whatever Visual Studio version the Python maintainers decided to use, I don't see this as a problem. As in: there is already a far larger and invasive dependency. CMake is readily available on all platforms, and it can be installed in a couple of seconds. -- Giovanni Bajo Develer S.r.l. http://www.develer.com

On Tue, Apr 7, 2009 at 10:08 PM, Alexander Neundorf wrote:

What is involved in building python extensions ? Can you please explain ?

Not much: at the core, a python extension is nothing more than a dynamically loaded library + a couple of options. One choice is whether to take options from distutils or to set them up independently. In my own scons tool to build python extensions, both are possible. The hard (or rather time consuming) work is to do everything else that distutils does related to the packaging. That's where scons/waf are more interesting than cmake IMO, because you can "easily" give up this task back to distutils, whereas it is inherently more difficult with cmake. cheers, David

On Wed, Apr 8, 2009 at 6:42 AM, Alexander Neundorf wrote:

What options ?

Compilation options. If you build an extension with distutils, the extension is built with the same flags as the ones used by python, the options are taken from distutils.sysconfig (except for MS compilers, which has its own options, which is one of the big pain in distutils).

Can you please explain ?

If you want to stay compatible with distutils, you have to do quite a lot of things. Cmake (and scons, and waf) only handle the build, but they can't handle all the packaging done by distutils (tarballs generation, binaries generation, in place build, develop mode of setuptools, eggs, .pyc and .pyo generation, etc...), so you have two choices: add support for this in the build tool (lot of work) or just use distutils once everything is built with your tool of choice.

It is easy to run external tools with cmake at cmake time and at build time, and it is also possible to run them at install time.

Sure, what can of build tool could not do that :) But given the design of distutils, if you want to keep all its packaging features, you can't just launch a few commands, you have to integrate them somewhat. Everytime you need something from distutils, you would need to launch python for cmake, whether with scons/waf, you can just use it as you would use any python library. That's just inherent to the fact that waf/scons are in the same language as distutils; if we were doing ocaml builds, having a build tool in ocaml would have been easier, etc... David