That written in Python, so Python would require Python to build which would require Python to build which would require... Besides, SCons sucks. If I were using a build system written in Python, I'd use fbuild or Waf. On Fri, Mar 13, 2015 at 2:27 PM, Wes Turner <wes.turner@gmail.com> wrote:

Scons? http://www.scons.org/doc/production/HTML/scons-user.html On Mar 13, 2015 1:44 PM, "Ryan Gonzalez" <rymg19@gmail.com> wrote:

...

So...I've been looking stuff up. Don't have too much free time right now, but it was mostly Googleing.

I have deemed three ideas to be best for Python:

- Write a custom makefile generator. I have some ideas, but they likely won't come to light any time soon.

- Use Boost.Build. It seems ugly and complicated, but it actually has a full configuring system that can detect libraries and try to compile source files. It's written in C89, not C++, and its dependency is the Boehm GC, which probably supports everywhere that Python runs on.

Only issue? It's undergoing a rewrite in Python. However, at the rate it's going, Python will likely be at version 7 by the time they finish.

- Stick with autotools. Trying to avoid this.

Thoughts? Again?

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:

That written in Python, so Python would require Python to build which would require Python to build which would require...

Well, you can't buid a C compiler without a C compiler either (or SOME compiler). And given that you should be able to do cross-compilation for platforms that don't currently have a python build, I'm not sure that's a killer. It it so wrong that python eats its own dog food in this manner? -CHB

--

Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, Mar 13, 2015 at 2:47 PM, Chris Barker <chris.barker@noaa.gov> wrote:

On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:

...

That written in Python, so Python would require Python to build which would require Python to build which would require...

Well, you can't buid a C compiler without a C compiler either (or SOME compiler). And given that you should be able to do cross-compilation for platforms that don't currently have a python build, I'm not sure that's a killer.

It it so wrong that python eats its own dog food in this manner?

It's not as much "Python eat its own dog food" as "cross-compiling natively would be even harder than it is right now." C compilers are available on virtually every UNIX that's ever been created since the 70's, including AIX, Plan 9, Solaris, etc. Python? Not as much.

-CHB

...

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

On Fri, Mar 13, 2015 at 3:01 PM, Wes Turner <wes.turner@gmail.com> wrote:

On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" <rymg19@gmail.com> wrote:

...

That written in Python, so Python would require Python to build which

would require Python to build which would require...

...

Besides, SCons sucks. If I were using a build system written in Python,

I'd use fbuild or Waf.

Apparently, Waf started as a fork of SCons. [1][2]

[1] http://stackoverflow.com/questions/12569463/choosing-between-scons-and-waf-i... [2] http://www.scons.org/wiki/SconsVsOtherBuildTools#Waf

But it's now better.

I don't know anything about fbuild; others might.

https://github.com/felix-lang/fbuild

A feature/alternative matrix could be helpful (in addition to a preliminary implementation) for making a cogent argument (in PEP form [3][4]). This is possible in RST with table or list-table syntax.

[3] https://www.python.org/dev/peps/pep-0001/#pep-formats-and-templates [4] https://www.python.org/dev/peps/pep-0009/

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

On Fri, Mar 13, 2015, at 20:07, Robert Collins wrote:

Wrong, no. harder yes. C compilers have mature cross compiler build chains including stage aware test suites and the like. When a tool is it's own build dependency bootstrapping it on a new platform gains a hard requirement on cross compiling rather than it being a thing for special cases like platforms that cannot compile the tool.

That's what bootstrapping is. If you're not doing this, you're not bootstrapping.

On Sat, Mar 14, 2015 at 6:47 AM, Chris Barker <chris.barker@noaa.gov> wrote:

On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:

...

That written in Python, so Python would require Python to build which would require Python to build which would require...

Well, you can't buid a C compiler without a C compiler either (or SOME compiler). And given that you should be able to do cross-compilation for platforms that don't currently have a python build, I'm not sure that's a killer.

It it so wrong that python eats its own dog food in this manner?

-CHB

I've been just skimming this thread rather than reading in detail, so please forgive me if this question is completely unrelated. Will this affect the commands required to build Python from source, or is all this just for actual development? Consider two completely different workflows here: Firstly, a core Python dev. S/he wants to be able to efficiently build, edit, etc, and wants to be able to edit any part of the code. and secondly, someone who has little knowledge of Python's internals, and just wants to build the latest version (either from hg or from a source tarball). In the first case, it's fine to require that some version of Python be already installed. If editing file XYZ requires that file ABC be rebuilt from it, and the rebuild process requires Python 3.3 or newer, that's not going to hurt people. The modified ABC gets checked in along with the change to XYZ, and all's well. But in the second case, it'd be much safer to minimize the requirements. If you can untar the source and just "./configure; make; sudo make install" your way to having the latest Python, that's a LOT easier than I've just been doing trying to get my Dad onto a newer Mozilla Thunderbird. (I don't understand what it's doing all along there, but it pulls a whole bunch of separate repositories when I say update, and there's a separate bootstrap step, and ... I dunno.) ChrisA

...

...

That written in Python, so Python would require Python to build which would require Python to build which would require...

A bit more here: It seems there are two cases: 1) Development on and for a common, mature platform: In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools. 2) Development for / porting to a new and/or obscure platform: In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general): a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it... b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process: i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools ii) run the configure script to generate Makefiles, etc. configured for the platform at hand. iii) run make to do the actual build. Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability. I have no idea if any of the known tools can do this, but if so, it would be pretty usable. And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work. But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions. Will this affect the commands required to build Python from source, or

is all this just for actual development?

well, it may or may not be both: If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery. someone

who has little knowledge of Python's internals, and just wants to build the latest version (either from hg or from a source tarball).

That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools... But in the second case, it'd be much safer to minimize the

requirements. If you can untar the source and just "./configure; make;

sudo make install" your way to having the latest Python, that's a LOT easier than I've just been doing trying to get my Dad onto a newer Mozilla Thunderbird. (I don't understand what it's doing all along there, but it pulls a whole bunch of separate repositories when I say update, and there's a separate bootstrap step, and ... I dunno.)

requiring _some_, not very recent, python is not exactly """a whole bunch of separate repositories when I say update, and there's a separate bootstrap step""" If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap". In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Tue, Mar 17, 2015 at 3:27 AM, Chris Barker <chris.barker@noaa.gov> wrote:

...

someone who has little knowledge of Python's internals, and just wants to build the latest version (either from hg or from a source tarball).

That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools...

The tricky part here is defining the oldest supported version. "To build Python X.Y, you must already have Python ??? or later installed" - how do you pick the minimum? The older that minimum, the more warped the build code has to be (especially if you want to support 2.x as well as 3.x - imagine trying to make it possible to use the system Python on RHEL, as well as any 3.[1-4]), but go too far the other way and you end up with a multi-step build process ("first build Python 3.4, then use that to build Python 3.7") which is just going to be a pain.

...

But in the second case, it'd be much safer to minimize the requirements. If you can untar the source and just "./configure; make; sudo make install" your way to having the latest Python, that's a LOT easier than I've just been doing trying to get my Dad onto a newer Mozilla Thunderbird. (I don't understand what it's doing all along there, but it pulls a whole bunch of separate repositories when I say update, and there's a separate bootstrap step, and ... I dunno.)

requiring _some_, not very recent, python is not exactly

"""a whole bunch of separate repositories when I say update, and there's a separate bootstrap step"""

If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap".

No, that's not what I'm talking about. I'm talking about trying to build Thunderbird and having to do a bunch of steps where I run this program, then run that, then let it chug its way through stuff, and then eventually it'll start building. Without detailed knowledge of the build internals I have no way of knowing what steps I can skip, so every time I want to test the build again, I have to go through them all. It took three parts of forever to figure out what was going on. I much prefer projects that ship with the standard "./configure; make" pattern. I don't care if the configure script is built by some arcane incantation, as long as it's checked into source control (or at very least is in the downloadable source tarball for any particular version). If I want to build your program, I don't want to have to hunt down your notes file, read it through in detail, and jump through a dozen hoops; I just want to point my compiler at your source, and then go make myself a hot chocolate while it does its work. ChrisA

On Mon, Mar 16, 2015 at 9:50 AM, Chris Angelico <rosuav@gmail.com> wrote:

The tricky part here is defining the oldest supported version. "To build Python X.Y, you must already have Python ??? or later installed" - how do you pick the minimum? The older that minimum, the more warped the build code has to be

sure -- but it seems what's on the table now is: must be buildable by an old version of C -- not even C99 which is nominally 16 years old... So yeah, the build system itself would need to support a "old" python -- though probably python2.7 would be OK at this point -- sorry, not py3 yet, but what can you do?

If there is an already build python (like OS-X, any general purpose linux

...

distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap".

No, that's not what I'm talking about. I'm talking about trying to build Thunderbird and having to do a bunch of steps where I run this program, then run that, then let it chug its way through stuff, and then eventually it'll start building.

It sounds like that build setup suck, yes -- but not sure why that's relevant. Clearly we don't want something that ugly. We want something that is both easy and robust to configure for new platfroms, etc, and easy on the end user to "jsut build" the source. All I"m proposing that is requireing a not-very-recent python as a first step is only a big hurdle for folks poritn got new platforms -- your use case would be, in INSTALL.txt: """ 1) You need at least python 2.7 to build this software. try: python --version if it starts up and reports a version greater than 2.7, then you are all set. If not, then It can probably be installed with your system package manager: apt_get install python or similar. On Windows, you can download an installer for Python2.7 from python.org 2) Once a python is working, you can build the this latest version from source with: ./buildme """ Of course, there may be options, etc, but that's the basic deal. And heck, if people REALLY want the ./configure && make && make install dance, then we could ship a little configure script that does nothing but call the python-based one... Of course, all this is predicated on there BEING some miraculous as-powerful-and-flexible-and-robust-as-autotools-but-much-easier-to-use python-based build system..... One more note: so adding a python dependency is a pain. But the current system requires separate stems for Windows and *nix-y systems, and who knows what for hypothetical systems that don't support autotools. So maybe a python-based build system that worked on all the common platforms, with an additional makefile that you have to hand-mangle to bootstrap a new platform isn't any more to maintain. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Something nobody's mentioned yet in this thread: The autotools stuff doesn't just handle different platforms, it also handles local differences (not every FooLinux x.y system has Tcl/Tk; if yours doesn't, you can still build Python, you just won't get tkinter) and user options (disabling features, using a specific OpenSSL instead of the older system one that would get autodetected, etc.). Autoconf makes this relatively easy (not that anything in autoconf is easy, but if you can write the OpenSSL detection, it's not significantly harder to write OpenSSL detection with a flag to override the path). Is the same thing true for other build configuration systems? Sent from my iPhone On Mar 16, 2015, at 9:27 AM, Chris Barker <chris.barker@noaa.gov> wrote:

...

...

...

That written in Python, so Python would require Python to build which would require Python to build which would require...

A bit more here:

It seems there are two cases:

1) Development on and for a common, mature platform:

In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools.

2) Development for / porting to a new and/or obscure platform:

In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general):

a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it...

b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process:

i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools

ii) run the configure script to generate Makefiles, etc. configured for the platform at hand.

iii) run make to do the actual build.

Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability.

I have no idea if any of the known tools can do this, but if so, it would be pretty usable.

And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work.

But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions.

...

Will this affect the commands required to build Python from source, or is all this just for actual development?

well, it may or may not be both:

If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery.

...

someone who has little knowledge of Python's internals, and just wants to build the latest version (either from hg or from a source tarball).

That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools...

...

But in the second case, it'd be much safer to minimize the requirements. If you can untar the source and just "./configure; make; sudo make install" your way to having the latest Python, that's a LOT easier than I've just been doing trying to get my Dad onto a newer Mozilla Thunderbird. (I don't understand what it's doing all along there, but it pulls a whole bunch of separate repositories when I say update, and there's a separate bootstrap step, and ... I dunno.)

requiring _some_, not very recent, python is not exactly

"""a whole bunch of separate repositories when I say update, and there's a separate bootstrap step"""

If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap".

In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building.

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Most due let you do that easily. For instance, fbuild lets you just trap the raised error and move on. On Mon, Mar 16, 2015 at 1:05 PM, Andrew Barnert < abarnert@yahoo.com.dmarc.invalid> wrote:

Something nobody's mentioned yet in this thread:

The autotools stuff doesn't just handle different platforms, it also handles local differences (not every FooLinux x.y system has Tcl/Tk; if yours doesn't, you can still build Python, you just won't get tkinter) and user options (disabling features, using a specific OpenSSL instead of the older system one that would get autodetected, etc.).

Autoconf makes this relatively easy (not that anything in autoconf is easy, but if you can write the OpenSSL detection, it's not significantly harder to write OpenSSL detection with a flag to override the path). Is the same thing true for other build configuration systems?

Sent from my iPhone

On Mar 16, 2015, at 9:27 AM, Chris Barker <chris.barker@noaa.gov> wrote:

...

...

That written in Python, so Python would require Python to build which

...

would require Python to build which would require...

A bit more here:

It seems there are two cases:

1) Development on and for a common, mature platform:

In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools.

2) Development for / porting to a new and/or obscure platform:

In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general):

a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it...

b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process:

i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools

ii) run the configure script to generate Makefiles, etc. configured for the platform at hand.

iii) run make to do the actual build.

Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability.

I have no idea if any of the known tools can do this, but if so, it would be pretty usable.

And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work.

But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions.

Will this affect the commands required to build Python from source, or

...

is all this just for actual development?

well, it may or may not be both:

If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery.

someone

...

who has little knowledge of Python's internals, and just wants to build the latest version (either from hg or from a source tarball).

That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools...

But in the second case, it'd be much safer to minimize the

...

requirements. If you can untar the source and just "./configure; make;

...

sudo make install" your way to having the latest Python, that's a LOT easier than I've just been doing trying to get my Dad onto a newer Mozilla Thunderbird. (I don't understand what it's doing all along there, but it pulls a whole bunch of separate repositories when I say update, and there's a separate bootstrap step, and ... I dunno.)

requiring _some_, not very recent, python is not exactly

"""a whole bunch of separate repositories when I say update, and there's a separate bootstrap step"""

If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap".

In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building.

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

On 16/03/2015 19:24, Ryan Gonzalez wrote:

On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah <stefan@bytereef.org <mailto:stefan@bytereef.org>> wrote:

Ryan Gonzalez <rymg19@...> writes: > I hate most tools that ship with configure scripts because 70% of the time some stupid test does some stupid thing wrong and gives the worst errors, 29% of the time the configure system doesn't pick up my options correctly, and 1% of the time it actually *works*.

That's just not true. Also, the small build issue that triggered this thread is already tracked here:

http://bugs.python.org/issue22625

You've evidently had better luck than I usually have.

Luck doesn't enter in to it. More likely a bad workan always blames his tools?

It seems more productive to me to fix that rather than rewrite the build system. Users don't have an unlimited tolerance for pointless churn.

Lots and lots of projects have moved from autotools to about everything else.

It's slow, annoying, and generates huge files (the 5k-line configure.ac <http://configure.ac> generates a 17k-line configure script). And largely pointless, since nowadays few like it anymore.

Perhaps everybody should move to MSVC, that way at least everything would be consistent :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

On Mon, Mar 16, 2015 at 11:31 AM, Chris Barker <chris.barker@noaa.gov> wrote:

On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <techtonik@gmail.com> wrote:

...

"Autotools has reached the Chandrasekhar mass limit of software and become a black hole of suck that consumes endless hours of bright peoples’ time without any good result. It is well past time for it to die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877

I couldn't sum up this better. It looks even worse from Windows machine.

Is it even used for building Python on Windows (for the "normal" python.org builds)?

No. Some random batch file is used. That's all I know. However, when building without MSVC/VS2008, you need MinGW, which means you'll end up using autotools.

But to miss-paraphrase Winston Churchill [1]:

"Autotools is the worst build system except for all the others."

Is there really something better out there?

Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...

-CHB

[1] http://www.goodreads.com/quotes/267224-democracy-is-the-worst-form-of-govern...

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily <nad@acm.org> wrote:

it would be a monumental undertaking to try to migrate the current build system to something substantially different. It's fine to let off steam about frustrations with build systems but talking about it here is not gonna cause it to change. And it won't change unless someone (or, more likely, some big company) is willing to invest an enormous effort in people time and machine resources to do so. Stefan's suggestion is much more practical.

This doesn't seem to me to be a winner takes all proposition -- there is no reason someone couldn't set up a new build system for Python, get it all working nicely on their platform(s) of choice, persuade others how wonderful it is, get contributions, etc. If it turns out to really be better/easier etc. than autoconf, then , and only then, would it make any sense to talk about replacing the existing official build system. And even then, the two could live in parallel for quite some time... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Ryan Gonzalez writes:

I have used fbuild a LOT

Actually, I suspect that in the relevant sense, you haven't used it a lot. Have you used it for a single program on 1. more than 3 CPUs, AND 2. on each CPU, more than one (and preferably more than two) OSes, AND 3. for each CPU-OS combination, more than two different configurations of the program being built, AND 4. for each CPU-OS-configuration combination, at least 3 versions of the program source (preferably spanning a major version bump)? The number of programs/versions you have built on a single (or small selection of) CPU-OS combinations is irrelevant to the discussion. The problem autotools helps to solve better than any other system I know of is bootstrapping on a new system. It's also quite good at handling foibles (== regressions) across version bumps of the source program. I agree it's suboptimal (at best :-) to have a PyPI module that depends on the autotools. I've found it to be a real PITA for Windows users who want to develop certain Emacs Lisp applications, for example, which is a similar kind of development environment to Python. But for bootstrapping Python itself, what else is there, really? Every time I've seen a project move to something like SCons, I've also seen their port rate to new platforms decrease dramatically. (Hard to interpret that trend, though -- if a project has the free cycles to change build systems, maybe they have ported to almost all of the plausible platforms already.) Autotools alternatives just have way fewer modules for dealing with new and/or old ;-) platforms. Bright! Shiny! New! system use crufty old system :-( + => + missing library :-( library Caveat: Autotools is the only build system (well, Imake -- now you know how old I am!) I have substantial experience with. But I've heard stories from colleagues about the limitations of Scons and cmake in this respect, and I find it hard to believe that there are systems out there with the accumulated knowledge of platforms that has been assembled in Autotools over the decades.

On Mar 16, 2015, at 12:19 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:

Most due let you do that easily. For instance, fbuild lets you just trap the raised error and move on.

How does trapping an error and moving on give you a user-configurable option to select which OpenSSL to use, whether he wants a fat i386/x86_64 build or just x86_64, whether he wants a framework bundle or a traditional layout...? (These are the first three flags that Homebrew passes to Python's configure aside from the install prefix, which I'd hope everything can handle.)

...

On Mon, Mar 16, 2015 at 1:05 PM, Andrew Barnert <abarnert@yahoo.com.dmarc.invalid> wrote: Something nobody's mentioned yet in this thread:

The autotools stuff doesn't just handle different platforms, it also handles local differences (not every FooLinux x.y system has Tcl/Tk; if yours doesn't, you can still build Python, you just won't get tkinter) and user options (disabling features, using a specific OpenSSL instead of the older system one that would get autodetected, etc.).

Autoconf makes this relatively easy (not that anything in autoconf is easy, but if you can write the OpenSSL detection, it's not significantly harder to write OpenSSL detection with a flag to override the path). Is the same thing true for other build configuration systems?

Sent from my iPhone

On Mar 16, 2015, at 9:27 AM, Chris Barker <chris.barker@noaa.gov> wrote:

...

...

...

...

That written in Python, so Python would require Python to build which would require Python to build which would require...

A bit more here:

It seems there are two cases:

1) Development on and for a common, mature platform:

In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools.

2) Development for / porting to a new and/or obscure platform:

In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general):

a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it...

b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process:

i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools

ii) run the configure script to generate Makefiles, etc. configured for the platform at hand.

iii) run make to do the actual build.

Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability.

I have no idea if any of the known tools can do this, but if so, it would be pretty usable.

And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work.

But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions.

...

Will this affect the commands required to build Python from source, or is all this just for actual development?

well, it may or may not be both:

If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery.

...

someone who has little knowledge of Python's internals, and just wants to build the latest version (either from hg or from a source tarball).

That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools...

...

But in the second case, it'd be much safer to minimize the requirements. If you can untar the source and just "./configure; make; sudo make install" your way to having the latest Python, that's a LOT easier than I've just been doing trying to get my Dad onto a newer Mozilla Thunderbird. (I don't understand what it's doing all along there, but it pulls a whole bunch of separate repositories when I say update, and there's a separate bootstrap step, and ... I dunno.)

requiring _some_, not very recent, python is not exactly

"""a whole bunch of separate repositories when I say update, and there's a separate bootstrap step"""

If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap".

In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building.

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

Ryan Gonzalez writes:

...

I have used fbuild a LOT

Actually, I suspect that in the relevant sense, you haven't used it a lot. Have you used it for a single program on

1. more than 3 CPUs, AND 2. on each CPU, more than one (and preferably more than two) OSes, AND 3. for each CPU-OS combination, more than two different configurations of the program being built, AND 4. for each CPU-OS-configuration combination, at least 3 versions of the program source (preferably spanning a major version bump)?

No...but I know that Felix uses it for iPhone, OSX, Linux, ... with separation of host and target and multiple configurations.

The number of programs/versions you have built on a single (or small selection of) CPU-OS combinations is irrelevant to the discussion. The problem autotools helps to solve better than any other system I know of is bootstrapping on a new system. It's also quite good at handling foibles (== regressions) across version bumps of the source program.

I agree it's suboptimal (at best :-) to have a PyPI module that depends on the autotools. I've found it to be a real PITA for Windows users who want to develop certain Emacs Lisp applications, for example, which is a similar kind of development environment to Python.

But for bootstrapping Python itself, what else is there, really? Every time I've seen a project move to something like SCons, I've also seen their port rate to new platforms decrease dramatically. (Hard to interpret that trend, though -- if a project has the free cycles to change build systems, maybe they have ported to almost all of the plausible platforms already.) Autotools alternatives just have way fewer modules for dealing with new and/or old ;-) platforms.

This is also SCons. Everybody loved it until they had to maintain it. :)

Bright! Shiny! New! system use crufty old system :-( + => + missing library :-( library

Caveat: Autotools is the only build system (well, Imake -- now you know how old I am!) I have substantial experience with. But I've heard stories from colleagues about the limitations of Scons and cmake in this respect, and I find it hard to believe that there are systems out there with the accumulated knowledge of platforms that has been assembled in Autotools over the decades.

SCons is very annoying. CMake is weird and only really useful for those huge, 20,000 file C++ projects. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

On Mon, Mar 16, 2015 at 8:18 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:

They have a build system genius who wrote Tup working for them

I like the conceptual brevity of the Tup page http://gittup.org/tup/ It is the core of that I am try to bring from SCons internals to the surface. It is very often that you need to process some DAG and touch nodes based on their changed dependencies. The problem is so common that it is even worth to spend time on visualization for it. So far no build system has this stuff, and I run of time to complete the SCons plumbing. -- anatoly t.

On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker <chris.barker@noaa.gov> wrote:

On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <techtonik@gmail.com> wrote:

...

"Autotools has reached the Chandrasekhar mass limit of software and become a black hole of suck that consumes endless hours of bright peoples’ time without any good result. It is well past time for it to die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877

I couldn't sum up this better. It looks even worse from Windows machine.

Is it even used for building Python on Windows (for the "normal" python.org builds)?

There was some success with MinGW toolchain, but I am not tracking if it was integrated into official source. The FUD was that compiled extensions are not compatible between MSVC builds and MinGW builds, but I could not confirm that. If I understand correctly, the incompatibility is only for C++ programs, because of strange SEH strategy and proprietary name mangling.

But to miss-paraphrase Winston Churchill [1]:

"Autotools is the worst build system except for all the others."

Is there really something better out there?

A lot of attempts are made, but reviewing them requires a lot of time and effort.

On Tue, Mar 17, 2015 at 2:45 AM, Chris Barker <chris.barker@noaa.gov> wrote:

This doesn't seem to me to be a winner takes all proposition -- there is no reason someone couldn't set up a new build system for Python, get it all working nicely on their platform(s) of choice, persuade others how wonderful it is, get contributions, etc.

If it turns out to really be better/easier etc. than autoconf, then , and only then, would it make any sense to talk about replacing the existing official build system. And even then, the two could live in parallel for quite some time...

Both Blender and Wesnoth have SCons and CMake running in parallel for a long time. Both camps are just maintaining their own chains and it works well.

On Tue, Mar 17, 2015 at 8:01 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:

On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

...

Ryan Gonzalez writes:

...

I have used fbuild a LOT

Actually, I suspect that in the relevant sense, you haven't used it a lot. Have you used it for a single program on

1. more than 3 CPUs, AND 2. on each CPU, more than one (and preferably more than two) OSes, AND 3. for each CPU-OS combination, more than two different configurations of the program being built, AND 4. for each CPU-OS-configuration combination, at least 3 versions of the program source (preferably spanning a major version bump)?

No...but I know that Felix uses it for iPhone, OSX, Linux, ... with separation of host and target and multiple configurations.

What or who is Felix?

This is also SCons. Everybody loved it until they had to maintain it. :)

I know a blind girl, who made a voice synthesizer, which works on Windows, Linux and Android, and she used SCons to build it. That gave me another reason to stick with this build system and try to enhance it.

SCons is very annoying. CMake is weird and only really useful for those huge, 20,000 file C++ projects.

For me the advantage of SCons is that I already know its codebase. There a lot of room for improvement if you can tag yourself as a Python hacker it addition to inherent potential user property. In any, I'd appreciate the feedback on most annoying SCons feature, because it will give a criteria for comparing it with others and for future development. -- anatoly t.

Then you need Node.js installed, which is kind of big. On Wed, Mar 18, 2015 at 2:21 AM, anatoly techtonik <techtonik@gmail.com> wrote:

On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:

...

...

But to miss-paraphrase Winston Churchill [1]:

"Autotools is the worst build system except for all the others."

Is there really something better out there?

Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...

Is there anything Javascriptey? Python needs to move to web. I heard something about Grunt, which was replaced by Gulp, but no idea if they really mature enough to include "system features detection" and "user configured builds" (or work with C/C++ compilers at all).

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

On Mar 18, 2015 8:14 AM, "anatoly techtonik" <techtonik@gmail.com> wrote:

On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker <chris.barker@noaa.gov>

...

On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <techtonik@gmail.com> wrote:

...

"Autotools has reached the Chandrasekhar mass limit of software and become a black hole of suck that consumes endless hours of bright peoples’ time without any good result. It is well past time for it to die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877

I couldn't sum up this better. It looks even worse from Windows

machine.

Is it even used for building Python on Windows (for the "normal"

...

builds)?

There was some success with MinGW toolchain, but I am not tracking if it was integrated into official source. The FUD was that compiled extensions are not compatible between MSVC builds and MinGW builds, but I could not confirm that. If I understand correctly, the incompatibility is only for C++

wrote: python.org programs,

because of strange SEH strategy and proprietary name mangling.

This isn't FUD, alas. mingw in almost all of its variants uses effectively a different "libc" than MSVC, so code compiled with two compilers can't even agree on the meaning of file numbers. It is mostly possible to set up a mingw-w64 build that produces MSVC-compatible extensions, but it requires using a configuration that's undocumented and untested upstream (I.e. you will run into mingw bugs). NumPy *may* switch to this at some point -- we've been working on it for like a year now -- but it's extremely non-trivial. -n

And unfortunately, a lot of that work may need to be redone for 3.5 and VC14... Sorry about that, but once the other compilers have caught up with the new CRT interface then it will (should) all be better in the long run. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Nathaniel Smith<mailto:njs@pobox.com> Sent: ‎3/‎18/‎2015 10:31 To: anatoly techtonik<mailto:techtonik@gmail.com> Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] Rewriting the build system (part 2) On Mar 18, 2015 8:14 AM, "anatoly techtonik" <techtonik@gmail.com<mailto:techtonik@gmail.com>> wrote:

On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker <chris.barker@noaa.gov<mailto:chris.barker@noaa.gov>> wrote:

...

On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <techtonik@gmail.com<mailto:techtonik@gmail.com>> wrote:

...

"Autotools has reached the Chandrasekhar mass limit of software and become a black hole of suck that consumes endless hours of bright peoples’ time without any good result. It is well past time for it to die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877

I couldn't sum up this better. It looks even worse from Windows machine.

Is it even used for building Python on Windows (for the "normal" python.org<http://python.org> builds)?

There was some success with MinGW toolchain, but I am not tracking if it was integrated into official source. The FUD was that compiled extensions are not compatible between MSVC builds and MinGW builds, but I could not confirm that. If I understand correctly, the incompatibility is only for C++ programs, because of strange SEH strategy and proprietary name mangling.

This isn't FUD, alas. mingw in almost all of its variants uses effectively a different "libc" than MSVC, so code compiled with two compilers can't even agree on the meaning of file numbers. It is mostly possible to set up a mingw-w64 build that produces MSVC-compatible extensions, but it requires using a configuration that's undocumented and untested upstream (I.e. you will run into mingw bugs). NumPy *may* switch to this at some point -- we've been working on it for like a year now -- but it's extremely non-trivial. -n

On Thu, Mar 19, 2015 at 1:48 PM, Andrew Barnert < abarnert@yahoo.com.dmarc.invalid> wrote:

On Mar 19, 2015, at 4:12 AM, anatoly techtonik <techtonik@gmail.com> wrote:

...

...

On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin <me@the-compiler.org>

wrote:

...

...

* anatoly techtonik <techtonik@gmail.com> [2015-03-18 10:21:46 +0300]:

...

On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rymg19@gmail.com> wrote:

...

...

But to miss-paraphrase Winston Churchill [1]:

"Autotools is the worst build system except for all the others."

Is there really something better out there?

Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...

Is there anything Javascriptey? Python needs to move to web. I heard something about Grunt, which was replaced by Gulp, but no idea if they really mature enough to include "system features detection" and "user configured builds" (or work with C/C++ compilers at all).

What does "Python needs to move to web" to do with it's build system being "Javascriptey"?

Because things like Twisted are de-facto standard in JavaScript world (Node.js), and Python will benefit more if people look for how things are implemented in non-C worlds.

Is this a sarcastic comment?

Or are you seriously suggesting that because people use things like the stdlib's asyncio, gevent, Tornado, etc. in Python, which have different paradigms than Twisted, the one Node copied, we need to force some other part of Python to be more like Node in some other way, because "move to the web" means "be more like the latest (but nowhere near most popular) framework for building web services"?

Also, correct me if I'm wrong, but isn't Node implemented in C, with a build system in C and Python? So if we really want to be more like Node for some reason, we should make the build system Pythony instead of Javascripty?

You're not wrong. Node uses Gyp, although, if Chromium switches to GN (written in C++), Node might switch, too.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/