[Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks

[Russell Keith-Magee]

On Tue, Mar 10, 2015 at 10:43 AM, Ryan <rymg19 at gmail.com> wrote:

>
>
> Russell Keith-Magee <russell at keith-magee.com> wrote:
> >Hi Ryan,
> >
> >First off I'm no fan of autoconf, but...
> >
>
> Who is other than the one who designed it? :)
>

Let me be clear - there's a difference between "not a fan" and "calling for
it's head". Autoconf is crufty, annoying, and sometimes aggravating - but
it *works*, and has had some pretty seriously battle testing - both in the
abstract "autoconf as a product", and in the specific "autoconf as used in
Python". This isn't something that can be said of most of the alternatives
in the abstract case, or *any* of the alternatives in the Python case.

>On Tue, Mar 10, 2015 at 1:40 AM, Ryan Gonzalez <rymg19 at gmail.com>
> >wrote:
> >
> >> I've been trying to work on Issue 23496. And want to tear my hair
> >out.
> >> I've noticed a few problems with Python's current build system:
> >>
> >> - There is no separation of the host C compiler and the target C
> >compiler.
> >> When you give configure a C compiler for cross-compiling, make will
> >happily
> >> try to build Parser/pgen with the cross-compiler. And try to run the
> >> generated executable. And fail. I know you can use `make touch` to
> >get
> >> around this, but it still sucks.
> >>
> >
> >True there is no separation of host/target compilers, but are you sure
> >this
> >matters? In my cross-compiling for iOS, the pgen steps are skipped --
> >Parser/pgen is compiled, but not ever used. Are you sure that this
> >failed
> >execution isn't an artefact of identifying the Android build a "linux"?
> >I've had to make a bunch of small modifications to configure.ac to make
> >the
> >iOS build process not build and run things that won't work on the build
> >platform.
> >
>
> Hmmm...
>
> But you gave an example. You shouldn't need to edit configure.ac to not
> try to run what can't be run because the host != the builder.
>

Agreed. My claim is that the behaviour you're seeing is a bug in the
existing configure script, probably born of the fact that using the "linux"
system type for Android confuses the build system into thinking that the
host executable will run on the build machine.

I suspect that the fix will be quite simple, and will reveal itself almost
immediately if you build with --host=armv7-google-android instead of
--host=armv7-linux-gnu or --host=armv7-linux-android

The existence of bugs isn't a reason to replace the entire build system,
unless you've got good reason to believe that a new build system won't be
subject to the same class of bugs.

>> - For some reason, autoconf decided to re-configure everything about
> >2
> >> minutes into make. But it didn't pass any of the flags I needed,
> >resulting
> >> in everything failing.
> >>
> >
> >That's the step compiling libffi/ctypes. libffi is vendored into the
> >Python
> >source, and the mid-build configure step is the configuration of that
> >vendored source code. You'll see the mid-build configure execute
> >regardless
> >of whether you're cross-compiling or not -- unless you're for OS/X,
> >because
> >the build instructions for OS/X libffi are hard-coded into setup.py.
> >
>
> Still sucks.
>

I don't see why. libffi is a complex piece of machinery. Every platform/CPU
architecture needs a different set of files to be compiled and included.
libffi's sources includes those definitions.

The alternative is hard-coding every possible target platform. Using
configure (the same configure script that libffi ships, I might add) means
this isn't required.

I'd also add that the other alternative is to configure using
--with-system-ffi, and ensure there is an appropriate libffi.so in your
path, etc. No internal configure step is required in this case.

>> - I do not like having to use this for configuring:
> >>
> >>    PYTHON_FOR_BUILD=`realpath ../cpython/python`
> >> CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++
> >> ./configure --host=arm-linux-androideabi --build=x86_64
> >--disable-ipv6
> >> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no
> >> ac_cv_have_long_long_format=yes
> >>
> >>
> >The PYTHON_FOR_BUILD specification *shouldn't* be necessary. If you're
> >compiling Python 3.X, as long as the build platform has a python3.X
> >executable on the path, PYTHON_FOR_BUILD should auto detect. It's only
> >necessary to manually specify PYTHON_FOR_BUILD if you *haven't* got an
> >appropriate python in your path.
>
> The build was giving errors because apparently 3.4's _sre module doesn't
> have some constant that the build script needs (MAXGROUPS).
>
> >
> >ac_cv_file__dev_ptmx=no and ac_cv_file__dev_ptc=no are manually
> >specified
> >flags because they can't be auto detected on a cross-compile (at least,
> >not
> >in any way I can think of). They're checking whether the *target*
> >device
> >has /dev/ptc and /dev/ptmx. In the case of Android/iOS, you could
> >probably
> >hard-code the responses on the basis that the target platform is a
> >known
> >quantity, but in the general case (which is the current case), you
> >can't
> >auto detect the properties of the target platform.
>
> It took me a while to figure out that that was the option I needed to
> pass, though.
>

Granted it isn't obvious. However, I'd suggest the logical extension of
that argument is "Lets document the existing build system", not "Lets
replace the build system with something new".

>I can't comment on ac_cv_have_long_long_format, but I'm guessing the
> >cause
> >is the same - you can't compile and run the configure test program.
> >
> >--disable-ipv6 is a feature flag - I don't know why it's necessary on
> >Android.
>
> I don't really know why, either. Something to do with a getaddrline bug.
>

Sure. And if it turns out that there's a good systemic reason why ipv6
won't ever work on Android, it's a couple of line patch to configure.ac
(around line 2666) to fix the problem permanently, removing the need for
the manual flag to be specified.

>At the same time, I'm not quite sure how many people would want to sift
> >> through the 5000 lines in configure.ac or the 1650 lines in
> >> Makefile.pre.in.
> >>
> >> I just get the feeling that, whenever I build Python, there are too
> >many
> >> holes/hacks to get around make/autoconf things.
> >>
> >
> >Sure - configure sucks. What's the alternative?
>
> All 1000 other build systems out there?
>

Care to state a specific preference, rather than engage in hyperbole?

>Sometimes things are complex because... they're complex. This is a
> >build
> >system that will support building everything from AiX to Android, and
> >in
> >*most* cases, with little more than a ./configure && make && make
> >install.
> >Android (and other cross compiling cases) are a little more complex
> >because
> >you have to *tell* configure what the target looks like, because it
> >can't
> >detect anything about the platform.
>
> But autotools tends to make things more complex than needed. When Qt (or
> was it KDE?) moved to CMake from autotools, the build script size dropped a
> large amount.
>
> Just take a look at the code in configure.ac that manually detects the
> compiler on OSX systems because autoconf SOMEHOW manages to get it wrong.
>

<sarcasm>
Wow - who would have thought that a complex set of build requirements that
have to work on a wide range of platforms and has been developed over 20+
years would occasionally yield edge cases?
</sarcasm>

Seriously, if your claim is that moving to a new build system will remove
the existence of edge cases, I want some of what you are smoking.

To address the specific example you gave: OSX is a particularly complex
platform to support because it spans 2 completely different CPU
architectures, has the concept of Framework builds that other platforms
don't have, and has, at various time, used clang, llvm-gcc and gcc as the
default compiler. Net result: AC_PROG_CC autoconf tool doesn't handle this
complexity well. But then, most build systems don't. However, Autoconf is
rich enough that it is possible to work around this problem.

I can't speak for the core team, but from my perspective, this sounds like
a proposal where the only argument that is going to work is a working (or
mostly working) port of the existing build system that proves it is
possible to support all the existing edge cases in a much simpler fashion.

For my money - of all the problems I've hit trying to port Python to iOS,
autoconf is the least of them. No, it isn't my favourite tool - but for all
it's faults, it works. If you want to convince me that there is a better
alternative, you're going to need to do more than just repeating that
"autoconf sucks".

Yours,
Russ Magee %-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150310/b4c36536/attachment.html>