Thanks all for the responses. Clearly this is a subject about which
people feel strongly, so that's good at least. David Murray's
guidance
in particular points to the most likely path to get improvements to
really happen.
Steve Dower:
> Building CPython for Windows is not something that needs
solving.
Not in your opinion, but numerous packagers of MinGW-based native or
cross-compiled package sets would love to include Python. The fact
that they currently can't, without many patches, is a problem.
> The culture on Windows is to redistribute binaries, not source,
There are many cultures using Windows. Including open-source ones.
> and both the core team and a number of redistributors have this
figured
> out (and it will only become easier with VC14 and Python 3.5).
With MSVC. It doesn't work with MinGW, it likely doesn't work with
Clang.
MSVC is not the only compiler on Windows. There are many use cases
for
preferring other compilers. Have you read this wiki page for example?
In my personal experience, having recently gotten Julia to compile
using
MSVC for the first time, MSVC as a compiler is highly deficient for
many
needs especially in the scientific software community:
- C99 (getting better recently, but still not done)
- AT&T syntax assembly
- C++11 features (also mostly okay now, but not if you're using an
older
MSVC version with Python 2.7, which many people still have to
do)
- 128-bit integer intrinsics
- cannot cross-compile from anything that isn't Windows
- build systems foreign relative to shell/makefile systems used by
most
open-source projects, few projects have time to maintain 2 separate
build
systems (cmake helps but takes a lot of initial effort to convert
to)
- no free-as-in-beer Fortran compiler available
I have none of these problems when I use MinGW-w64. Hence the desire
to
be able to curate an all-MinGW software stack. It's not a matter of
open-
source ideology for me, it's brass tacks "can I do the work I need to
do."
With MSVC I can't, with MinGW-w64 I can. Not being able to include
CPython
in an all-MinGW stack hurts, a lot.
Only cross-compilation and the build system in the above list are
relevant
to CPython, but I hope I have convinced you, Paul Moore, etc. that there
are
real reasons for some groups of users and developers to prefer
MinGW-w64
over MSVC.
> I'd rather see this effort thrown behind compiling extensions,
> including cross compilation.
There are patches awaiting review that improve this as well. Efforts
to
improve CPython's build system and the handling of extensions are not
completely independent, in many cases the patches are written by the
same
set of MinGW users. One of these sets of patches is not inherently
evil,
you understandably have less interest in them but it's still
disappointing
to see so little movement on either.
> Having different builds of CPython out there will only fragment
the
> community and hurt extension authors far more than it may seem to
help.
The community of people developing and using open-source projects,
either
CPython or otherwise, is already highly fragmented. Ignoring it makes
it
worse. python.org does not have to distribute or endorse
MinGW-compiled
builds of CPython. If the build option never gets incorporated, then
it
will continue to be reverse-engineered.
Guido van Rossum:
> Here's the crux of the matter. We want compiled extension
modules
> distributed via PyPI to work with the binaries distributed from
python.org.
Absolutely. I don't think additional options in the build system
would
change this.
R. David Murray:
> And, at this point, we would NEED A BUILDBOT. That is, a machine
that
> has whatever tools are required installed such that tests added to
the
> test suite to test MinGW support in distutils would run, so we can
be
> sure we don't break anything when making other changes.
That's not too hard. I've done this for other projects. AppVeyor works
if
your build is short enough, and I've done cross-compilation from
Travis
CI for other projects. Or Jenkins, or a Vagrant VM. I don't know
PSF's
infrastructure, but I can offer guidance if it would help.
Steve Dower:
> I'm afraid of users having numpy crash because they're using an
MSVC
> CPython instead of a mingw CPython. I'm afraid of users not being
able
> to use library A and library B at the same time because A requires
MSVC
> CPython and B requires mingw CPython. (I can produce more examples if
you
> like, but the general concern is having a fragmented community, as I
said
> in my previous post.)
A valid fear. Mixing C runtimes can cause problems, I've seen this
myself.
Correct me if I'm wrong, but this is nearly as much of an issue if
someone
wants to use a different version of MSVC to compile CPython than the
version
used to build the official binaries. It requires care, but you can't
deny
that there are use cases where people will want and need to do such
things.
Is possible fragmentation a good enough reason to resist making it
possible
in the build system?
> though I suspect most would like to see some traction achieved on a
fork
> first
Those of us who consider this important should probably just do this.
Ray,
Roumen, the maintainer of the Arch MinGW packages, myself and others
could
look into making an actual fork on Github or Bitbucket where we merge
the
various patches and come up with an out-of-the-box
MinGW-[cross-]compilable
version of CPython. I'll happily write the spec files to get this
building
from Fedora or openSUSE. That would help us test the feasibility from
a
centralized repository. Ray, what do you think? Do you know xantares'
email
address to ask if he'd be interested in helping or using the result?
Zachary Ware:
> I'm failing to see where that's simpler :)
If it were hypothetically merged instead of out in an external fork,
it
could be ./configure --host=x86_64-w64-mingw32 && make to
cross-compile
from Linux or Cygwin, or just ./configure && make after installing
MSYS2
(which is just about as easy as installing MSVC) on Windows.
Paul Moore:
> If it were possible to cross-compile compatible extensions on
Linux,
> projects developed on Linux could supply Windows binaries much
more
> easily, which would be a huge benefit to the whole Windows
Python
> community.
I want to do exactly this in an automated repeatable way, preferably
on
a build service. This seems harder to do when CPython cannot itself
be
built and handled as a dependency by that same automated, repeatable
build service. Unless it becomes possible to cross-compile extensions
using the build machine's own version of Python, which might be the
right
approach.
> acutely aware of the common pain points for Python users on
Windows.
> And they are all about binary extensions, and none at all about
> building Python itself.
I've done a lot of recent work keeping Julia working well on Windows,
and
the interoperability we have with Python packages has propagated most
of
these pain points to us as well. We have to rely on Conda in order to
have
a reliable way of installing, as an example, IPython with the
notebook
interface, in order for IJulia to work. This is not an ideal solution as
it
requires a great deal of user intervention and manual steps to get up
and
running (and it would be far worse without Conda). We are, so far,
built
around MinGW-w64 on Windows, for the reasons I listed above. Having
cross-
compiled builds of CPython and binary extensions available from the
same
build services we already use to install other binary packages (Gtk,
Cairo,
Tk, Nettle, HDF5, etc) on Windows would be enormously helpful for us.
There's a real use case. Its size and importance can be debated. For
now
I'll take David Murray's post to heart and see where I have time or
ability
to help things along.
Sincerely,
Tony