Can Python Be Built Without Distutils

Related to my earlier question regarding building Python on Android and an undefined reference to dlopen error...I have the following question: Is it possible to build and install Python without having to build and install...or use...distutils? Some background: I can build the python interpreter on my device, and I can build a bunch of modules. The problem appears when make reaches the part where setup.py is used to build and import modules...specifically when setup.py attempts to import distutils.core. Thanks, Cyd

On 01/23/2015 06:30 PM, Cyd Haselton wrote:
you can do this using Setup.local. This works for me building additional extensions as builtins. It might require some tweaking to build everything. Otoh, I would like to get rid off the setup.py altogether (/me ducks ...). Matthias

On 23.01.2015 19:48, Matthias Klose wrote:
You may want to have a look at the Setup files we're using in eGenix PyRun, which uses them to force static builds of the various built-in extensions. Look for these files: PyRun/Runtime/Setup.PyRun-2.7 PyRun/Runtime/Setup.PyRun-3.4 in the source archives: http://www.egenix.com/products/python/PyRun/
Otoh, I would like to get rid off the setup.py altogether (/me ducks ...).
Why ? It's great for finding stuff on your system and configuring everything without user intervention (well, most of the time :-)). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 23 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 23.01.2015 21:56, Cyd Haselton wrote:
No, but we're thinking of mirroring our open-source packages on Github or Bitbucket.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 23 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Fri Jan 23 2015 at 11:20:02 AM M.-A. Lemburg <mal@egenix.com> wrote:
Because our setup.py is a nightmare of arbitrary code run in a linear fashion with ad-hoc checks for things that are unlike how any other project on the planet determines what is available on your system. It may have seemed "great" when it was created in 2001. It really shows its age now. It defeats build parallelism and dependency declaration. It also prevents cross compilation. Building an interpreter with a limited standard library on your build host so that you can run said interpreter to have it drive the remainder of your build is way more self hosting that we rightfully need to be for CPython. -gps

On 24.01.2015 21:23, Zachary Ware wrote:
The Windows installers comes with all dependencies included. On other systems, this is not the case and so determining the various settings is done using Python, which is a lot easier to do than e.g. relying on autoconf finding everything and setting up the correct compile options for each built-in. All that said: if you can come up with a better system that's both easy to maintain and provides the same level of user friendliness, I'm sure this would be considered. BTW: Parallel execution, cross compilation can be added to setup.py. I don't think parallel execution is all that important, but cross compilation would certainly be an interesting feature to have (this currently has to be done using the Setup.in approach) and an option to build the modules statically linked to the interpreter would also be nice. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 24 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Sun Jan 25 2015 at 7:08:53 AM Antoine Pitrou <solipsis@pitrou.net> wrote:
Thanks, glad to see that this was added! :) This looks like a feature that was added in 3.5. Rather than passing the flag to setup.py from the Makefile (where I had been looking for it), setup.py looks at the MAKEFLAGS environment variable and turns on distutils.common.build_ext's parallel flag if "-j" is found in that. By default it will then do a parallel build using os.cpu_count() number of tasks. -gps

Since (judging from the lack of responses) setup.py can't be removed from the Makefile, I kept troubleshooting.I've managed to get the build to complete and make install runs instead of throwing an undefined reference right off the bat, unfortunately I've run into the following: ImportError: No module named _struct This also happens when make test is run. I;ve checked...the _struct module is there. On Sat, Jan 24, 2015 at 11:22 PM, Gregory P. Smith <greg@krypto.org> wrote:

I tried a Makefile based build of python (+ some module) in the past for Android (and macos): https://bitbucket.org/cavallo71/android There was no particular problem in dropping autoconfigure+setup.py in the process: the only real problem was the pgen must be compiled on the host machine (but that could have changed since). In general python was a plain compile and link stuff with not much magic involved and that is (very) good in my opinion: magic detection tends to fail, autoconfigure need to "execute" stuff to detect things (reason why is quite useless in cross-compile mode) etc. I would've tried also cmake and qmake builds as well, but simply I didn't have time to spend on it. I hope this helps, Antonio Zachary Ware wrote:

Thanks...this looks interesting Antonio. I'm not familiar enough with Makefile syntax and creation to know if this can help in my case and (fingers crossed) I believe I've just about solved my original undefined reference to dlopen problem, but I'll be bookmarking it for future reference (and in case I'm wrong about being close to solving my original Android build issues). On Sun, Jan 25, 2015 at 4:08 AM, Antonio Cavallo <a.cavallo@cavallinux.eu> wrote:

On Fri, Jan 23, 2015 at 12:48 PM, Matthias Klose <doko@ubuntu.com> wrote:
Regarding previous question, build completes when using Setup, and changing *shared* to *static* but 'make install' fails. If I run 'make altbininstall' or 'make altmaninstall' thst works...actually they all work except any of the libinstall targets, which is probably because in the makefile all lib install targets run setup.py. Is there a special way to install python without setup.py

On 01/23/2015 06:30 PM, Cyd Haselton wrote:
you can do this using Setup.local. This works for me building additional extensions as builtins. It might require some tweaking to build everything. Otoh, I would like to get rid off the setup.py altogether (/me ducks ...). Matthias

On 23.01.2015 19:48, Matthias Klose wrote:
You may want to have a look at the Setup files we're using in eGenix PyRun, which uses them to force static builds of the various built-in extensions. Look for these files: PyRun/Runtime/Setup.PyRun-2.7 PyRun/Runtime/Setup.PyRun-3.4 in the source archives: http://www.egenix.com/products/python/PyRun/
Otoh, I would like to get rid off the setup.py altogether (/me ducks ...).
Why ? It's great for finding stuff on your system and configuring everything without user intervention (well, most of the time :-)). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 23 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 23.01.2015 21:56, Cyd Haselton wrote:
No, but we're thinking of mirroring our open-source packages on Github or Bitbucket.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 23 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Fri Jan 23 2015 at 11:20:02 AM M.-A. Lemburg <mal@egenix.com> wrote:
Because our setup.py is a nightmare of arbitrary code run in a linear fashion with ad-hoc checks for things that are unlike how any other project on the planet determines what is available on your system. It may have seemed "great" when it was created in 2001. It really shows its age now. It defeats build parallelism and dependency declaration. It also prevents cross compilation. Building an interpreter with a limited standard library on your build host so that you can run said interpreter to have it drive the remainder of your build is way more self hosting that we rightfully need to be for CPython. -gps

On 24.01.2015 21:23, Zachary Ware wrote:
The Windows installers comes with all dependencies included. On other systems, this is not the case and so determining the various settings is done using Python, which is a lot easier to do than e.g. relying on autoconf finding everything and setting up the correct compile options for each built-in. All that said: if you can come up with a better system that's both easy to maintain and provides the same level of user friendliness, I'm sure this would be considered. BTW: Parallel execution, cross compilation can be added to setup.py. I don't think parallel execution is all that important, but cross compilation would certainly be an interesting feature to have (this currently has to be done using the Setup.in approach) and an option to build the modules statically linked to the interpreter would also be nice. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 24 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Sun Jan 25 2015 at 7:08:53 AM Antoine Pitrou <solipsis@pitrou.net> wrote:
Thanks, glad to see that this was added! :) This looks like a feature that was added in 3.5. Rather than passing the flag to setup.py from the Makefile (where I had been looking for it), setup.py looks at the MAKEFLAGS environment variable and turns on distutils.common.build_ext's parallel flag if "-j" is found in that. By default it will then do a parallel build using os.cpu_count() number of tasks. -gps

Since (judging from the lack of responses) setup.py can't be removed from the Makefile, I kept troubleshooting.I've managed to get the build to complete and make install runs instead of throwing an undefined reference right off the bat, unfortunately I've run into the following: ImportError: No module named _struct This also happens when make test is run. I;ve checked...the _struct module is there. On Sat, Jan 24, 2015 at 11:22 PM, Gregory P. Smith <greg@krypto.org> wrote:

I tried a Makefile based build of python (+ some module) in the past for Android (and macos): https://bitbucket.org/cavallo71/android There was no particular problem in dropping autoconfigure+setup.py in the process: the only real problem was the pgen must be compiled on the host machine (but that could have changed since). In general python was a plain compile and link stuff with not much magic involved and that is (very) good in my opinion: magic detection tends to fail, autoconfigure need to "execute" stuff to detect things (reason why is quite useless in cross-compile mode) etc. I would've tried also cmake and qmake builds as well, but simply I didn't have time to spend on it. I hope this helps, Antonio Zachary Ware wrote:

Thanks...this looks interesting Antonio. I'm not familiar enough with Makefile syntax and creation to know if this can help in my case and (fingers crossed) I believe I've just about solved my original undefined reference to dlopen problem, but I'll be bookmarking it for future reference (and in case I'm wrong about being close to solving my original Android build issues). On Sun, Jan 25, 2015 at 4:08 AM, Antonio Cavallo <a.cavallo@cavallinux.eu> wrote:

On Fri, Jan 23, 2015 at 12:48 PM, Matthias Klose <doko@ubuntu.com> wrote:
Regarding previous question, build completes when using Setup, and changing *shared* to *static* but 'make install' fails. If I run 'make altbininstall' or 'make altmaninstall' thst works...actually they all work except any of the libinstall targets, which is probably because in the makefile all lib install targets run setup.py. Is there a special way to install python without setup.py
participants (8)
-
Antoine Pitrou
-
Antonio Cavallo
-
Brett Cannon
-
Cyd Haselton
-
Gregory P. Smith
-
M.-A. Lemburg
-
Matthias Klose
-
Zachary Ware