buildtime vs runtime in Distutils
Hello, http://bugs.python.org/issue4359 reminds me that Distutils reads build files like Makefile or pyconfig.h to get some environment variables through the sysconfig module at *runtime*. This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora) Since I am already working on the refactoring of this distutils/sysconfig module, so it lands in a stdlib module together with some elements from site.py (see a previous mail on the topic, and my tarek_sysconfig branch - work in progress), I was wondering if we couldn't make this module a template, and inject at its beginning new values extracted from Makefile and pyconfig.h. This could happen when "configure" is called, Regards, Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与
http://bugs.python.org/issue4359 reminds me that Distutils reads build files like Makefile or pyconfig.h to get some environment variables through the sysconfig module at *runtime*.
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
I don't see a problem with that: you'll need the python-devel package *anyway* when running distutils, for many packages. Regards, Martin
2009/11/15 "Martin v. Löwis" <martin@v.loewis.de>:
http://bugs.python.org/issue4359 reminds me that Distutils reads build files like Makefile or pyconfig.h to get some environment variables through the sysconfig module at *runtime*.
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
I don't see a problem with that: you'll need the python-devel package *anyway* when running distutils, for many packages.
The problem is that the main python distribution ("python") is not working as advertised since it contains distutils, which requires "python-devel" to work. This implies that "python" has a dependency on "python-devel", which does not make sense anymore for linux distros to have two distinct packages for Python. Having some of the makefile vars stored in stdlib solve this problem. Tarek
The problem is that the main python distribution ("python") is not working as advertised since it contains distutils, which requires "python-devel" to work.
This implies that "python" has a dependency on "python-devel", which does not make sense anymore for linux distros to have two distinct packages for Python.
Having some of the makefile vars stored in stdlib solve this problem.
I don't see how this can solve the problem. Distutils contains the build_ext command, which definitely will require the devel package. Regards, Martin
2009/11/15 "Martin v. Löwis" <martin@v.loewis.de>:
The problem is that the main python distribution ("python") is not working as advertised since it contains distutils, which requires "python-devel" to work.
This implies that "python" has a dependency on "python-devel", which does not make sense anymore for linux distros to have two distinct packages for Python.
Having some of the makefile vars stored in stdlib solve this problem.
I don't see how this can solve the problem. Distutils contains the build_ext command, which definitely will require the devel package.
Sure that's the edge case where files like Python.h is required for compiling C extensions. And compiler errors can make it pretty clear at this level, that the devel package is required. Not because of the need to read a Makefile, but to get the header files to compile. And it should be possible to compile extensions that don't require python-devel at all, when these extensions are not CPython extensions. Last, commands like "install" also requires using sysconfig, (thus reading Makefile) to get some values. I can change this for "install" it's not hard, but we would still provide some public APIs that are reading Makefile from within the stlib. right now, importing "disutils.sysconfig" triggers this behavior. That makes several use case of requiring python-devel just to read a few values that could be injected in the stdlib at build time. Regards Tarek
2009-11-15 02:21:41 Tarek Ziadé napisał(a):
Hello,
http://bugs.python.org/issue4359 reminds me that Distutils reads build files like Makefile or pyconfig.h to get some environment variables through the sysconfig module at *runtime*.
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
Since I am already working on the refactoring of this distutils/sysconfig module, so it lands in a stdlib module together with some elements from site.py (see a previous mail on the topic, and my tarek_sysconfig branch - work in progress),
I was wondering if we couldn't make this module a template, and inject at its beginning new values extracted from Makefile and pyconfig.h.
This could happen when "configure" is called,
It seems to be a good idea. You should create a .py.in file and use AC_SUBST and AC_CONFIG_FILES macros in configure.in. -- Arfrever Frehtes Taifersar Arahesis
On Sun, Nov 15, 2009 at 2:41 AM, Arfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com> wrote: [..]
This could happen when "configure" is called,
It seems to be a good idea. You should create a .py.in file and use AC_SUBST and AC_CONFIG_FILES macros in configure.in.
I need to investigate on these, thx Tarek
Tarek Ziadé wrote:
Hello,
http://bugs.python.org/issue4359 reminds me that Distutils reads build files like Makefile or pyconfig.h to get some environment variables through the sysconfig module at *runtime*.
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
Do we really want to change distutils to solve a problem of a third party packaging system when the problem was created by the very same third party in the first place? In other words: Should you spend your precious development time with fixing a problem that isn't our fault? The decision to split the header files into a separate package, that isn't installed by default, has already created tons of bad user experience in the past. Do you want to endorse the split even further? -0 from me Christian
Christian Heimes wrote:
Tarek Ziadé wrote:
Hello,
http://bugs.python.org/issue4359 reminds me that Distutils reads build files like Makefile or pyconfig.h to get some environment variables through the sysconfig module at *runtime*.
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
Do we really want to change distutils to solve a problem of a third party packaging system when the problem was created by the very same third party in the first place? In other words: Should you spend your precious development time with fixing a problem that isn't our fault? The decision to split the header files into a separate package, that isn't installed by default, has already created tons of bad user experience in the past. Do you want to endorse the split even further?
-0 from me
If the third party disables distutils by removing some of the things it (sometimes) needs, then it seems to me that they should also remove disutils into the separate package. If *that* creates a problem, that should be *their* problem.
On Sun, Nov 15, 2009 at 4:02 AM, Terry Reedy <tjreedy@udel.edu> wrote:
Christian Heimes wrote:
Tarek Ziadé wrote:
Hello,
http://bugs.python.org/issue4359 reminds me that Distutils reads build files like Makefile or pyconfig.h to get some environment variables through the sysconfig module at *runtime*.
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
Do we really want to change distutils to solve a problem of a third party packaging system when the problem was created by the very same third party in the first place? In other words: Should you spend your precious development time with fixing a problem that isn't our fault? The decision to split the header files into a separate package, that isn't installed by default, has already created tons of bad user experience in the past. Do you want to endorse the split even further?
-0 from me
If the third party disables distutils by removing some of the things it (sometimes) needs, then it seems to me that they should also remove disutils into the separate package. If *that* creates a problem, that should be *their* problem.
Ok. Fair enough, I'll work with them this way. Regards Tarek
On Sun, Nov 15, 2009 at 10:32 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Ok. Fair enough, I'll work with them this way.
Although packagers should certainly fix the problems they introduce in the first place, the second suggestion in the bug report would be useful, independently on how linux distributions package things. Especially if the data can be obtained for every build (autoconf and VS-based), this would help packages which use something else than distutils for their build. David
On Sun, Nov 15, 2009 at 3:35 AM, Christian Heimes <lists@cheimes.de> wrote: [..]
Do we really want to change distutils to solve a problem of a third party packaging system when the problem was created by the very same third party in the first place? In other words: Should you spend your precious development time with fixing a problem that isn't our fault? The decision to split the header files into a separate package, that isn't installed by default, has already created tons of bad user experience in the past. Do you want to endorse the split even further?
I didn't know the split story went like this. I took it like the "natural" split everyone agreed on, and I saw this distutils <-> Makefile link like something to fix. So, it sounds like a bad idea now :) Tarek.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tarek Ziadé wrote:
On Sun, Nov 15, 2009 at 3:35 AM, Christian Heimes <lists@cheimes.de> wrote: [..]
Do we really want to change distutils to solve a problem of a third party packaging system when the problem was created by the very same third party in the first place? In other words: Should you spend your precious development time with fixing a problem that isn't our fault? The decision to split the header files into a separate package, that isn't installed by default, has already created tons of bad user experience in the past. Do you want to endorse the split even further?
I didn't know the split story went like this. I took it like the "natural" split everyone agreed on, and I saw this distutils <-> Makefile link like something to fix.
So, it sounds like a bad idea now :)
Parsing the Makefile at runtime seems like an insane choice anyway to me: +1 for your new module having constants generated at ./configure time. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksA1LEACgkQ+gerLs4ltQ5gCQCgtYpBewlnocHJf5hp33TfkLjG 72IAnRW1d9n2CO8S2V+9ewcMb81oWNL+ =GG45 -----END PGP SIGNATURE-----
Tres Seaver wrote:
Tarek Ziadé wrote:
I didn't know the split story went like this. I took it like the "natural" split everyone agreed on, and I saw this distutils <-> Makefile link like something to fix.
So, it sounds like a bad idea now :)
Parsing the Makefile at runtime seems like an insane choice anyway to me: +1 for your new module having constants generated at ./configure time.
I'm with Tres here - having distutils (aside from build_ext) depend on non-Python parts of the source tree seem a little strange. However, given that the recommended packaging includes the needed files, putting an RFE on the tracker to reduce the runtime dependency on the source code would be an acceptable response if there is something else you'd rather be working on. Cheers, Nick. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Sun, Nov 15, 2009 at 11:27:29PM -0500, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Tarek Ziadé wrote:
On Sun, Nov 15, 2009 at 3:35 AM, Christian Heimes <lists@cheimes.de> wrote: [..]
Do we really want to change distutils to solve a problem of a third party packaging system when the problem was created by the very same third party in the first place? In other words: Should you spend your precious development time with fixing a problem that isn't our fault? The decision to split the header files into a separate package, that isn't installed by default, has already created tons of bad user experience in the past. Do you want to endorse the split even further?
I didn't know the split story went like this. I took it like the "natural" split everyone agreed on, and I saw this distutils <-> Makefile link like something to fix.
So, it sounds like a bad idea now :)
Parsing the Makefile at runtime seems like an insane choice anyway to me: +1 for your new module having constants generated at ./configure time.
+1 There have been bugs in the past about distutils.sysconfig.parse_makefile() not knowing the full make syntax and there will no doubt still be bugs like this. Substituting these variables at ./configure time into a .py.in file would be much safer. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Sun, Nov 15, 2009 at 03:35:00AM +0100, Christian Heimes wrote:
Do we really want to change distutils to solve a problem of a third party packaging system when the problem was created by the very same third party in the first place? In other words: Should you spend your precious development time with fixing a problem that isn't our fault? The decision to split the header files into a separate package, that isn't installed by default, has already created tons of bad user experience in the past. Do you want to endorse the split even further?
The problem being addressed is that distutils is using an artefact of the build system (a Makefile) as a data file. I think the issue of whether or not to "endorse the split" is a red herring. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
This practice is stupid anyway, because it means you have to install python-devel even to install pure Python packages with setuptools/distribute. Just ask Fedora, Mandriva and friends to change their packaging practice (Mandriva already has a bug open for that by the way). In Debian/Ubuntu, the Makefile is correctly part of the main Python package: $ dpkg -S /usr/lib/python2.5/config/Makefile python2.5: /usr/lib/python2.5/config/Makefile Regards Antoine.
Antoine Pitrou schrieb:
Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
This practice is stupid anyway, because it means you have to install python-devel even to install pure Python packages with setuptools/distribute. Just ask Fedora, Mandriva and friends to change their packaging practice (Mandriva already has a bug open for that by the way).
+1. They are the ones splitting what "make install" installs into several packages, so they are the ones who have to fix the resulting dependency problems. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
On Sun, Nov 15, 2009 at 05:31, Georg Brandl <g.brandl@gmx.net> wrote:
Antoine Pitrou schrieb:
Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
This practice is stupid anyway, because it means you have to install python-devel even to install pure Python packages with setuptools/distribute. Just ask Fedora, Mandriva and friends to change their packaging practice (Mandriva already has a bug open for that by the way).
+1. They are the ones splitting what "make install" installs into several packages, so they are the ones who have to fix the resulting dependency problems.
+1 from me as well. Python is designed to run as a whole. If they choose to ignore our design decisions they do so at their own peril. Now if you want to put the time in, Tarek, to make sure distutils can be removed from the stdlib and have everything still work so the linux distros can add distutils to python-devel that's fine, but once again, they are the ones mucking around with things in a way we did not design for, so only do it if you really want to. -Brett
On Sun, Nov 15, 2009 at 02:31:45PM +0100, Georg Brandl wrote:
Antoine Pitrou schrieb:
Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
This practice is stupid anyway, because it means you have to install python-devel even to install pure Python packages with setuptools/distribute. Just ask Fedora, Mandriva and friends to change their packaging practice (Mandriva already has a bug open for that by the way).
+1. They are the ones splitting what "make install" installs into several packages, so they are the ones who have to fix the resulting dependency problems.
I agree with this, however, my point on the bug was more akin to this: Tres Seaver wrote:
Parsing the Makefile at runtime seems like an insane choice anyway to me: +1 for your new module having constants generated at ./configure time.
Makefiles and C header files are not intended as general purpose data formats. Using them as such has a variety of disadvantages: * If someone else wants to get at the data, they have to go through the API in distutils. Any data that's not exposed by the API is unavailable. * Since disturils doesn't implement a full parser for the make and C syntax it is possible to break distutils when making legitimate changes to those build files. These are the reasons I opened the bug to get that information into a real data file rather than parsing the Makefile and header files. I'll also mention two further things: The reason that python-devel was split off was to make it more useful for livecds, olpc, embedded systems, and other places where disk space is at a premimum. Being able to combine an operating system that is used by people beyond your immediate community is great for finding and fixing bugs before your users run into them. Being able to program in a high level language on these platforms has benefits that I'm sure everyone here can appreciate. I've brought the issue of Makefile and pyconfig.h being needed for distutils to the attention of every new Fedora python maintainer since the package split was made. The current maintainer, David Malcolm, agrees that distutils.sysconfig needs to be able to use this data and he has moved the Makefile and header files into the main python package. This doesn't change the problems with using a Makefile and C header files as a data format for python. -Toshio
On Mon, Nov 16, 2009 at 8:15 PM, Toshio Kuratomi <a.badger@gmail.com> wrote: [..]
I've brought the issue of Makefile and pyconfig.h being needed for distutils to the attention of every new Fedora python maintainer since the package split was made. The current maintainer, David Malcolm, agrees that distutils.sysconfig needs to be able to use this data and he has moved the Makefile and header files into the main python package. This doesn't change the problems with using a Makefile and C header files as a data format for python.
Great News ! Now for the format problem, I agree that it seems more robust to pre-process the variables and inject them in the stdlib when ./configure is run. I am not sure what is the best strategy here, but I would rather not add yet another configuration file (wether its an xml format or an ini-like format). So what I am proposing is to inject those values in a private dict in the new sysconfig.py module, that can be read through the get_config_vars / get_config_var APIs. This means that sysconfig.py will be added as "sysconfig.py.in" Regards Tarek
Tarek Ziadé wrote:
Now for the format problem, I agree that it seems more robust to pre-process the variables and inject them in the stdlib when ./configure is run.
I am not sure what is the best strategy here, but I would rather not add yet another configuration file (wether its an xml format or an ini-like format).
So what I am proposing is to inject those values in a private dict in the new sysconfig.py module, that can be read through the get_config_vars / get_config_var APIs.
This means that sysconfig.py will be added as "sysconfig.py.in"
Are you planing to implement the proposal for Windows, too? There may be no need to change the code because we have full control over the official installers. Also the VisualStudio Windows build doesn't use ./configure but hand crafted header files. You'd have to fall back to a hand crafter sysconfig.py. Christian
Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
So what I am proposing is to inject those values in a private dict in the new sysconfig.py module, that can be read through the get_config_vars / get_config_var APIs.
This means that sysconfig.py will be added as "sysconfig.py.in"
This means you have to relaunch the whole configure thing each time you make a change to sysconfig? This doesn't sound like a good idea to me. You could make the private dict a dedicated module instead, e.g. "_sysconfig_data.py.in". As for Windows, if people have the ability to choose the installation directory when installing, I'm not sure how you're gonna handle it.
On Thu, Nov 19, 2009 at 11:04 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
So what I am proposing is to inject those values in a private dict in the new sysconfig.py module, that can be read through the get_config_vars / get_config_var APIs.
This means that sysconfig.py will be added as "sysconfig.py.in"
This means you have to relaunch the whole configure thing each time you make a change to sysconfig? This doesn't sound like a good idea to me. You could make the private dict a dedicated module instead, e.g. "_sysconfig_data.py.in".
That's much better indeed.
As for Windows, if people have the ability to choose the installation directory when installing, I'm not sure how you're gonna handle it.
I don't know. I am investigating with what Christian has explained on Windows installers, to see how it can be hooked in the installation process, and a fallback for the win32 build version. If it gets too complex, I'll probably postpone this and finish a first version of sysconfig that does use Makefile. Tarek
Tarek Ziadé wrote:
On Mon, Nov 16, 2009 at 8:15 PM, Toshio Kuratomi <a.badger@gmail.com> wrote: [..]
I've brought the issue of Makefile and pyconfig.h being needed for distutils to the attention of every new Fedora python maintainer since the package split was made. The current maintainer, David Malcolm, agrees that distutils.sysconfig needs to be able to use this data and he has moved the Makefile and header files into the main python package. This doesn't change the problems with using a Makefile and C header files as a data format for python.
Great News !
Now for the format problem, I agree that it seems more robust to pre-process the variables and inject them in the stdlib when ./configure is run.
I am not sure what is the best strategy here, but I would rather not add yet another configuration file (wether its an xml format or an ini-like format).
So what I am proposing is to inject those values in a private dict in the new sysconfig.py module, that can be read through the get_config_vars / get_config_var APIs.
This means that sysconfig.py will be added as "sysconfig.py.in"
Wouldn't it be cleaner to place the extracted information into a new module _sysconfig_settings.py which gets created by the configure script or the Makefile later during the build ? This new module could then be imported into sysconfig.py at run-time. BTW: There's already a Tools/scripts/h2py.py we could use for converting pyconfig.h into a Python file. For the Makefile we could use the distutils Makefile parser to add the extra values. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 19 2009)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Nov 19, 2009 at 12:04 PM, M.-A. Lemburg <mal@egenix.com> wrote: [..]
BTW: There's already a Tools/scripts/h2py.py we could use for converting pyconfig.h into a Python file.
Good to know,
For the Makefile we could use the distutils Makefile parser to add the extra values.
Yes, there's also a pyconfig.h parser function. It seems though, for Makefile, that the tools available during the build phase are simpler/better at extracting values. (currently experimenting those) Tarek
Tarek Ziadé <ziade.tarek <at> gmail.com> writes:
This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora)
This practice is stupid anyway, because it means you have to install python-devel even to install pure Python packages with setuptools/distribute. Just ask Fedora, Mandriva and friends to change their packaging practice (Mandriva already has a bug open for that by the way).
In Debian/Ubuntu, the Makefile is correctly part of the main Python package:
$ dpkg -S /usr/lib/python2.5/config/Makefile python2.5: /usr/lib/python2.5/config/Makefile For what it's worth, this is fixed in Fedora as of python-2.6.4-2.fc13 ;
On Sun, 2009-11-15 at 12:42 +0000, Antoine Pitrou wrote: the bug tracking it was: https://bugzilla.redhat.com/show_bug.cgi?id=531901
On Mon, Nov 16, 2009 at 10:09 PM, David Malcolm <dmalcolm@redhat.com> wrote: [..]
$ dpkg -S /usr/lib/python2.5/config/Makefile python2.5: /usr/lib/python2.5/config/Makefile
For what it's worth, this is fixed in Fedora as of python-2.6.4-2.fc13 ; the bug tracking it was: https://bugzilla.redhat.com/show_bug.cgi?id=531901
Thanks David !
participants (16)
-
"Martin v. Löwis" -
Andrew McNabb -
Antoine Pitrou -
Arfrever Frehtes Taifersar Arahesis -
Brett Cannon -
Christian Heimes -
David Cournapeau -
David Malcolm -
Floris Bruynooghe -
Georg Brandl -
M.-A. Lemburg -
Nick Coghlan -
Tarek Ziadé -
Terry Reedy -
Toshio Kuratomi -
Tres Seaver