I've updated my patch "make setup.py less chatty by default." http://sourceforge.net/tracker/?func=detail&atid=305470&aid=504889&group_id=5470 I've attached a new patch that uses a global log module. (The added file, log.py, is a separate attachment to this patch submission.) The patch is incomplete, as it removes the announce() method of several objects. I haven't bothered to replace all the calls to announce in the commands; that needs to be done before this patch can be accepted, but it's too tedious for today :-). The new approach is to eliminate all uses of verbose except one on the Distriubtion object that can be controlled by -v and -q arguments. The value of Distribution.verbose is used to set a logging threshold, stored in a global variable in the log module. All code that wants to print a message calls log.method(). The default verbosity is 1, which skips messages that say "skipping ...". It does print any activity actually performed, like calling a C compiler. I'm expect more thought needs to be put into the level that each message is logged at. Jeremy
Jeremy Hylton wrote:
I've updated my patch "make setup.py less chatty by default." http://sourceforge.net/tracker/?func=detail&atid=305470&aid=504889&group_id=5470
I've attached a new patch that uses a global log module. (The added file, log.py, is a separate attachment to this patch submission.) The patch is incomplete, as it removes the announce() method of several objects. I haven't bothered to replace all the calls to announce in the commands; that needs to be done before this patch can be accepted, but it's too tedious for today :-).
The new approach is to eliminate all uses of verbose except one on the Distriubtion object that can be controlled by -v and -q arguments. The value of Distribution.verbose is used to set a logging threshold, stored in a global variable in the log module. All code that wants to print a message calls log.method().
The default verbosity is 1, which skips messages that say "skipping ...". It does print any activity actually performed, like calling a C compiler.
I'm expect more thought needs to be put into the level that each message is logged at.
See my reply on SF. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ Meet us at EuroPython 2002: http://www.europython.org/
M.-A. Lemburg writes:
Good idea in general.
Removing .announce() however is not since this is used by third party tools extending distutils with their own subclasses (BTW, this is something you always have to keep in mind: distutils is made to be extended, so any change to the core API will break packaging tools).
Good point. Is the core API documented anywhere? I can't tell what is intended to be part of the API and what is accidentally exposed by the implementation. Some particulars: The getopt code is probably the most complex argument processing code I've ever seen. I've got no idea if I've preserved backwards compatibility where necessary. Does it mean we need to support all the optional verbose keyword args even though they will be ignored? Jeremy
On Fri, May 31, 2002 at 01:43:24PM -0400, Jeremy Hylton wrote:
Good point. Is the core API documented anywhere? I can't tell what is intended to be part of the API and what is accidentally exposed by the implementation.
No reference docs were ever written, AFAIK, so we'll just have to follow Python convention: methods prefixed with an underscore are private, otherwise public.
The getopt code is probably the most complex argument processing code I've ever seen. I've got no idea if I've preserved backwards compatibility where necessary.
Assuming OptionParser (or whatever Greg names it) gets checked in, distutils.fancy_getopt can get deprecated.
Does it mean we need to support all the optional verbose keyword args even though they will be ignored?
I guess so; maybe there should be a warning when 'verbose' is supplied. It would be nice if there was a list of packages that do fancy Distutil customization, so that we'd know what to test against. Anyone want to volunteer for this? --amk
Andrew Kuchling wrote:
On Fri, May 31, 2002 at 01:43:24PM -0400, Jeremy Hylton wrote:
Good point. Is the core API documented anywhere? I can't tell what is intended to be part of the API and what is accidentally exposed by the implementation.
No reference docs were ever written, AFAIK, so we'll just have to follow Python convention: methods prefixed with an underscore are private, otherwise public.
Right.
The getopt code is probably the most complex argument processing code I've ever seen. I've got no idea if I've preserved backwards compatibility where necessary.
Assuming OptionParser (or whatever Greg names it) gets checked in, distutils.fancy_getopt can get deprecated.
Is there are strong need to replace it ? (Again, this would probably third break code using it to e.g. make setup.py files command line customizable.)
Does it mean we need to support all the optional verbose keyword args even though they will be ignored?
I guess so; maybe there should be a warning when 'verbose' is supplied.
It would be nice if there was a list of packages that do fancy Distutil customization, so that we'd know what to test against. Anyone want to volunteer for this?
Hmm, subclasses will usually pass the verbose argument along, so removing it would cause these to fail. Ignoring it would probably be fine though. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ Meet us at EuroPython 2002: http://www.europython.org/
"AMK" == Andrew Kuchling <akuchlin@mems-exchange.org> writes:
AMK> On Fri, May 31, 2002 at 01:43:24PM -0400, Jeremy Hylton wrote:
Good point. Is the core API documented anywhere? I can't tell what is intended to be part of the API and what is accidentally exposed by the implementation.
AMK> No reference docs were ever written, AFAIK, so we'll just have AMK> to follow Python convention: methods prefixed with an AMK> underscore are private, otherwise public. I'm wondering more about the goal of preserving backwards compatibility of an undocumented interface for extension writers. How many extensions have been written? Do we have any idea how many people use this or would be affected? If it's just a handful of people, I wonder if it would be easier to trying to define a small API that they can rely on and give up on backwards compatibility. not-inteding-to-deliberately-incite-riot-ly y'rs, Jeremy
On Tue, 4 Jun 2002, Jeremy Hylton wrote:
"AMK" == Andrew Kuchling <akuchlin@mems-exchange.org> writes:
AMK> On Fri, May 31, 2002 at 01:43:24PM -0400, Jeremy Hylton wrote:
Good point. Is the core API documented anywhere? I can't tell what is intended to be part of the API and what is accidentally exposed by the implementation.
AMK> No reference docs were ever written, AFAIK, so we'll just have AMK> to follow Python convention: methods prefixed with an AMK> underscore are private, otherwise public.
I'm wondering more about the goal of preserving backwards compatibility of an undocumented interface for extension writers. How many extensions have been written? Do we have any idea how many people use this or would be affected?
If it's just a handful of people, I wonder if it would be easier to trying to define a small API that they can rely on and give up on backwards compatibility.
For your information, the module scipy_distutils from the SciPy (www.scipy.org) package extends distutils quite a bit: it provides definitions for a dozen of Fortran 77/90/95 compilers from different vendors, it supports compilation of Fortran sources and building (automatically generated) extension modules to wrap Fortran and C libraries, etc. To achieve all that many hooks from the standard distutils needed to be extended and I cannot imagine if all that could have be done using just a "small" API (whatever that would be). However, I don't know what kind of changes are you planning and therefore I am not sure if they would have any impact to scipy_distutils. Hopefully it will be positive in the sense that it would work with Python versions starting from 2.1 and ending with the latest. So, some backwards compatibility is appreciated. What is the definition of handful people? Do you count extension developers or the users of these extensions? If the later, one would really need lots of hands when considering the current user base of SciPy alone (to give some idea, currently there are more than 100 subscribers to the scipy users mailing list formed in less than one year). Regards, Pearu
"PP" == Pearu Peterson <pearu@cens.ioc.ee> writes:
PP> For your information, the module scipy_distutils from the SciPy PP> (www.scipy.org) package extends distutils quite a bit: it PP> provides definitions for a dozen of Fortran 77/90/95 compilers PP> from different vendors, it supports compilation of Fortran PP> sources and building (automatically generated) extension modules PP> to wrap Fortran and C libraries, etc. Thanks for the pointer! I had no idea what kind of extensions people had come up with and it's quite educational (for me at least) to see one. PP> to wrap Fortran and C libraries, etc. To achieve all that many PP> hooks from the standard distutils needed to be extended and I PP> cannot imagine if all that could have be done using just a PP> "small" API (whatever that would be). I guess I don't know what a "small" API would be either. My concern is that an extension could rely on accidents of the current implementation without realizing it. Since there's no documented API, there's no way for an extension writer to know what is intentional and what is accidental. PP> However, I don't know what kind of changes are you planning and PP> therefore I am not sure if they would have any impact to PP> scipy_distutils. I had specifically wondered about all the verbose arguments passed to distutils functions and the announce() method on some objects. I just checked in some changes that effectively ignores the verbose argument, and I wondered if it needed to be preserved at all. I see that the scipy_distutils passes the verbose argument to several functions, so removal of it would affect you. PP> What is the definition of handful people? Do you count extension PP> developers or the users of these extensions? If the later, one PP> would really need lots of hands when considering the current PP> user base of SciPy alone (to give some idea, currently there are PP> more than 100 subscribers to the scipy users mailing list formed PP> in less than one year). I think we would have to consider the users of the extensions, too. (But if you're the only one, we can figure something out <wink>.) Jeremy
eremy Hylton wrote:
"PP" == Pearu Peterson <pearu@cens.ioc.ee> writes:
PP> For your information, the module scipy_distutils from the SciPy PP> (www.scipy.org) package extends distutils quite a bit: it PP> provides definitions for a dozen of Fortran 77/90/95 compilers PP> from different vendors, it supports compilation of Fortran PP> sources and building (automatically generated) extension modules PP> to wrap Fortran and C libraries, etc.
Wouldn't it be a good idea to move some of those into the distutils core ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ Meet us at EuroPython 2002: http://www.europython.org/
On Wed, 5 Jun 2002, M.-A. Lemburg wrote:
eremy Hylton wrote:
>"PP" == Pearu Peterson <pearu@cens.ioc.ee> writes:
PP> For your information, the module scipy_distutils from the SciPy PP> (www.scipy.org) package extends distutils quite a bit: it PP> provides definitions for a dozen of Fortran 77/90/95 compilers PP> from different vendors, it supports compilation of Fortran PP> sources and building (automatically generated) extension modules PP> to wrap Fortran and C libraries, etc.
Wouldn't it be a good idea to move some of those into the distutils core ?
Additing Fortran stuff to distutils would mean that somebody must also maintain it. Currently, I think, scipy is most active in using Fortran stuff in Python and therefore ideal also for maintaining these hooks. But on the other hand, Fortran stuff in scipy_distutils is generic in the sense that it can be used in other projects as well (and indeed, scipy_distutils is used separately from scipy already). So, ideal place for Fortran stuff would be in the standard distutils, IMHO. However, there might be some voices against adding Fortran stuff to distutils core. See the thread starting at http://mail.python.org/pipermail/distutils-sig/2001-August/002569.html I would volunteer to work with moving Fortran stuff to distutils provided that it will not be too painful -- due to relatively low interest in Fortran among Python users and developers I am a bit afraid to hit my head against a wall for nothing. Regards, Pearu
Pearu Peterson wrote:
On Wed, 5 Jun 2002, M.-A. Lemburg wrote:
eremy Hylton wrote:
>>"PP" == Pearu Peterson <pearu@cens.ioc.ee> writes: > PP> For your information, the module scipy_distutils from the SciPy PP> (www.scipy.org) package extends distutils quite a bit: it PP> provides definitions for a dozen of Fortran 77/90/95 compilers PP> from different vendors, it supports compilation of Fortran PP> sources and building (automatically generated) extension modules PP> to wrap Fortran and C libraries, etc.
Wouldn't it be a good idea to move some of those into the distutils core ?
Additing Fortran stuff to distutils would mean that somebody must also maintain it. Currently, I think, scipy is most active in using Fortran stuff in Python and therefore ideal also for maintaining these hooks.
But on the other hand, Fortran stuff in scipy_distutils is generic in the sense that it can be used in other projects as well (and indeed, scipy_distutils is used separately from scipy already). So, ideal place for Fortran stuff would be in the standard distutils, IMHO.
How much code are we talking about here ?
However, there might be some voices against adding Fortran stuff to distutils core. See the thread starting at
http://mail.python.org/pipermail/distutils-sig/2001-August/002569.html
Paul may be right in the sense that a) Fortran isn't generally used b) the code is too compiler specific
I would volunteer to work with moving Fortran stuff to distutils provided that it will not be too painful -- due to relatively low interest in Fortran among Python users and developers I am a bit afraid to hit my head against a wall for nothing.
-- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ Meet us at EuroPython 2002: http://www.europython.org/
On Wed, 5 Jun 2002, M.-A. Lemburg wrote:
Pearu Peterson wrote:
But on the other hand, Fortran stuff in scipy_distutils is generic in the sense that it can be used in other projects as well (and indeed, scipy_distutils is used separately from scipy already). So, ideal place for Fortran stuff would be in the standard distutils, IMHO.
How much code are we talking about here ?
About 1400LOC in command/{build_flib,run_f2py}.py that are directly related to Fortran support (Fortran compiler definitions, extension generator, etc) plus some additional hooks to glue it with distutils.
However, there might be some voices against adding Fortran stuff to distutils core. See the thread starting at
http://mail.python.org/pipermail/distutils-sig/2001-August/002569.html
Paul may be right in the sense that a) Fortran isn't generally used b) the code is too compiler specific
Paul may be also wrong in the sense that a) Fortran is generally used in scientific community. b) What code? In scipy_distutils/command/build_flib.py compiler specific code is well factored considering the number of supported compilers and that scipy_distutils is used both in Unix and Windows platforms. c) I have been in the business of Fortran-Python connection for more than three years and I know the releated issues well but I don't share Paul's pessimism. Considering the current development I can tell that this pessimism was not justified. Regards, Pearu
I don't think what I wrote is incorrect; what I said basically was that Fortran support would have to be done by hand and could not piggy-backed off the Python configuration. I also thought the default compiler flags would be satisfactory for a smaller portion of the files to be compiled than is the case with C. I still think both those things. Remember, I wrote what I wrote at a time when it seemed to me that distutils still needed a lot of work on the C side. However, things have changed now in that there is more manpower available, someone has already done a lot of the Fortran work, and distutils itself is farther along. I think we should proceed. If there had been a certain amount of managed manpower available and I was managing the project, I would have seen to it that the documentation for the C side had more priority than Fortran support. I would have wanted C++ support before Fortran too (I haven't followed developments closely -- is that in now?) I keep wondering too what the relationship with SCons should be. Someone wake up Pearu -- he fainted when I agreed with him.
On Wed, 5 Jun 2002, Jeremy Hylton wrote:
"PP" == Pearu Peterson <pearu@cens.ioc.ee> writes:
PP> However, I don't know what kind of changes are you planning and PP> therefore I am not sure if they would have any impact to PP> scipy_distutils.
I had specifically wondered about all the verbose arguments passed to distutils functions and the announce() method on some objects. I just checked in some changes that effectively ignores the verbose argument, and I wondered if it needed to be preserved at all. I see that the scipy_distutils passes the verbose argument to several functions, so removal of it would affect you.
I took a look on your patch and I think as a first step we (in scipy) can remove the usage of verbose arguments from scipy_distutils and start using log facilities so that scipy_distutils would still work with distutils from Python 2.1 and 2.2. And after that, scipy_distutils should be immune to your changes in distutils from Python 2.3. BTW, can Python 2.1,2.2 users later upgrade their distutils or is distutils now inseparable part of the latest Python version? In scipy_distutils we would like to avoid dublicating code with the standard distutils but still support Python versions starting from 2.1.
PP> What is the definition of handful people? Do you count extension PP> developers or the users of these extensions? If the later, one PP> would really need lots of hands when considering the current PP> user base of SciPy alone (to give some idea, currently there are PP> more than 100 subscribers to the scipy users mailing list formed PP> in less than one year).
I think we would have to consider the users of the extensions, too. (But if you're the only one, we can figure something out <wink>.)
(I guess by "...figure something out" you mean the possbile changes to scipy_distutils that I mentioned above. But I don't get this <winking>, I see it a lot in various python lists but still not sure about its meaning, is it good or bad, or something else. <wink, whatever it means> ) Regards, Pearu
participants (5)
-
Andrew Kuchling -
Jeremy Hylton -
M.-A. Lemburg -
Paul F Dubois -
Pearu Peterson