Hi, I wrote patches for the issue #3137 and I would like your opinion about my solution (not directly about the patches): I propose to consider a SIGINT during startup as a fatal error. If you press CTRL+c while Python is starting, Python displays an error and exit with the code 1. Currently, if you press CTRL+c while Python is starting: the SIGINT signal will be *ignored* somewhere between Python initialization and the first Python instruction. First problem is that if site import is interrupted, Python is not completly initialized (if a SIGINT is sent just after the call to initsigs(), initsite() is completly skipped). Example: ------------------ $ python ^C Traceback (most recent call last): File "/home/SHARE/SVN/python-trunk/Lib/site.py", line 62, in <module> import os File "/home/SHARE/SVN/python-trunk/Lib/os.py", line 44, in <module> from posix import * KeyboardInterrupt $ echo $? 1 ------------------ It's not easy to reproduce this issue because Python initialization is really fast (yeah really! or at least to fast for me ;-)). I'm using "valgrind --log- file=/dev/null ./python -v" to simulate a very slow computer :-) -- Victor Stinner http://www.haypocalc.com/
On 08/03/2010 13:12, Victor Stinner wrote:
Hi,
I wrote patches for the issue #3137 and I would like your opinion about my solution (not directly about the patches): I propose to consider a SIGINT during startup as a fatal error. If you press CTRL+c while Python is starting, Python displays an error and exit with the code 1.
That sounds like a good solution to me. +1 Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead of "*". On Mon, Mar 8, 2010 at 9:14 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 08/03/2010 13:12, Victor Stinner wrote:
Hi,
I wrote patches for the issue #3137 and I would like your opinion about my solution (not directly about the patches): I propose to consider a SIGINT during startup as a fatal error. If you press CTRL+c while Python is starting, Python displays an error and exit with the code 1.
That sounds like a good solution to me. +1
Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido@python.org> wrote:
Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead of "*".
There is at least one that prints "import 'site' failed" and continues to run your program. Cheers, fijal
On Mon, Mar 8, 2010 at 10:21 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido@python.org> wrote:
Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead of "*".
There is at least one that prints "import 'site' failed" and continues to run your program.
Yeah, it shouldn't do that. The ^C should fall through. -- --Guido van Rossum (python.org/~guido)
Hi, Some news about my patch. Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a écrit :
On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido@python.org> wrote:
Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead of "*".
There is at least one that prints "import 'site' failed" and continues to run your program.
I commited to patch and broke all buildbots \o/ There were errors in test_sysconfig and test_subprocess related to issues #7774 and #7880 (problems with sys.executable and modified sys.argv[0]). I consider that it's a good point to detect the errors earlier, but it breaks code based on the errors :-) I will no backport the fix to 2.6 (or at least, not the fix about importing the site module) Should I backport the fix to 3.1? (The backport to py3k is not done yet) -- Victor Stinner http://www.haypocalc.com/
On Wed, Mar 10, 2010 at 17:21, Victor Stinner <victor.stinner@haypocalc.com>wrote:
Hi,
Some news about my patch.
Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a écrit :
On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido@python.org> wrote:
Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead of "*".
There is at least one that prints "import 'site' failed" and continues to run your program.
I commited to patch and broke all buildbots \o/ There were errors in test_sysconfig and test_subprocess related to issues #7774 and #7880 (problems with sys.executable and modified sys.argv[0]).
I consider that it's a good point to detect the errors earlier, but it breaks code based on the errors :-) I will no backport the fix to 2.6 (or at least, not the fix about importing the site module)
Should I backport the fix to 3.1? (The backport to py3k is not done yet)
Once you are happy with the stability then please port it to 3.1. -Brett
-- Victor Stinner http://www.haypocalc.com/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
2010/3/11 Brett Cannon <brett@python.org>:
On Wed, Mar 10, 2010 at 17:21, Victor Stinner <victor.stinner@haypocalc.com> wrote:
Hi,
Some news about my patch.
Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a écrit :
On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido@python.org> wrote:
Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead of "*".
There is at least one that prints "import 'site' failed" and continues to run your program.
I commited to patch and broke all buildbots \o/ There were errors in test_sysconfig and test_subprocess related to issues #7774 and #7880 (problems with sys.executable and modified sys.argv[0]).
I consider that it's a good point to detect the errors earlier, but it breaks code based on the errors :-) I will no backport the fix to 2.6 (or at least, not the fix about importing the site module)
Should I backport the fix to 3.1? (The backport to py3k is not done yet)
Once you are happy with the stability then please port it to 3.1. -Brett
But after 3.1.2 please! -- Regards, Benjamin
On Thu, Mar 11, 2010 at 13:33, Benjamin Peterson <benjamin@python.org>wrote:
2010/3/11 Brett Cannon <brett@python.org>:
On Wed, Mar 10, 2010 at 17:21, Victor Stinner <
victor.stinner@haypocalc.com>
wrote:
Hi,
Some news about my patch.
Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a écrit :
On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido@python.org> wrote:
Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead
of
"*".
There is at least one that prints "import 'site' failed" and continues to run your program.
I commited to patch and broke all buildbots \o/ There were errors in test_sysconfig and test_subprocess related to issues #7774 and #7880 (problems with sys.executable and modified sys.argv[0]).
I consider that it's a good point to detect the errors earlier, but it breaks code based on the errors :-) I will no backport the fix to 2.6 (or at least, not the fix about importing the site module)
Should I backport the fix to 3.1? (The backport to py3k is not done yet)
Once you are happy with the stability then please port it to 3.1. -Brett
But after 3.1.2 please!
Sorry, I meant py3k (aka 3.2) and not 3.1. Since this is a behavior change I don't think it should be in 2.6/3.1. -Brett
-- Regards, Benjamin
Le jeudi 11 mars 2010 21:58:11, Brett Cannon a écrit :
(...) Should I backport the fix to 3.1? (The backport to py3k is not done yet)
Once you are happy with the stability then please port it to 3.1.
My commit broke test_sysconfig, test_platform and test_subprocess. Some commits later, #7880 is fixed and part of #7774 is also fixed. So yes, it will be possible to port the fix to py3k. -- Victor Stinner http://www.haypocalc.com/
Hi, Le lundi 08 mars 2010 18:47:18, Guido van Rossum a écrit :
Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead of "*".
Most of my patches (for SIGINT) are exactly doing that: check the exception type instead of "if (result == NULL) PyErr_Clear();" (try/except: pass). About initsite(): I consider any error as fatal. If initsite() failed, something goes wrong. I think that my patches are enough to catch SIGINT during the Python initialization. I used different tricks to test Python: add kill(getpid(), SIGINT), use "signal SIGINT" in gdb, add PyErr_SetNone(PyExc_KeyboardInterrupt), ... valgrind is a very good tool to slow down any process :-D -- Victor Stinner http://www.haypocalc.com/
On Mon, Mar 8, 2010 at 12:50 PM, Victor Stinner <victor.stinner@haypocalc.com> wrote:
Le lundi 08 mars 2010 18:47:18, Guido van Rossum a écrit :
Actually it sounds like there's some overly general except clause somewhere that should be adjusted to catch just "Exception" instead of "*".
Most of my patches (for SIGINT) are exactly doing that: check the exception type instead of "if (result == NULL) PyErr_Clear();" (try/except: pass). About initsite(): I consider any error as fatal. If initsite() failed, something goes wrong.
I think that my patches are enough to catch SIGINT during the Python initialization. I used different tricks to test Python: add kill(getpid(), SIGINT), use "signal SIGINT" in gdb, add PyErr_SetNone(PyExc_KeyboardInterrupt), ... valgrind is a very good tool to slow down any process :-D
I see. I thought for a moment that by "fatal" you meant something like calling Py_FatalError(). I am trying to remember why I made site.py failures non-fatal in the first place. I don't have any specific recollection but it must've been either from before the separation between site.py (part of the stdlib) and sitecustomize.py (site-specific) or out of a worry that if some external cause broke site.py (which does a lot of I/O) it would be a fatal breakdown of all Python execution. I am fine with making an exception for KeyboardInterrupt (and other exceptions not deriving from Exception but only from BaseException) but I don't think I want to go as far as making all errors coming out of site.py fatal. (At least not without more understanding of the use case for making these non-fatal in the first place.) OTOH I think the try/except in site.py around the execution of sitecustomize.py might be changed so that it prints a full traceback whenever it raises an exception other than ImportError or, again, exceptions inheriting from BaseException but not from Exception. IOW I think that exceptions coming out of sitecustomize.py should continued to be treated non-fatally, apart from ^C and friends, but I think that exceptions coming out of site.py *might* be considered more fatally once the change to exceptions coming out of sitecustomize.py is made. -- --Guido van Rossum (python.org/~guido)
On Mar 8, 2010, at 4:06 PM, Guido van Rossum wrote:
I am trying to remember why I made site.py failures non-fatal in the first place. I don't have any specific recollection but it must've been either from before the separation between site.py (part of the stdlib) and sitecustomize.py (site-specific) or out of a worry that if some external cause broke site.py (which does a lot of I/O) it would be a fatal breakdown of all Python execution.
The thing that occurs to me is that one might want to write an administrative tool in Python to manipulate site.py, or even just some data that something in site.py would load. If exceptions from site.py were fatal, then bugs in such a tool would be completely unrecoverable; in trying to run it to un-do the buggy operation, it would crash immediately. On the other hand, such a tool should *really* be invoked with the -S option anyway, so... maybe not that pressing of a concern.
On Mon, Mar 8, 2010 at 1:18 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Mar 8, 2010, at 4:06 PM, Guido van Rossum wrote:
I am trying to remember why I made site.py failures non-fatal in the first place. I don't have any specific recollection but it must've been either from before the separation between site.py (part of the stdlib) and sitecustomize.py (site-specific) or out of a worry that if some external cause broke site.py (which does a lot of I/O) it would be a fatal breakdown of all Python execution.
The thing that occurs to me is that one might want to write an administrative tool in Python to manipulate site.py, or even just some data that something in site.py would load.
This would be a more likely theory if we didn't have sitecustomize.py to be manipulated.
If exceptions from site.py were fatal, then bugs in such a tool would be completely unrecoverable; in trying to run it to un-do the buggy operation, it would crash immediately. On the other hand, such a tool should *really* be invoked with the -S option anyway, so... maybe not that pressing of a concern.
Right. I'm leaning towards the theory that treating site.py failures as non-fatal is older than the separation between site.py and sitecustomize.py. -- --Guido van Rossum (python.org/~guido)
Le lundi 08 mars 2010 22:18:34, Glyph Lefkowitz a écrit :
On Mar 8, 2010, at 4:06 PM, Guido van Rossum wrote:
I am trying to remember why I made site.py failures non-fatal in the first place. I don't have any specific recollection but it must've been either from before the separation between site.py (part of the stdlib) and sitecustomize.py (site-specific) or out of a worry that if some external cause broke site.py (which does a lot of I/O) it would be a fatal breakdown of all Python execution.
The thing that occurs to me is that one might want to write an administrative tool in Python to manipulate site.py, or even just some data that something in site.py would load. If exceptions from site.py were fatal, then bugs in such a tool would be completely unrecoverable; in trying to run it to un-do the buggy operation, it would crash immediately.
I don't think that modifying the site.py file is a good idea. Why not editing/generating the sitecustomize or usercustomize module? Anyway, if you write such tool: use a different language or use a failsafe Python interpreter dedicated to this tool ;-) -- Victor Stinner http://www.haypocalc.com/
Le lundi 08 mars 2010 22:06:36, vous avez écrit :
OTOH I think the try/except in site.py around the execution of sitecustomize.py might be changed so that it prints a full traceback whenever it raises an exception other than ImportError or, again, exceptions inheriting from BaseException but not from Exception. IOW I think that exceptions coming out of sitecustomize.py should continued to be treated non-fatally, apart from ^C and friends, but I think that exceptions coming out of site.py *might* be considered more fatally once the change to exceptions coming out of sitecustomize.py is made.
Do you mean something like the following code? ---------------------- def execsitecustomize(): """Run custom site specific code, if available.""" try: import sitecustomize except ImportError: pass except Exception: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: print >>sys.stderr, "'import sitecustomize' failed; use -v for traceback" def execusercustomize(): """Run custom user specific code, if available.""" try: import usercustomize except ImportError: pass except Exception: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: print >>sys.stderr, "'import usercustomize' failed; use -v for traceback" ---------------------- Using these functions, even if sitecustomize fails, usercustomize is imported. Can it be a problem? Does usercustomize requires that sitecustomize has been initialized correctly? (The code can be changed to only import usercustomize if sitecutomize succeed or failed with an ImportError, but not if it fails with a different error) -- Victor Stinner http://www.haypocalc.com/
On Mon, Mar 8, 2010 at 2:01 PM, Victor Stinner <victor.stinner@haypocalc.com> wrote:
Le lundi 08 mars 2010 22:06:36, vous avez écrit :
OTOH I think the try/except in site.py around the execution of sitecustomize.py might be changed so that it prints a full traceback whenever it raises an exception other than ImportError or, again, exceptions inheriting from BaseException but not from Exception. IOW I think that exceptions coming out of sitecustomize.py should continued to be treated non-fatally, apart from ^C and friends, but I think that exceptions coming out of site.py *might* be considered more fatally once the change to exceptions coming out of sitecustomize.py is made.
Do you mean something like the following code? ---------------------- def execsitecustomize(): """Run custom site specific code, if available.""" try: import sitecustomize except ImportError: pass except Exception: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: print >>sys.stderr, "'import sitecustomize' failed; use -v for traceback"
def execusercustomize(): """Run custom user specific code, if available.""" try: import usercustomize except ImportError: pass except Exception: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: print >>sys.stderr, "'import usercustomize' failed; use -v for traceback" ----------------------
Yes, roughly.
Using these functions, even if sitecustomize fails, usercustomize is imported.
Until 5 minutes ago I didn't even know we had usercustomize. :-)
Can it be a problem? Does usercustomize requires that sitecustomize has been initialized correctly?
I don't know, but if it is, execsitecustomize() could return a flag to be checked by the code that calls execusercustomize().
(The code can be changed to only import usercustomize if sitecutomize succeed or failed with an ImportError, but not if it fails with a different error)
Right. It doesn't strike me as a big deal either way. -- --Guido van Rossum (python.org/~guido)
participants (7)
-
Benjamin Peterson -
Brett Cannon -
Glyph Lefkowitz -
Guido van Rossum -
Maciej Fijalkowski -
Michael Foord -
Victor Stinner