[Python-checkins] r52214 - python/branches/release24-maint/Parser/tokenizer.c

[andrew.kuchling]

Author: andrew.kuchling
Date: Fri Oct  6 20:59:10 2006
New Revision: 52214

Modified:
   python/branches/release24-maint/Parser/tokenizer.c
Log:
[Backport r46602 | neal.norwitz]

Patch #1357836:

Prevent an invalid memory read from test_coding in case the done flag is set.
In that case, the loop isn't entered.  I wonder if rather than setting
the done flag in the cases before the loop, if they should just exit early.

This code looks like it should be refactored.

Backport candidate (also the early break above if decoding_fgets fails)




Modified: python/branches/release24-maint/Parser/tokenizer.c
==============================================================================
--- python/branches/release24-maint/Parser/tokenizer.c	(original)
+++ python/branches/release24-maint/Parser/tokenizer.c	Fri Oct  6 20:59:10 2006
@@ -865,6 +865,11 @@
 				if (decoding_fgets(tok->inp,
 					       (int)(tok->end - tok->inp),
 					       tok) == NULL) {
+					/* Break out early on decoding
+					   errors, as tok->buf will be NULL
+					 */
+					if (tok->decoding_erred)
+						return EOF;
 					/* Last line does not end in \n,
 					   fake one */
 					strcpy(tok->inp, "\n");
@@ -872,14 +877,16 @@
 				tok->inp = strchr(tok->inp, '\0');
 				done = tok->inp[-1] == '\n';
 			}
-			tok->cur = tok->buf + cur;
-			/* replace "\r\n" with "\n" */
-			/* For Mac we leave the \r, giving a syntax error */
-			pt = tok->inp - 2;
-			if (pt >= tok->buf && *pt == '\r') {
-				*pt++ = '\n';
-				*pt = '\0';
-				tok->inp = pt;
+ 			if (tok->buf != NULL) {
+				tok->cur = tok->buf + cur;
+				/* replace "\r\n" with "\n" */
+				/* For Mac we leave the \r, giving a syntax error */
+				pt = tok->inp - 2;
+				if (pt >= tok->buf && *pt == '\r') {
+					*pt++ = '\n';
+					*pt = '\0';
+					tok->inp = pt;
+				}
 			}
 		}
 		if (tok->done != E_OK) {