[Python-3000-checkins] r58557 - python/branches/py3k/Parser/tokenizer.c

brett.cannon python-3000-checkins at python.org
Sat Oct 20 05:46:49 CEST 2007


Author: brett.cannon
Date: Sat Oct 20 05:46:49 2007
New Revision: 58557

Modified:
   python/branches/py3k/Parser/tokenizer.c
Log:
Fix PyTokenizer_FindEncoding() for OS X 10.4.  Turns out that seeking to the
beginning of a file through a file pointer is not reflected when reading from a
file descriptor.  Using both fflush() and fpurge() does not solve it.  One must
use lseek() directly on the file descriptor to get the desired effect.

This might suggest that we standardize on either file pointers (FILE) or file
descriptors (int) for all C code used.


Modified: python/branches/py3k/Parser/tokenizer.c
==============================================================================
--- python/branches/py3k/Parser/tokenizer.c	(original)
+++ python/branches/py3k/Parser/tokenizer.c	Sat Oct 20 05:46:49 2007
@@ -1605,8 +1605,11 @@
 /* Get -*- encoding -*- from a Python file
 
    PyTokenizer_FindEncoding returns NULL when it can't find the encoding in
-   the first or second line of the file. In this case the encoding is
-   PyUnicode_GetDefaultEncoding().
+   the first or second line of the file (in which case the encoding 
+   should be assumed to be PyUnicode_GetDefaultEncoding()).
+
+   The char * returned was malloc'ed from PyMem_MALLOC() and thus must be freed
+   when no longer needed.
 */
 char *
 PyTokenizer_FindEncoding(FILE *fp) {
@@ -1614,14 +1617,18 @@
 	char *p_start=NULL, *p_end=NULL, *encoding=NULL;
 
 	if ((tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL)) == NULL) {
-		rewind(fp);
+		/* lseek() usage is on purpose; see note later in code. */
+		lseek(fileno(fp), 0, 0);
 		return NULL;
 	}
 	while(((tok->lineno < 2) && (tok->done == E_OK))) {
 		PyTokenizer_Get(tok, &p_start, &p_end);
 	}
 
-	rewind(fp);
+	/* lseek() must be used instead of fseek()/rewind() as those fail on
+	   OS X 10.4 to properly seek back to the beginning when reading from
+	   the file descriptor instead of the file pointer.  */
+	lseek(fileno(fp), 0, 0);
 
 	if (tok->encoding) {
             encoding = (char *)PyMem_MALLOC(strlen(tok->encoding));


More information about the Python-3000-checkins mailing list