[Python-Dev] very long python files

[The Intermission Elf]

Hey Folks,

I just got this message from a developer inside of ILM today.  Before
I start digging, can anyone tell me if this has been dealt with in any 
version of python more recent than 1.5.2?  (and where I might look for 
this info myself- I'm very unfamiliar with sourceforge and its bug
tracker)...

thanks!



Garrick Meeker writes:
| ******** is starting to produce very long project files (over 32K lines)
| which it currently can't read.  (Because curves are imbedded in the
| files, it's not difficult to hit that limit.)
| 
| Here's what I've found so far (in both python 1.4 and 1.5):
| 
| ******** read the file by calling PyRun_SimpleFile.  This byte compiles
| the entire file into memory and then runs that (which is probably not
| ideal).
| 
| The file is tokenized into nodes that have a 'short' for the line
| number, so this wraps around to negative numbers.
| 
| The line number is entered into the byte code with a call to
| 'com_addint', which calls:
| 
| 	com_addbyte(c, x & 0xff);
| 	com_addbyte(c, x >> 8); /* XXX x should be positive */
| 
| (notice that 'x >> 8' for a negative number will be a negative number).
| 
| com_addbyte has a sanity check:
| 
| 	if (byte < 0 || byte > 255) {
| 		com_error(c, PyExc_SystemError, "com_addbyte: byte out of range");
| 
| This is what prevents ******** from loading long files.  The options I
| see are:
| 
| Fix our build of python to fix this problem.  We can't change the line
| number to 'int' because we'd be changing the byte code, but we could
| stop the number from wrapping around.
| 
| Manually read each line from the file and feed it to the interpreter as
| if we were interactive.  We'd lose the line number information if
| there's an error in the file (but we can't save numbers over 32K
| anyway).  I think it would also be friendlier on memory because it
| wouldn't byte compile the entire file first.  There is a call to pass
| strings in this manner, but I don't know if it will actually work this
| way.
| 
| Tommy, have you heard of this before?  I know I still missing part of
| the story because I can create a simple file with 40K lines of:
| 
| a = 0
| 
| and python accepts it without error.