Is there a maximum size to a Python program?

[Steven D'Aprano]

On Mon, 27 Apr 2009 13:01:22 +1000, Carbon Man wrote:

> I have a program that is generated from a generic process. It's job is
> to check to see whether records (replicated from another system) exist
> in a local table, and if it doesn't, to add them. I have 1 of these
> programs for every table in the database. 


Sounds like a needlessly complicated way of doing things. Surely 
replicating data from a database is a solved problem?


> Everything works well until I
> do the postcode table. The generated code is 5MB for a system with no
> current data. 

I don't understand what you mean by "with no current data". Do you mean 
that you generate 5MG of source code when the original postcode table is 
empty? Or 5MB of source code when the *replicated* table is empty? Or 
something else?


> Normally the file would not be this big as only the
> changes are copied over. Python just quits, I have tried stepping
> through the code in the debugger but it doesn't even start.

That depends on how your generated code looks.

Some years ago, somebody posted here complaining that his generated code 
wasn't working. It turned out that he was generating a single function 
looking something vaguely like this:

def function(alist):
    alist[0] = 0
    alist[1] = 0
    alist[2] = 0
    alist[3] = 0
    ...
    alist[29999997] = 0
    alist[29999998] = 0
    alist[29999999] = 0
    return alist

Naturally, Python would need to load all 30 million lines of code into a 
*single* code object, which was too large to load.

After changing the generated code to look something like this:

def function(alist):
    for i in xrange(30000000):
        alist[i] = 0
    return alist

it worked perfectly.

(I am a little fuzzy on the precise details, and googling is not helping, 
so please excuse any discrepancies between my memory of the thread and 
the actual reality.)



> I am thinking that dynamically generating the programs to run might not
> be such a good idea. It would be a shame to drop it because the system
> needs to be generic and it runs from an XML file so the resulting code
> could be pretty complex, and I am new to Python. The program did
> generate a pyc so it was able to compile.
> Thoughts anyone?

Perhaps you could explain what the 5MB of code is supposed to do it, and 
how you generate the code, and show a *small* number of sample lines.




-- 
Steven