CPython loading modules into memory

[Robert Kern]

On 2009-02-11 15:30, sjbrown wrote:
> Can someone describe the details of how Python loads modules into
> memory?  I assume once the .py file is compiled to .pyc that it is
> mmap'ed in.  But that assumption is very naive.  Maybe it uses an
> anonymous mapping?  Maybe it does other special magic?  This is all
> very alien to me, so if someone could explain it in terms that a
> person who never usually worries about memory could understand, that
> would be much appreciated.

Python will read the .py file and compile into bytecode. This bytecode will be 
written out to a .pyc file as a caching mechanism; if the .pyc exists and has a 
newer timestamp than the .py file, the .py file will not be read or compiled. 
The bytecode will simply be read from the .pyc file.

The .pyc file is not really a map of a memory structure, per se. It is simply a 
disk representation of the bytecode. This bytecode is *executed* by the Python 
VM to populate a module's namespace (which is just a dict) in memory. I believe 
it is then discarded.

> Follow up: is this process different if the modules are loaded from a
> zipfile?

Only in that the .py or .pyc will be extracted from the zipfile instead of being 
read directly from disk.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco