[Tutor] Understanding (Complex) Modules

[Steven D'Aprano]

On Thu, 4 Mar 2010 12:24:35 pm Wayne Watson wrote:
> First a little preamble before my questions.
>
> Most of my work in Python has required modifying a program that uses
> modules that were imported by the original program. I've made some
> use of modules on a command line like math, and have used the idea of
> a qualifier.  On occasion, I've used examples from matplotlib that
> required from matplotlib.image import AxesImage. Further, I've
> created some simple classes, and produced  objects with them. No use
> of inheritance though.  So far so good.  In a few places, it is said
> modules are objects. I'm slightly puzzled by that, but in some way it
> seems reasonable from the standpoint of period notation. So far I
> have not created a module.

Yes you have, you just don't know it.

In simple language, a file with Python-usable code in it is a module. 
There are some technicalities and complexities, but in basic terms, 
that's all it is.

Technically, "module" refers only to the object which exists inside the 
Python environment after you call "import thing". The import statement 
does a whole lot of tricks, but in a nutshell it:

* finds a file containing code called 'thing'
* loads it into memory, executing code as needed
* creates a 'module' object to store the code and data in the file
* and makes it available to your code

Python can create module objects from:

compiled Windows DLLs .dll
compiled Linux object files .so
Python source code .py
Python byte code .pyc and .pyo (and .pyw on Windows)
Zip files containing any of the above

and probably other things as well, but they're the main ones.

So any Python file you create (any .py file) is a module, or at least it 
would be a module if you import it.

Ignoring all the various compiled files (.dll, .so, etc) what happens 
when you run a .py file from the command line (or from IDLE or another 
IDE). E.g. you type something like:

python.exe myscript.py [enter]


Python reads the file myscript.py and executes it, but no module object 
is created. It is possible that a module object *is* created, for 
internal use, then thrown away when the script is finished. But your 
script doesn't see the module object.

However, if you enter the Python interactive environment, and do this:

>>> import myscript  # no .py

Python loads the file myscript.py into a module object, executing any 
code in it, and makes it available as a module.



> Here comes the questions. Recently I began to use matplotlib, scipy,
> and pylab, mostly matplotlib. I've ground out a few useful pieces of
> code, but I'm fairly baffled by the imports required to get at
> various elements, of say, matplotlib (MPL).  Some of the MPL examples
> use of imports make sense, but how the writer pulled in the necessary
> elements is not.  How does one go about understanding the
> capabilities of such modules? 

Time, experience, and a lot of hard work. Welcome to programming!

If the documentation is good, then read the documentation.

If the documentation is lacking, or bad, or even wrong, then read the 
source code, or search the Internet for a tutorial, or buy a book about 
it (the numpy people, for example, sell books about numpy).

Python makes experimentation easy: there are a lot of powerful 
introspection facilities in Python. The interactive interpreter is your 
best friend. You will live with it, eat with it, sleep with it, take it 
on double-dates, and throw yourself on live grenades to protect it. 
Whenever I'm programming, I almost always have three or five 
interactive sessions open for experimentation.

The dir() and help() functions are also good friends. In an interactive 
session:

import math
dir(math)  # prints a list of names in the math module
help(math.sin)
help(math)


Don't feel that you have to understand the entire module before you use 
it. Many (alas, not all) modules have a reasonably gentle learning 
curve: you can start using math.sin without needing to know what 
math.sinh is for.

Google and Wikipedia are also your friends, although not your *best* 
friends. (Don't necessarily believe *everything* you read on the 
Internet.) Don't forget other search engines apart from Google: they're 
good, but not perfect.



> MPL seems to have a lot of lower level 
> components. Some of them are laid out over numerous pages as in the
> form of a, say, English language, description. How does one decipher
> this stuff.  For example, open the module in an editor and start
> looking at the organization? I thinkthe so called MPL guide ins 900
> pages long. Even the numpy guide (reference?), I believe borders on
> 1000 pages. There must be some way to untangle these complex modules,
> I would think. Some of the tutorials seem nothing more than a
> template to follow for a particular problem. So far, looking at the
> plentiful number of examples of MPL, and probably some of the other
> modules mentioned above have not provided a lot of insight.

Big, complex modules tend to have steep learning curves. There's no 
magic path to learning how to use a big complex module any more than 
there is a magic path to learning how to be a brain surgeon, or a car 
mechanic.


>   Is there some relationship between modules and objects that I'm not
> seeing that could be of value?

Modules are themselves objects. Everything in Python is an object: 
strings, ints, floats, lists, tuples, everything.

Modules are compound objects, in that they contain other objects 
accessible by name:

math.sin 

means "look up the name 'sin' in the math module, and return whatever 
you find", which in this case is a function object.

And that's pretty much it.


-- 
Steven D'Aprano