[BangPypers] Python packages and modules (was: Favorite tips/techniques)

[Saager Mhatre]

On Fri, Sep 13, 2013 at 1:10 PM, Saju M <sajuptpm at gmail.com> wrote:

> Saager,
>
> As per python module-import semantics, sub-modules don't end up as names
> in the
> package-module[*] unless explicitly added.
>
> I didn't get it, what you mean by package-module[*] ?.
>

Ah, there it is... the sound of your head exploding! :P

Buckle up, this is going to be a fast, but rough ride!

Basically...
----
* A module is a dict-like object that binds names to values.
* A package is a namespace that can contain (only) other modules (which
could, in turn, be package-modules themselves).
* A package-module[1] would be a module that also serves as a package.

I guess it'd be easier to explain with with an example.

Semantically...
---
* Lets stick to json.tool. In this case, json is a module and tool is a
sub-module of the json module.
* They are both modules in that each can contain bindings to names =>
json.dump and tool.main.
* But, json is also a package in that it contains the tool module =>
json.tool
* The sub-module relationship is mostly evident from the fact that the tool
module is referenced by prefixing 'json.' to it => import json.tool;
* Or providing 'json' as the package to look for the module => from json
import tool

Physically...
---
* Any .py file can be loaded as a module.
* Any directory with an __init__.py file can be treated as a package.
* The __init__.py file itself serves as the package-module, i.e., the
module with same name as the package
* Any .py files inside the directory (except __init__.py, of course) can be
loaded as sub-modules of the above package.
* Any sub-directories inside the directory (containing __init__.py, of
course) can be loaded as sub-packages of the above package.
* Turtles all the way...

Funda-mentally...
---
* The confusion basically stems from the fact that Python chose to conflate
physical storage and namespacing with just enough overlap to be
inconsistent.
* They are conflated in that package/module naming and their lookup
(finding the code for a module) is tied to the physical storage hierarchy.
* They are inconsistent that module loading is transitive only upwards in
the hierarchy, i.e., when you load a module, all packages above it in the
hierarchy are automatically loaded.[2]
* However, sub-modules are not loaded even though the physical hierarchy
evidences it.
* The conflation extends further as we look as modules as namespaces,
because sub-modules do not end up as names in package-modules until they
are loaded; see below
Python 2.7.4 (default, Apr 19 2013, 18:28:01)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
*>>> json*
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'json' is not defined
*>>> import json*
*>>> json*
<module 'json' from '/usr/lib/python2.7/json/__init__.py'>
*>>> json.tool*
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'tool'
*>>> from json import tool*
*>>> json*
<module 'json' from '/usr/lib/python2.7/json/__init__.py'>
*>>> tool*
<module 'json.tool' from '/usr/lib/python2.7/json/tool.py'>
*>>> json.tool*
<module 'json.tool' from '/usr/lib/python2.7/json/tool.py'>
*>>> *

Finally...
---
On a closing note, it goes without saying that these packages are not to be
confused with packages as published on package indexes such as
https://pypi.python.org.[3]

(steps behind transparent blast shield to calmly enjoy the sight of more
exploding heads)
- d

[1] I don't believe it would be entirely in the best interest of either of
our healths to use this term outside of this thread!
[2] IIRC, this was not true for python <2.5 (I think); I recall hitting
some really weird import errors when running newer code on a really old
interpreter when it suddenly couldn't reference the packages that a loaded
module belonged to until they were explicitly loaded.
[3] To those in the know, I would be tremendously obliged if you could tell
me what brand of blow they were using when they came up with this
byzantine, labyrinthine nomenclature and related implementation. I bet it
was, as they say, "like, totally radical dude!"