[BangPypers] Python packages and modules (was: Favorite tips/techniques)

[Dhananjay Nene]

On Fri, Sep 13, 2013 at 6:50 PM, Dhananjay Nene
<dhananjay.nene at gmail.com> wrote:
> On Fri, Sep 13, 2013 at 3:58 PM, Saager Mhatre <saager.mhatre at gmail.com> wrote:
>> On Fri, Sep 13, 2013 at 1:10 PM, Saju M <sajuptpm at gmail.com> wrote:
>>
>>> Saager,
>>>
>>> As per python module-import semantics, sub-modules don't end up as names
>>> in the
>>> package-module[*] unless explicitly added.
>>>
>>> I didn't get it, what you mean by package-module[*] ?.
>>>
>>
>> Ah, there it is... the sound of your head exploding! :P
>>
>> Buckle up, this is going to be a fast, but rough ride!
>>
>> Basically...
>> ----
>> * A module is a dict-like object that binds names to values.
>> * A package is a namespace that can contain (only) other modules (which
>> could, in turn, be package-modules themselves).
>> * A package-module[1] would be a module that also serves as a package.
>>
>> I guess it'd be easier to explain with with an example.
>>
>> Semantically...
>> ---
>> * Lets stick to json.tool. In this case, json is a module and tool is a
>> sub-module of the json module.
>> * They are both modules in that each can contain bindings to names =>
>> json.dump and tool.main.
>> * But, json is also a package in that it contains the tool module =>
>> json.tool
>> * The sub-module relationship is mostly evident from the fact that the tool
>> module is referenced by prefixing 'json.' to it => import json.tool;
>> * Or providing 'json' as the package to look for the module => from json
>> import tool
>>
>> Physically...
>> ---
>> * Any .py file can be loaded as a module.
>> * Any directory with an __init__.py file can be treated as a package.
>> * The __init__.py file itself serves as the package-module, i.e., the
>> module with same name as the package
>> * Any .py files inside the directory (except __init__.py, of course) can be
>> loaded as sub-modules of the above package.
>> * Any sub-directories inside the directory (containing __init__.py, of
>> course) can be loaded as sub-packages of the above package.
>> * Turtles all the way...
>>
>> Funda-mentally...
>> ---
>> * The confusion basically stems from the fact that Python chose to conflate
>> physical storage and namespacing with just enough overlap to be
>> inconsistent.
>> * They are conflated in that package/module naming and their lookup
>> (finding the code for a module) is tied to the physical storage hierarchy.
>> * They are inconsistent that module loading is transitive only upwards in
>> the hierarchy, i.e., when you load a module, all packages above it in the
>> hierarchy are automatically loaded.[2]
>> * However, sub-modules are not loaded even though the physical hierarchy
>> evidences it.
>> * The conflation extends further as we look as modules as namespaces,
>> because sub-modules do not end up as names in package-modules until they
>> are loaded; see below
>> Python 2.7.4 (default, Apr 19 2013, 18:28:01)
>> [GCC 4.7.3] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> *>>> json*
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> NameError: name 'json' is not defined
>> *>>> import json*
>> *>>> json*
>> <module 'json' from '/usr/lib/python2.7/json/__init__.py'>
>> *>>> json.tool*
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> AttributeError: 'module' object has no attribute 'tool'
>> *>>> from json import tool*
>> *>>> json*
>> <module 'json' from '/usr/lib/python2.7/json/__init__.py'>
>> *>>> tool*
>> <module 'json.tool' from '/usr/lib/python2.7/json/tool.py'>
>> *>>> json.tool*
>> <module 'json.tool' from '/usr/lib/python2.7/json/tool.py'>
>> *>>> *
>>
>> Finally...
>> ---
>> On a closing note, it goes without saying that these packages are not to be
>> confused with packages as published on package indexes such as
>> https://pypi.python.org.[3]
>
> Nice. I reached out for the "Favourite" button but realised
> emails/mailing lists don't have it.
>
> I do think it can be a little easier to understand by doing
> s/package-module/namespace/g. Having done that I realised a few things
> needed to be reworded, and then noticed a few minor corrections were
> required.
>
> Here's the result (Have attempted to only make minor structural
> corrections without adjusting any of the opinions or rationale
> expressed herein)
> ---------------------------------------------------------------------------------------------------------
> Ah, there it is... the sound of your head exploding! :P
>
> Buckle up, this is going to be a fast, but rough ride!
>
> Basically...
> ----
> * A namespace is a dict-like object that binds names to values.
> * A namespace is an abstract notion, which is implemented as packages or modules
>   Thus a package is-a namespace and a module is-a namespace
> * The package namespace includes the modules in it and the stuff in __init__.py
>
> I guess it'd be easier to explain with with an example.
>
> Semantically...
> ---
> * Lets stick to json.tool. In this case, json is a package and tool is
> a module
> * They are both namespaces in that each can contain bindings to names
> => json.dump and tool.main.
> * json is a package namespace in that it contains the tool module => json.tool
> * The sub-module relationship is mostly evident from the fact that the
> tool module is referenced by prefixing 'json.' to it => import
> json.tool;
> * Or providing 'json' as the package to look for the module => from
> json import tool
>
> Physically...
> ---
> * Any .py file can be loaded as a module.
> * Any directory with an __init__.py file can be treated as a package.
> * The __init__.py file itself serves as part of the package namespace,
> i.e., ie. it has the same name as the package
> * Any .py files inside the directory (except __init__.py, of course)
> can be loaded as sub-modules of the above package.
> * Any sub-directories inside the directory (containing __init__.py, of
> course) can be loaded as sub-packages of the above package.
> * Turtles all the way...
>
> Funda-mentally...
> ---
> * The confusion basically stems from the fact that Python chose to
> conflate physical storage and namespacing with just enough overlap to
> be inconsistent.
> * They are conflated in that package/module naming and their lookup
> (finding the code for a module) is tied to the physical storage
> hierarchy.
> * They are inconsistent that module loading is transitive only upwards
> in the hierarchy, i.e., when you load a module, all packages above it
> in the hierarchy are automatically loaded.[2]
> * However, sub-modules are not loaded even though the physical
> hierarchy evidences it.
> * The conflation extends further as we look as packages as namespaces,
> because sub-modules do not end up as names in the package namespace
> until they are loaded; see below
> Python 2.7.4 (default, Apr 19 2013, 18:28:01)
> [GCC 4.7.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> *>>> json*
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NameError: name 'json' is not defined
> *>>> import json*
> *>>> json*
> <module 'json' from '/usr/lib/python2.7/json/__init__.py'>
> *>>> json.tool*
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> AttributeError: 'module' object has no attribute 'tool'
> *>>> from json import tool*
> *>>> json*
> <module 'json' from '/usr/lib/python2.7/json/__init__.py'>
> *>>> tool*

Apologies : that was an incomplete paste .. here's a continuation
-----------------------------------
>>> tool*
<module 'json.tool' from '/usr/lib/python2.7/json/tool.py'>
*>>> json.tool*
<module 'json.tool' from '/usr/lib/python2.7/json/tool.py'>
*>>> *

Finally...
---
On a closing note, it goes without saying that these packages are not
to be confused with packages as published on package indexes such as
https://pypi.python.org.[3]

>>
>> (steps behind transparent blast shield to calmly enjoy the sight of more
>> exploding heads)
>> - d
>>
>> [1] I don't believe it would be entirely in the best interest of either of
>> our healths to use this term outside of this thread!
>> [2] IIRC, this was not true for python <2.5 (I think); I recall hitting
>> some really weird import errors when running newer code on a really old
>> interpreter when it suddenly couldn't reference the packages that a loaded
>> module belonged to until they were explicitly loaded.
>> [3] To those in the know, I would be tremendously obliged if you could tell
>> me what brand of blow they were using when they came up with this
>> byzantine, labyrinthine nomenclature and related implementation. I bet it
>> was, as they say, "like, totally radical dude!"
>> _______________________________________________
>> BangPypers mailing list
>> BangPypers at python.org
>> https://mail.python.org/mailman/listinfo/bangpypers
>
>
>
> --
> ----------------------------------------------------------------------------------------------------------------------------------
> http://blog.dhananjaynene.com twitter: @dnene google plus:
> http://gplus.to/dhananjaynene



-- 
----------------------------------------------------------------------------------------------------------------------------------
http://blog.dhananjaynene.com twitter: @dnene google plus:
http://gplus.to/dhananjaynene