From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Sat Nov 10 08:30:25 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Sat, 10 Nov 2001 14:00:25 +0530
Subject: [Import-sig] Proposal for a modified import mechanism.
Message-ID: <15340.58785.94412.122093@monster.linux.in>

Hi,

Sorry about the cross posting.

Over the last couple of weeks I described a problem and inconsistency
with the way Python imports modules.  For more details look at this
thread:

  http://mail.python.org/pipermail/python-list/2001-November/070719.html

In short, currently, import allows one to use non-absolute module
names for modules that are in the current directory.  If a module is
not in the same directory, import then looks for modules in sys.path.
Consequently, dealing with packages that are re-nested is a pain.
Complex package structure also causes problems.  I'd like to note that
I was also not the only person who suffered from this issue -- four
others on mentioned similar problems and some asked me to let them
know if I found a solution.

Subsequently, I proposed another approach that first looks in the
local directory and then walks up the current package tree looking for
modules before looking at sys.path.  I also modified knee.py to obtain
a working solution.  More information is here:

  http://mail.python.org/pipermail/python-list/2001-November/071212.html

the threading is messed up and starts here:

  http://mail.python.org/pipermail/python-list/2001-November/071218.html

You can find the new module and a simple test package here:

  http://av.stanford.edu/~prabhu/download/

There is also a slightly enhanced version of knee.py included that
supports caching module lookup failures suggested by Rainer and Gordon
in:

  http://mail.python.org/pipermail/python-list/2001-November/071218.html

it also fixes a bug where the parent package is an extension module.

I therefore have a working import style that seems to handle importing
packages in a more natural(?) and consistent(?) manner.  I've also
tested it out with a large package like scipy (http://www.scipy.org)
with no trouble or significant performance problems:

  http://mail.python.org/pipermail/python-list/2001-November/071325.html


I'd like to ask the Python developers if they'd consider

    (a) changing the way the current import works to do what I
    proposed, or, 

    (b) add a new keyword like 'rimport' (or something) that does this
    recursive search through parent packages and loads modules.  This
    was actually suggested by Gordon McMillan.  Gordon actually
    suggested something stronger -- import only supports absolute
    names, rimport is relative import and rrimport is a recursive
    relative import.  But this would break the current import since
    import currently aupports some relative lookup.  So maybe import
    and rimport is a workable solution? 

    (c) patch the existing knee.py with my fixes.  Note: these fixes
    have nothing to do with the recirsive module lookup stuff --
    knee.py is merely an improved version of the older one.


Thanks for listening patiently and sorry again for all the cross
posting.

prabhu


From frederic.giacometti@arakne.com  Sat Nov 10 18:13:47 2001
From: frederic.giacometti@arakne.com (Frederic Giacometti)
Date: Sat, 10 Nov 2001 10:13:47 -0800
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
References: <E162bVD-0001nY-00@mail.python.org>
Message-ID: <3BED6E5B.8E142057@arakne.com>


>
> I'd like to ask the Python developers if they'd consider
>
>     (a) changing the way the current import works to do what I
>     proposed, or,
>
>     (b) add a new keyword like 'rimport' (or something) that does this
>     recursive search through parent packages and loads modules.  This
>     was actually suggested by Gordon McMillan.  Gordon actually
>     suggested something stronger -- import only supports absolute
>     names, rimport is relative import and rrimport is a recursive
>     relative import.  But this would break the current import since
>     import currently aupports some relative lookup.  So maybe import
>     and rimport is a workable solution?

I'd rather introduce a __parent__ module attribute (in addition to the
existing __name__) so that, for instance, the following would do your job:

from __parent__.__parent__.toto import something

In its spirit, this is similar to the '..' of the file systems.

For top-level modules, __parent__ would be None.

I'm personnally against anything that enlarges the search path uselessly;
because the obvious reason of increased name space collision, increased
run-time overhead etc...

Frederic Giacometti






From ej@ee.duke.edu  Sat Nov 10 19:03:02 2001
From: ej@ee.duke.edu (eric)
Date: Sat, 10 Nov 2001 14:03:02 -0500
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
References: <E162bVD-0001nY-00@mail.python.org> <3BED6E5B.8E142057@arakne.com>
Message-ID: <033301c16a1a$59c1c690$c300a8c0@ericlaptop>

I have to agree with Prabhu on this one.  The current behavior of import,
while fine for standard modules and even simple packages with a single
level, is sub-optimal for packages that contain sub-packages.  The proposed
behavior solves the problem.

Handling the packaging issues in SciPy was difficult, and even resulted in a
(not always popular) decision to build and overwrite the Numeric package on
machines that install SciPy.  Prabhu's import doesn't resolve all the issues
(I think packages may just be difficult...), but it would have solved this
one.  The proposed import allows us to put our own version of Numeric in the
top SciPy directory.  Then all SciPy sub-packages would grab this one
instead of an existing site-packages/Numeric.  That makes SciPy
self-contained and allows people to try it out without worrying that it
might break their current installation.  There are other solutions to this
problem, but Prabhu's fix is by far the easiest and most robust.

Prabhu's import also has some other nice benefits.  Some of the sub-packages
in SciPy are useful outside of SciPy.  Also sometimes it is easier to
develop a packages outside of the SciPy framework.  It would be nice to be
able to develop a module or package 'foo' outside of SciPy and then move it
into SciPy at a later date.  However, every SciPy sub-package that referred
to foo prior to its inclusion in SciPy now has to be updated from 'import
foo' to 'import scipy.foo'.  These kind of issues make it very painful and
time consuming to rearrange package structures or move modules and
sub-packages in and out of the package.  Simplifying this will improves
package development.

> I'm personnally against anything that enlarges the search path uselessly;

Hopefully I've explained why it is useful for complex packages.

> because the obvious reason of increased name space collision, increased
> run-time overhead etc...

I'm missing something here because I don't understand why this increases
name space collision.  If the objection is to the fact that SciPy can have a
version of Numeric in it that masks a Numeric installed in site-packages, I
guess I consider this a feature, not a bug.  Afterall, this is already the
behavior for single level packages, extending it to multi-level packages
seems natural.  If this isn't your objection, please explain.

The current runtime overhead isn't so bad.  Prabhu sent me a few numbers on
the SciPy import (which contains maybe 10-15 nested packages).  I attached
them below -- the overhead is less than 10%.  It should be negligible for
standard modules as only packages are really affected (right Prabhu?).

$ python
>>> import time
>>> s = time.time (); import scipy; print time.time()-s
1.37971198559
>>>
$ python
>>> import my_import
>>> import time
>>> s = time.time (); import scipy; print time.time()-s
1.48667407036

There may be technical issues under the covers that make this hairier than
it appears, but, from the standpoint of someone working on a large
multi-level package, it looks like a good idea.

see ya,
eric

----- Original Message -----
From: "Frederic Giacometti" <frederic.giacometti@arakne.com>
To: <import-sig@python.org>
Cc: <ej@ee.duke.edu>; <prabhu@cyberwaveindia.com>; <python-list@python.org>
Sent: Saturday, November 10, 2001 1:13 PM
Subject: Re: Proposal for a modified import mechanism.


>
>
> >
> > I'd like to ask the Python developers if they'd consider
> >
> >     (a) changing the way the current import works to do what I
> >     proposed, or,
> >
> >     (b) add a new keyword like 'rimport' (or something) that does this
> >     recursive search through parent packages and loads modules.  This
> >     was actually suggested by Gordon McMillan.  Gordon actually
> >     suggested something stronger -- import only supports absolute
> >     names, rimport is relative import and rrimport is a recursive
> >     relative import.  But this would break the current import since
> >     import currently aupports some relative lookup.  So maybe import
> >     and rimport is a workable solution?
>
> I'd rather introduce a __parent__ module attribute (in addition to the
> existing __name__) so that, for instance, the following would do your job:
>
> from __parent__.__parent__.toto import something
>
> In its spirit, this is similar to the '..' of the file systems.
>
> For top-level modules, __parent__ would be None.
>
> I'm personnally against anything that enlarges the search path uselessly;
> because the obvious reason of increased name space collision, increased
> run-time overhead etc...
>
> Frederic Giacometti
>
>
>




From frederic.giacometti@arakne.com  Sat Nov 10 21:43:54 2001
From: frederic.giacometti@arakne.com (Frederic Giacometti)
Date: Sat, 10 Nov 2001 13:43:54 -0800
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
References: <E162bVD-0001nY-00@mail.python.org> <3BED6E5B.8E142057@arakne.com> <033301c16a1a$59c1c690$c300a8c0@ericlaptop>
Message-ID: <3BED9F9A.30F77DB3@arakne.com>


eric wrote:

> I have to agree with Prabhu on this one.  The current behavior of import,
> while fine for standard modules and even simple packages with a single
> level, is sub-optimal for packages that contain sub-packages.  The proposed
> behavior solves the problem.
>
> Handling the packaging issues in SciPy was difficult, and even resulted in a
> (not always popular) decision to build and overwrite the Numeric package on
> machines that install SciPy.  Prabhu's import doesn't resolve all the issues
> (I think packages may just be difficult...), but it would have solved this
> one.  The proposed import allows us to put our own version of Numeric in the
> top SciPy directory.  Then all SciPy sub-packages would grab this one
> instead of an existing site-packages/Numeric.

But then, this is not an import problem.
If you use Numeric, you call Numeric. If you call something other than Numeric,
just give a different name, and all the confusion will go away.
If you're worried that you've already encoded the Numeric name 50 times  into
300 files; run a python script over these 300 files; this will do the renaming
of the 15.000 occurences of the Numeric name.

> That makes SciPy
> self-contained and allows people to try it out without worrying that it
> might break their current installation.  There are other solutions to this
> problem, but Prabhu's fix is by far the easiest and most robust.

And then, in maintenance/integration phase, sometimes 'Numeric' will call
Numeric, some other times it will your package ?

What if somebody, for some reason I know nothing of (e.g. probably some
integration) wants to call Numeric and your Numeric package in the same module ?
Wish them tough luck to sort out this poisoned gift....

> Prabhu's import also has some other nice benefits.  Some of the sub-packages
> in SciPy are useful outside of SciPy.  Also sometimes it is easier to
> develop a packages outside of the SciPy framework.  It would be nice to be
> able to develop a module or package 'foo' outside of SciPy and then move it
> into SciPy at a later date.  However, every SciPy sub-package that referred
> to foo prior to its inclusion in SciPy now has to be updated from 'import
> foo' to 'import scipy.foo'.  These kind of issues make it very painful and
> time consuming to rearrange package structures or move modules and
> sub-packages in and out of the package.

There are basic python scripts which do this painlessly. If you're really
working on a large project, there's a project architect which normally would
take care of such things, and for whom this should not be a too much of a
problem.


> Simplifying this will improves
> package development.
>
> > I'm personnally against anything that enlarges the search path uselessly;
>
> Hopefully I've explained why it is useful for complex packages.

Python helps in many areas, but expecting it to palliate for the package design
and architecture flaws that inexorably surface anytimes something non-trivial is
developped, might be somehow at the edge. Python has not yet replaced the need
for relevant software architects.

>
> > because the obvious reason of increased name space collision, increased
> > run-time overhead etc...
>
> I'm missing something here because I don't understand why this increases
> name space collision.  If the objection is to the fact that SciPy can have a
> version of Numeric in it that masks a Numeric installed in site-packages, I
> guess I consider this a feature, not a bug.

Actually, it is normally worse than a bug: it is a source of bug tomorrow in
your application - of all the bugs you'll have when your programmer will be
confusing the two Numeric packages, as well as all the mainteance and
integration problems you'll have down the line -.

But by then, hopefully for you, you'll be somewhere else... The sad reality of
most projects :((

>  Afterall, this is already the
> behavior for single level packages, extending it to multi-level packages
> seems natural.  If this isn't your objection, please explain.
>
> The current runtime overhead isn't so bad.  Prabhu sent me a few numbers on
> the SciPy import (which contains maybe 10-15 nested packages).  I attached
> them below -- the overhead is less than 10%.  It should be negligible for
> standard modules as only packages are really affected (right Prabhu?).

And that's how, when you cumulate of the overheads for all new features, you get
potenially +100-200% overhead on the new releases.
Albeit all the efforts of the Python team, Python 2.0 is up to 70% slower than
python 1.5.2; Python 2.1.1 is up to 30% slower than python 2.0, and so on...
So, +10% on only such a minor features is anything but negligible :(((

FG



From gmcm@hypernet.com  Sat Nov 10 21:50:32 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Sat, 10 Nov 2001 16:50:32 -0500
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
In-Reply-To: <033301c16a1a$59c1c690$c300a8c0@ericlaptop>
Message-ID: <3BED5AD8.24350.78407262@localhost>

eric wrote:

[Frederic Giacometti]

> > because the obvious reason of increased name space collision,
> > increased run-time overhead etc...
> 
> I'm missing something here because I don't understand why this
> increases name space collision.  

Currently, os.py in a package masks the real one from 
anywhere inside the package. This would extend that to 
anywhere inside any nested subpackage. Whether that's a 
"neat" or a "dirty" trick is pretty subjective. The wider the 
namespace you can trample on, the more it tends to be "dirty".

> If the objection is to the fact
> that SciPy can have a version of Numeric in it that masks a
> Numeric installed in site-packages, I guess I consider this a
> feature, not a bug.  Afterall, this is already the behavior for
> single level packages, extending it to multi-level packages seems
> natural.  If this isn't your objection, please explain.

Well, it's a feature that can crash Python. If the package 
(which the user has, and you have a hijacked, incompatible 
copy of) contains an extension module, all kinds of nasty 
things can happen when both are loaded.

Submit patches to the package authors, or require a specific 
version, or write a wrapper that adapts to different versions or 
fork or do without.  This is definitely a dirty trick.
 
> The current runtime overhead isn't so bad. 

Under anything near normal usage, no - packages structures 
are nearly always shallow. It wouldn't be much work to 
construct a case where time spent in import doubled, however.

When the "try relative, then try absolute" strategy was 
introduced with packages, it added insignificant overhead. It's 
not so insignificant now. When (and if) the standard library 
moves to a package structure, it's possilbe it will be seen as a 
burden.


- Gordon


From gmcm@hypernet.com  Sat Nov 10 23:06:48 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Sat, 10 Nov 2001 18:06:48 -0500
Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism.
In-Reply-To: <15340.58785.94412.122093@monster.linux.in>
Message-ID: <3BED6CB8.27669.78864436@localhost>

Prabhu wrote:

[Just singling out one section, since I've said plenty on this 
subject at other points in these threads]

[Prabhu works on knee.py]

> it also fixes a bug where the parent package is an extension
> module.

Python provides no support for an extension module being a 
package parent module. More precisely, I think the fact that 
an extension module can be made to behave like a package 
parent module is an accident. There is special code in import 
for modules named __init__, and the code is bypassed for 
extension modules.

I suspect you'd have to provide a pretty strong justification 
before this would become supported behavior.


- Gordon


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Sun Nov 11 04:35:13 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Sun, 11 Nov 2001 10:05:13 +0530
Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism.
In-Reply-To: <3BED6CB8.27669.78864436@localhost>
References: <15340.58785.94412.122093@monster.linux.in>
 <3BED6CB8.27669.78864436@localhost>
Message-ID: <15342.1.830165.25074@monster.linux.in>

>>>>> "GMcM" == Gordon McMillan <gmcm@hypernet.com> writes:

    GMcM> [Prabhu works on knee.py]

    >> it also fixes a bug where the parent package is an extension
    >> module.

    GMcM> Python provides no support for an extension module being a
    GMcM> package parent module. More precisely, I think the fact that
    GMcM> an extension module can be made to behave like a package
    GMcM> parent module is an accident. There is special code in
    GMcM> import for modules named __init__, and the code is bypassed
    GMcM> for extension modules.

    GMcM> I suspect you'd have to provide a pretty strong
    GMcM> justification before this would become supported behavior.

I guess this was unclear.  My addition is extremely simple and does
not do anything new.  Here is an illustration

>>> import knee
>>> import Numeric.array
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/local/lib/python2.1/knee.py", line 17, in import_hook
    m = load_tail(q, tail)
  File "/usr/local/lib/python2.1/knee.py", line 68, in load_tail
    m = import_module(head, mname, m)
  File "/usr/local/lib/python2.1/knee.py", line 97, in import_module
    parent and parent.__path__)
AttributeError: 'Numeric' module has no attribute '__path__'
>>>

Point is, there is a line in knee.py (line 97 that assumes that there
is a __path__ attribute for the passed parent.  However, if parent is
an extension module there is none.  So I simply modified it.  Here is
the diff.

$ diff  knee.py  /usr/local/lib/python2.1/knee.py
98,101c98
<     except (ImportError, AttributeError):
<         # extension modules dont have a __path__ attribute.
<         # caching failures.
<         sys.modules[fqname] = None
---
>     except ImportError:

In fact that is all I changed in knee.py!  Which is why I said the
changes are very small.  Maybe I should have shown a patch but the
mail was already long.

prabhu



From ej@ee.duke.edu  Sun Nov 11 04:44:44 2001
From: ej@ee.duke.edu (eric)
Date: Sat, 10 Nov 2001 23:44:44 -0500
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
References: <3BED5AD8.24350.78407262@localhost>
Message-ID: <037a01c16a6b$993421a0$c300a8c0@ericlaptop>

Hey Gordon,

> eric wrote:
>
> [Frederic Giacometti]
>
> > > because the obvious reason of increased name space collision,
> > > increased run-time overhead etc...
> >
> > I'm missing something here because I don't understand why this
> > increases name space collision.
>
> Currently, os.py in a package masks the real one from
> anywhere inside the package. This would extend that to
> anywhere inside any nested subpackage. Whether that's a
> "neat" or a "dirty" trick is pretty subjective. The wider the
> namespace you can trample on, the more it tends to be "dirty".

Yeah, I guess I come down on the "neat" side in this one.  If I have a
module or package called 'common' at the top level of a deep hierarchy,
I'd like all sub-packages to inherit it.  That seems intuitive to me and
inline with the concept of a 'package'.  Perhaps the hijacking of the
Numeric example strikes a nerve, but inheriting the 'common' module
shouldn't be so contentious.  Also, if someone has the gall to hijack
os.py at the top of your package directory structure, it seems very
likely you want this new behavior everywhere within your package.

I have a feeling this discussion has been around the block a few times
when packages were first being developed...

>
> > If the objection is to the fact
> > that SciPy can have a version of Numeric in it that masks a
> > Numeric installed in site-packages, I guess I consider this a
> > feature, not a bug.  Afterall, this is already the behavior for
> > single level packages, extending it to multi-level packages seems
> > natural.  If this isn't your objection, please explain.
>
> Well, it's a feature that can crash Python. If the package
> (which the user has, and you have a hijacked, incompatible
> copy of) contains an extension module, all kinds of nasty
> things can happen when both are loaded.

This I need to know about.  How does this happen?  So you have
two extension modules, with the same name, one living in a package
and the other living in site-packages.  If you import both of these, their
namespaces don't conflict in some strange way do they?

Or are you talking about passing a structure (like a numeric array)
generated in one ext module into a routine in the other ext module
(expecting a different format of a numeric array) and then getting
some strange (seg-fault even) kind of behavior?  Anyway, I'd like
a few more details for reasons orthogonal to this discussion.

> Submit patches to the package authors, or require a specific
> version, or write a wrapper that adapts to different versions or
> fork or do without.  This is definitely a dirty trick.

We've done the "require specific version" option here, and
"conveniently" upgraded the user's Numeric package for them.
The problem is that some people use old versions of Numeric
in production code, and don't want to risk an upgrade -- but still
want to try out SciPy.  I consider our solution a dirtier trick
than encapsulating things completely within SciPy.  I also don't
think any nasty things could happen in this specific situation of
having two relatively recent Numerics loaded up, but I could be wrong.

>
> > The current runtime overhead isn't so bad.
>
> Under anything near normal usage, no - packages structures
> are nearly always shallow. It wouldn't be much work to
> construct a case where time spent in import doubled, however.
>
> When the "try relative, then try absolute" strategy was
> introduced with packages, it added insignificant overhead. It's
> not so insignificant now. When (and if) the standard library
> moves to a package structure, it's possilbe it will be seen as a
> burden.

I haven't followed the discussion as to whether the standard library
will move to packages, but I'll be surpised if it does.  I very much
like the encapsulation offered by packages, but have found their
current incarnation difficult to develop compared to simple modules.
The current discussion concerns only one of the issues.

As for overhead, I thought I'd get a couple more data points from
distutils and xml since they are standard packages.  The distutils import
is pretty much a 0% hit.  However, the xml import is *much* slower --
a factor of 3.5.  Thats a huge hit and worth complaining about.  I don't
know if this can be optimized or not.  If not, it may be a show stopper,
even if the philosophical argument was uncontested.

eric

import speed numbers below:

C:\temp>python
ActivePython 2.1, build 210 ActiveState)
based on Python 2.1 (#15, Apr 23 2001, 18:00:35) [MSC 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> t1 = time.time();import distutils.command.build; t2 = time.time();print
t2-t
1
0.519999980927
>>>

C:\temp>python
ActivePython 2.1, build 210 ActiveState)
based on Python 2.1 (#15, Apr 23 2001, 18:00:35) [MSC 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_import
>>> import time
>>> t1 = time.time();import distutils.command.build; t2 = time.time();print
t2-t
1
0.511000037193
>>>

C:\temp>python
ActivePython 2.1, build 210 ActiveState)
based on Python 2.1 (#15, Apr 23 2001, 18:00:35) [MSC 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_import
>>> import time
>>> t1 = time.time();import xml.sax.saxutils; t2 = time.time();print t2-t1
1.35199999809
>>>

C:\temp>python
ActivePython 2.1, build 210 ActiveState)
based on Python 2.1 (#15, Apr 23 2001, 18:00:35) [MSC 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> t1 = time.time();import xml.sax.saxutils; t2 = time.time();print t2-t1
0.381000041962
>>>








From ej@ee.duke.edu  Sun Nov 11 05:40:54 2001
From: ej@ee.duke.edu (eric)
Date: Sun, 11 Nov 2001 00:40:54 -0500
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
References: <E162bVD-0001nY-00@mail.python.org> <3BED6E5B.8E142057@arakne.com> <033301c16a1a$59c1c690$c300a8c0@ericlaptop> <3BED9F9A.30F77DB3@arakne.com>
Message-ID: <038301c16a73$70c5f790$c300a8c0@ericlaptop>

Hey Frederic,

> But then, this is not an import problem.
> If you use Numeric, you call Numeric. If you call something other than
Numeric,
> just give a different name, and all the confusion will go away.

This is certainly an option, but not a good one in my opinion.  The main
issue is
we want to force a specific version of Numeric for SciPy while allowing
people
to keep their old standard version of Numeric available for their production
code.
Single level packages provide a handy way of doing this.  Multi-level
packages
(like SciPy) do not.  I guess I just don't see why making a package
multi-level
should inherently make it harder to do things.

> If you're worried that you've already encoded the Numeric name 50 times
into
> 300 files; run a python script over these 300 files; this will do the
renaming
> of the 15.000 occurences of the Numeric name.

Sure, but this is inconvenient, and something I think should be handled
by the packaging facility, not by running a renaming script.

>
> > That makes SciPy
> > self-contained and allows people to try it out without worrying that it
> > might break their current installation.  There are other solutions to
this
> > problem, but Prabhu's fix is by far the easiest and most robust.
>
> And then, in maintenance/integration phase, sometimes 'Numeric' will call
> Numeric, some other times it will your package ?

In the integration phase, users would need to change the
"from Numeric import *" to "from scipy import *" (or search and replace
Numeric with scipy) in their code as Numeric is completely subsumed
into scipy.  So, when not using SciPy, their legacy code can continue using
an old version of Numeric.  When switching to SciPy, the make the
replacement.  As I said, its mainly a version issue (with a few minor
changes).

>
> What if somebody, for some reason I know nothing of (e.g. probably some
> integration) wants to call Numeric and your Numeric package in the same
module ?
> Wish them tough luck to sort out this poisoned gift....
>
> > Prabhu's import also has some other nice benefits.  Some of the
sub-packages
> > in SciPy are useful outside of SciPy.  Also sometimes it is easier to
> > develop a packages outside of the SciPy framework.  It would be nice to
be
> > able to develop a module or package 'foo' outside of SciPy and then move
it
> > into SciPy at a later date.  However, every SciPy sub-package that
referred
> > to foo prior to its inclusion in SciPy now has to be updated from
'import
> > foo' to 'import scipy.foo'.  These kind of issues make it very painful
and
> > time consuming to rearrange package structures or move modules and
> > sub-packages in and out of the package.
>
> There are basic python scripts which do this painlessly. If you're really
> working on a large project, there's a project architect which normally
would
> take care of such things, and for whom this should not be a too much of a
> problem.

Hmmm.  I guess the "project architect" in this case is jointly held by
Travis Oliphant and yours truely.  Neither of us are packaging guru's, but
do
have a fair amount of experience with Python.  We worked quite a while on
(and are still working on) packaging issues.  Incidently,
I have know idea what Travis O.'s opinion is on this specific topic.

>
>
> > Simplifying this will improves
> > package development.
> >
> > > I'm personnally against anything that enlarges the search path
uselessly;
> >
> > Hopefully I've explained why it is useful for complex packages.
>
> Python helps in many areas, but expecting it to palliate for the package
design
> and architecture flaws that inexorably surface anytimes something
non-trivial is
> developped, might be somehow at the edge. Python has not yet replaced the
need
> for relevant software architects.

Them thars fightin' words. ; )  I'm biased, but don't thinking scipy's
architecture is
flawed.  It is simply a *very* large package of integrated sub-packages that
also
relies heavily on a 3rd evolving group of modules (Numeric).  As such, it
reveals
the difficult issues that arise when trying to build large packages of
integrated
sub-packages that rely on a 3rd evolving group of modules...

>
> >
> > > because the obvious reason of increased name space collision,
increased
> > > run-time overhead etc...
> >
> > I'm missing something here because I don't understand why this increases
> > name space collision.  If the objection is to the fact that SciPy can
have a
> > version of Numeric in it that masks a Numeric installed in
site-packages, I
> > guess I consider this a feature, not a bug.
>
> Actually, it is normally worse than a bug: it is a source of bug tomorrow
in
> your application - of all the bugs you'll have when your programmer will
be
> confusing the two Numeric packages, as well as all the mainteance and
> integration problems you'll have down the line -.


I disagree and don't think that is true in this (and many other) situations.
People
who want to use SciPy will migrate completely to it since it includes
Numeric.
What the sub-package option offers is a way to test SciPy and optionally use
it while keeping their standard Numeric around for their production code.

>
> But by then, hopefully for you, you'll be somewhere else... The sad
reality of
> most projects :((
>
> >  Afterall, this is already the
> > behavior for single level packages, extending it to multi-level packages
> > seems natural.  If this isn't your objection, please explain.
> >
> > The current runtime overhead isn't so bad.  Prabhu sent me a few numbers
on
> > the SciPy import (which contains maybe 10-15 nested packages).  I
attached
> > them below -- the overhead is less than 10%.  It should be negligible
for
> > standard modules as only packages are really affected (right Prabhu?).
>
> And that's how, when you cumulate of the overheads for all new features,
you get
> potenially +100-200% overhead on the new releases.
> Albeit all the efforts of the Python team, Python 2.0 is up to 70% slower
than
> python 1.5.2; Python 2.1.1 is up to 30% slower than python 2.0, and so
on...
> So, +10% on only such a minor features is anything but negligible :(((


The computational cost of additional functionality is always a question of
what
portion of a program is impacted.  If we were talking about 10% hit on
looping
structures or dictionary lookups or local variable lookups, then yes it
needs
extreme scrutiny.  Adding 10% to a rare event is not worthy of note.  I
expect
(and see) 0% overhead for importing standard modules (by far the most
common case).  Adding 10% overhead to importing a very large package
with 10-15 nested sub-packages is just not a big deal.  The 350% cost I
saw (noted in a response to Gordon) is a *huge* deal and would need to be
solved (moving to C would help) before this became standard.

eric

----- Original Message -----
From: "Frederic Giacometti" <frederic.giacometti@arakne.com>
To: "eric" <ej@ee.duke.edu>
Cc: <import-sig@python.org>; <prabhu@cyberwaveindia.com>;
<python-list@python.org>; <python-dev@python.org>
Sent: Saturday, November 10, 2001 4:43 PM
Subject: Re: Proposal for a modified import mechanism.


>
>
> eric wrote:
>
> > I have to agree with Prabhu on this one.  The current behavior of
import,
> > while fine for standard modules and even simple packages with a single
> > level, is sub-optimal for packages that contain sub-packages.  The
proposed
> > behavior solves the problem.
> >
> > Handling the packaging issues in SciPy was difficult, and even resulted
in a
> > (not always popular) decision to build and overwrite the Numeric package
on
> > machines that install SciPy.  Prabhu's import doesn't resolve all the
issues
> > (I think packages may just be difficult...), but it would have solved
this
> > one.  The proposed import allows us to put our own version of Numeric in
the
> > top SciPy directory.  Then all SciPy sub-packages would grab this one
> > instead of an existing site-packages/Numeric.
>
> But then, this is not an import problem.
> If you use Numeric, you call Numeric. If you call something other than
Numeric,
> just give a different name, and all the confusion will go away.
> If you're worried that you've already encoded the Numeric name 50 times
into
> 300 files; run a python script over these 300 files; this will do the
renaming
> of the 15.000 occurences of the Numeric name.
>
> > That makes SciPy
> > self-contained and allows people to try it out without worrying that it
> > might break their current installation.  There are other solutions to
this
> > problem, but Prabhu's fix is by far the easiest and most robust.
>
> And then, in maintenance/integration phase, sometimes 'Numeric' will call
> Numeric, some other times it will your package ?
>
> What if somebody, for some reason I know nothing of (e.g. probably some
> integration) wants to call Numeric and your Numeric package in the same
module ?
> Wish them tough luck to sort out this poisoned gift....
>
> > Prabhu's import also has some other nice benefits.  Some of the
sub-packages
> > in SciPy are useful outside of SciPy.  Also sometimes it is easier to
> > develop a packages outside of the SciPy framework.  It would be nice to
be
> > able to develop a module or package 'foo' outside of SciPy and then move
it
> > into SciPy at a later date.  However, every SciPy sub-package that
referred
> > to foo prior to its inclusion in SciPy now has to be updated from
'import
> > foo' to 'import scipy.foo'.  These kind of issues make it very painful
and
> > time consuming to rearrange package structures or move modules and
> > sub-packages in and out of the package.
>
> There are basic python scripts which do this painlessly. If you're really
> working on a large project, there's a project architect which normally
would
> take care of such things, and for whom this should not be a too much of a
> problem.
>
>
> > Simplifying this will improves
> > package development.
> >
> > > I'm personnally against anything that enlarges the search path
uselessly;
> >
> > Hopefully I've explained why it is useful for complex packages.
>
> Python helps in many areas, but expecting it to palliate for the package
design
> and architecture flaws that inexorably surface anytimes something
non-trivial is
> developped, might be somehow at the edge. Python has not yet replaced the
need
> for relevant software architects.
>
> >
> > > because the obvious reason of increased name space collision,
increased
> > > run-time overhead etc...
> >
> > I'm missing something here because I don't understand why this increases
> > name space collision.  If the objection is to the fact that SciPy can
have a
> > version of Numeric in it that masks a Numeric installed in
site-packages, I
> > guess I consider this a feature, not a bug.
>
> Actually, it is normally worse than a bug: it is a source of bug tomorrow
in
> your application - of all the bugs you'll have when your programmer will
be
> confusing the two Numeric packages, as well as all the mainteance and
> integration problems you'll have down the line -.
>
> But by then, hopefully for you, you'll be somewhere else... The sad
reality of
> most projects :((
>
> >  Afterall, this is already the
> > behavior for single level packages, extending it to multi-level packages
> > seems natural.  If this isn't your objection, please explain.
> >
> > The current runtime overhead isn't so bad.  Prabhu sent me a few numbers
on
> > the SciPy import (which contains maybe 10-15 nested packages).  I
attached
> > them below -- the overhead is less than 10%.  It should be negligible
for
> > standard modules as only packages are really affected (right Prabhu?).
>
> And that's how, when you cumulate of the overheads for all new features,
you get
> potenially +100-200% overhead on the new releases.
> Albeit all the efforts of the Python team, Python 2.0 is up to 70% slower
than
> python 1.5.2; Python 2.1.1 is up to 30% slower than python 2.0, and so
on...
> So, +10% on only such a minor features is anything but negligible :(((
>
> FG




From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Sun Nov 11 07:42:03 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Sun, 11 Nov 2001 13:12:03 +0530
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
In-Reply-To: <037a01c16a6b$993421a0$c300a8c0@ericlaptop>
References: <3BED5AD8.24350.78407262@localhost>
 <037a01c16a6b$993421a0$c300a8c0@ericlaptop>
Message-ID: <15342.11211.377711.175571@monster.linux.in>

Hi,

>>>>> "ej" == "eric" <ej@ee.duke.edu> writes:

    >> Currently, os.py in a package masks the real one from anywhere
    >> inside the package. This would extend that to anywhere inside
    >> any nested subpackage. Whether that's a "neat" or a "dirty"
    >> trick is pretty subjective. The wider the namespace you can
    >> trample on, the more it tends to be "dirty".

    ej> Yeah, I guess I come down on the "neat" side in this one.  If
    ej> I have a module or package called 'common' at the top level of
    ej> a deep hierarchy, I'd like all sub-packages to inherit it.
    ej> That seems intuitive to me and inline with the concept of a
    ej> 'package'.  Perhaps the hijacking of the Numeric example
    ej> strikes a nerve, but inheriting the 'common' module shouldn't
    ej> be so contentious.  Also, if someone has the gall to hijack
    ej> os.py at the top of your package directory structure, it seems
    ej> very likely you want this new behavior everywhere within your
    ej> package.

I agree on this.  Also each package is kind of isolated.  Any module
like os.py inside a sub package won't affect _every_ other sub package
and will only affect packages that are nested inside this particular
package.  So there is some kind of safety net and its not like
sticking everything inside sys.path. :)

Also, right now, what prevents someone from sticking an os.py
somewhere in sys.path and completely ruining standard behaviour.  So,
its not asif this new approach to importing package makes things
dirty, you can very well do 'bad' things right now.

[snip]

    ej> As for overhead, I thought I'd get a couple more data points
    ej> from distutils and xml since they are standard packages.  The
    ej> distutils import is pretty much a 0% hit.  However, the xml
    ej> import is *much* slower -- a factor of 3.5.  Thats a huge hit
    ej> and worth complaining about.  I don't know if this can be
    ej> optimized or not.  If not, it may be a show stopper, even if
    ej> the philosophical argument was uncontested.

    >>>> import my_import import time t1 = time.time();import
    >>>> xml.sax.saxutils; t2 = time.time();print t2-t1
    1.35199999809

    >>>> import time t1 = time.time();import xml.sax.saxutils; t2 =
    >>>> time.time();print t2-t1
    0.381000041962

IMHO, this is an unfair/wrong comparison.

      (0) I suspect that you did not first clean things up by doing a
      plain import xml.sax.saxutils a few times and then start
      testing.

      (1) import itself is implemented in C.  my_import is pretty much
      completely in Python.  

Here is a fairer comparison (done after a few imports).

>>> import time
>>> s = time.time (); import xml.sax.saxutils; print time.time()-s
0.0434629917145

>>> import my_import
>>> import time
>>> s = time.time (); import xml.sax.saxutils; print time.time()-s
0.0503059625626

Which is still not bad at all and nothing close to 350% slowdown.  But
to see if the presently measured slowdown is not the parent lookup we
really need to compare things against the modified (to cache failures)
knee.py:

>>> import knee
>>> import time
>>> s = time.time (); import xml.sax.saxutils; print time.time()-s
0.0477709770203

>>> import my_import
>>> import time
>>> s = time.time (); import xml.sax.saxutils; print time.time()-s
0.0501489639282

Which is really not very bad since its just a 5% slowdown.

Here are more tests for scipy:

>>> import time
>>> s = time.time (); import scipy; print time.time()-s
1.36110007763

>>> import knee, time
>>> s = time.time (); import scipy; print time.time()-s
1.48176395893

>>> import my_import, time
>>> s = time.time (); import scipy; print time.time()-s
1.5150359869

Which means that doing the parent lookup stuff in this case is really
not so bad and the biggest slowdown is mostly thanks to knee being
implemented in Python.  And there is no question of a 350% slowdown!!
:)

prabhu


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Sun Nov 11 08:03:59 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Sun, 11 Nov 2001 13:33:59 +0530
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
In-Reply-To: <033301c16a1a$59c1c690$c300a8c0@ericlaptop>
References: <E162bVD-0001nY-00@mail.python.org>
 <3BED6E5B.8E142057@arakne.com>
 <033301c16a1a$59c1c690$c300a8c0@ericlaptop>
Message-ID: <15342.12527.94615.833898@monster.linux.in>

>>>>> "ej" == ej  <eric> writes:

[snip]
    ej> The current runtime overhead isn't so bad.  Prabhu sent me a
    ej> few numbers on the SciPy import (which contains maybe 10-15
    ej> nested packages).  I attached them below -- the overhead is
    ej> less than 10%.  It should be negligible for standard modules
    ej> as only packages are really affected (right Prabhu?).

It depends on how you do it.  If you have a sub-package that tries to
import a standard module it will go through all the parent packages
searching for the module and when it doesn't find one it will check in
sys.path.  There are a few things to note:

  (1) For a module in a package, the first import will be naturally
  the slowest.

  (2) Subsequent imports will be faster since failures are cached and
  the package is already imported.

  (3) If the module in question is not inside a package there will be
  no slowdown whatsoever since there is no parent package at all.
  I've timed this with vtk and it seems to be correct.

>>> import my_import, time # NOTE: I am not inside any package.
>>> s = time.time (); import vtkpython; print time.time()-s
1.06130003929

>>> import time
>>> s = time.time (); import vtkpython; print time.time()-s
1.06413698196

Its slower with standard import you may say - but that might just be
my kernel's scheduling affecting things.  I think its fair to conclude
that there is no slowdown if you are not inside a package and based on
my earlier timings, that recursive searching thru package parents is
not too expensive either.

There is one issue.  lets say we have two sub-packages that have
modules of the same name.  Then if we explicitly want the other
sub-packages module to be imported there is currently no way of doing
it.  In such a case maybe adding a __parent__ or using (__ as ni did)
might be a good idea too.

prabhu


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Sun Nov 11 08:08:18 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Sun, 11 Nov 2001 13:38:18 +0530
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
In-Reply-To: <3BED5AD8.24350.78407262@localhost>
References: <033301c16a1a$59c1c690$c300a8c0@ericlaptop>
 <3BED5AD8.24350.78407262@localhost>
Message-ID: <15342.12786.547708.67405@monster.linux.in>

>>>>> "GMcM" == Gordon McMillan <gmcm@hypernet.com> writes:

[snipped off other issues raised]

    >> The current runtime overhead isn't so bad.

    GMcM> Under anything near normal usage, no - packages structures
    GMcM> are nearly always shallow. It wouldn't be much work to
    GMcM> construct a case where time spent in import doubled,
    GMcM> however.

But that can be said of almost anything.  A nicer question to ask is
-- for most circumstances (99%) is the import mechanism fast enough?

    GMcM> When the "try relative, then try absolute" strategy was
    GMcM> introduced with packages, it added insignificant
    GMcM> overhead. It's not so insignificant now. When (and if) the
    GMcM> standard library moves to a package structure, it's possilbe
    GMcM> it will be seen as a burden.

Yes, which is why maybe adding an 'rimport' keyword (which you
suggested) would be a more conservative option?

prabhu


From mal@lemburg.com  Sun Nov 11 14:05:50 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Sun, 11 Nov 2001 15:05:50 +0100
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
References: <E162bVD-0001nY-00@mail.python.org> <3BED6E5B.8E142057@arakne.com> <033301c16a1a$59c1c690$c300a8c0@ericlaptop>
Message-ID: <3BEE85BE.130B8B3C@lemburg.com>

[Please leave this on the import-sig; severe cross-posting won't get
 you many friends...]

eric wrote:
> 
> I have to agree with Prabhu on this one.  The current behavior of import,
> while fine for standard modules and even simple packages with a single
> level, is sub-optimal for packages that contain sub-packages.  The proposed
> behavior solves the problem.

I have been doing business in the package and subpackage area
for years now and can't really say that I have serious problems 
with the current situation.

If all you need is relative imports, then writing a small
helper for this should do the trick. The needed information
is readily available now. 

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/


From gmcm@hypernet.com  Sun Nov 11 15:52:20 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Sun, 11 Nov 2001 10:52:20 -0500
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
In-Reply-To: <037a01c16a6b$993421a0$c300a8c0@ericlaptop>
Message-ID: <3BEE5864.23998.7C1EDB5A@localhost>

> Hey Gordon,
> 
> > eric wrote:
> >
> > [Frederic Giacometti]
> >
> > > > because the obvious reason of increased name space
> > > > collision, increased run-time overhead etc...
> > >
> > > I'm missing something here because I don't understand why
> > > this increases name space collision.
> >
> > Currently, os.py in a package masks the real one from
> > anywhere inside the package. This would extend that to
> > anywhere inside any nested subpackage. Whether that's a
> > "neat" or a "dirty" trick is pretty subjective. The wider the
> > namespace you can trample on, the more it tends to be "dirty".
> 
> Yeah, I guess I come down on the "neat" side in this one.  If I
> have a module or package called 'common' at the top level of a
> deep hierarchy, I'd like all sub-packages to inherit it.  

I have no objection to having an easy way of getting to 
common from subpackages.

[snip]

> Also, if someone has the gall to hijack os.py at the top of your
> package directory structure, it seems very likely you want this
> new behavior everywhere within your package.

Perhaps. The trouble is that under this proposal there is no 
way to get to the real os.py, even when you know what's going 
on. The proposal doesn't fix a flaw in today's import, it extends 
the flaw. If there were different ways of spelling "absolute" and 
"relative", you would get what you want, and I would get what I 
want.
 
> I have a feeling this discussion has been around the block a few
> times when packages were first being developed...

Not really. I think it was Mark Hammond that convinced Guido 
of the usefulness of packages. It got hacked into import.c. It 
worked for the small existing population of packages and it 
basically hasn't been touched since then.

Package authors certainly haven't ended up following Java's 
model.
 
> >
> > > If the objection is to the fact
> > > that SciPy can have a version of Numeric in it that masks a
> > > Numeric installed in site-packages, I guess I consider this a
> > > feature, not a bug.  Afterall, this is already the behavior
> > > for single level packages, extending it to multi-level
> > > packages seems natural.  If this isn't your objection, please
> > > explain.
> >
> > Well, it's a feature that can crash Python. If the package
> > (which the user has, and you have a hijacked, incompatible copy
> > of) contains an extension module, all kinds of nasty things can
> > happen when both are loaded.
> 
> This I need to know about.  How does this happen?  So you have
> two extension modules, with the same name, one living in a
> package and the other living in site-packages.  If you import
> both of these, their namespaces don't conflict in some strange
> way do they?
> 
> Or are you talking about passing a structure (like a numeric
> array) generated in one ext module into a routine in the other
> ext module (expecting a different format of a numeric array) and
> then getting some strange (seg-fault even) kind of behavior? 
> Anyway, I'd like a few more details for reasons orthogonal to
> this discussion.

Much depends on platform. Windows will happily load 2, so 
errors are likely to be of the 2nd sort (although it may not take 
explicitly passing something around - the extension may be 
relying on TLS data, for example).

On platforms that do a real dynamic link, it's going to be C 
symbols that get stomped on.

[snip]

> I haven't followed the discussion as to whether the standard
> library will move to packages, but I'll be surpised if it does. 

It's been brought up (and delayed) with each Python release 
since packages became official.

> I very much like the encapsulation offered by packages, but have
> found their current incarnation difficult to develop compared to
> simple modules. The current discussion concerns only one of the
> issues.

And it's my contention that the problem this proposal "fixes" is 
a symptom of a deeper problem - the fact that the import 
namespace has a concept of "relative" but no way to spell it. 
The proposal makes some things work, and makes others 
worse.
 
> As for overhead, I thought I'd get a couple more data points from
> distutils and xml since they are standard packages.  The
> distutils import is pretty much a 0% hit.  However, the xml
> import is *much* slower -- a factor of 3.5.  

Prabhu objects to your numbers. Perhaps he's using the 
standard xml package, and you're using _xmlplus? The latter 
is vastly more complex.

- Gordon


From gmcm@hypernet.com  Sun Nov 11 15:52:20 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Sun, 11 Nov 2001 10:52:20 -0500
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
In-Reply-To: <15342.11211.377711.175571@monster.linux.in>
References: <037a01c16a6b$993421a0$c300a8c0@ericlaptop>
Message-ID: <3BEE5864.21151.7C1EDCCD@localhost>

Prabhu wrote:

> Also, right now, what prevents someone from sticking an os.py
> somewhere in sys.path and completely ruining standard behaviour. 

If they stick it after the real one, it has no effect. If the user 
sticks it in front, it's an installation issue, and the user's 
responsibility. If an author sticks it in front, it's a dirty trick that 
may or may not work.

> So, its not asif this new approach to importing package makes
> things dirty, you can very well do 'bad' things right now.

Of course. The problem here is that it's all implicit and there's 
no way around it.

 sys.modules['os'] = mydirtyhack
is pretty obvious. Putting an os.py (or os/__init__.py) inside a 
package is not. And once it's done, there's no workaround.



- Gordon


From gmcm@hypernet.com  Sun Nov 11 15:52:19 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Sun, 11 Nov 2001 10:52:19 -0500
Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism.
In-Reply-To: <15342.1.830165.25074@monster.linux.in>
References: <3BED6CB8.27669.78864436@localhost>
Message-ID: <3BEE5863.11711.7C1ED9A1@localhost>

Prabhu wrote:

> >>>>> "GMcM" == Gordon McMillan <gmcm@hypernet.com> writes:
> 
>     GMcM> [Prabhu works on knee.py]
> 
>     >> it also fixes a bug where the parent package is an
>     extension >> module.
> 
>     GMcM> Python provides no support for an extension module
>     being a GMcM> package parent module. More precisely, I think
>     the fact that GMcM> an extension module can be made to behave
>     like a package GMcM> parent module is an accident. 

[snip]

> I guess this was unclear.  My addition is extremely simple and
> does not do anything new.  Here is an illustration

And I wasn't clear. I wasn't complaining about your 
implementation. I was complaining that the rules of import are 
ill-specified. There are many import hacks in use today that 
rely on undocumented features, some of which are accidental.

When package support was introduced, no one knew how 
packages would end up being used. I would hate to see 
Python stuck supporting some of this crap under the guise of 
"backwards compatibility".

- Gordon


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Sun Nov 11 18:02:57 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Sun, 11 Nov 2001 23:32:57 +0530
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
In-Reply-To: <3BEE5864.23998.7C1EDB5A@localhost>
References: <037a01c16a6b$993421a0$c300a8c0@ericlaptop>
 <3BEE5864.23998.7C1EDB5A@localhost>
Message-ID: <15342.48465.531579.718808@monster.linux.in>

>>>>> "GMcM" == Gordon McMillan <gmcm@hypernet.com> writes:

[snip]

    >> Also, if someone has the gall to hijack os.py at the top of
    >> your package directory structure, it seems very likely you want
    >> this new behavior everywhere within your package.

    GMcM> Perhaps. The trouble is that under this proposal there is no
    GMcM> way to get to the real os.py, even when you know what's
    GMcM> going on. The proposal doesn't fix a flaw in today's import,
    GMcM> it extends the flaw. If there were different ways of
    GMcM> spelling "absolute" and "relative", you would get what you
    GMcM> want, and I would get what I want.

I agree.  My proposal was simply a way to clear up my problem.  I'll
admit I didn't think of the bigger picture.  What do you propose?  You
mentioned that import should be narrower in its search, rimport a
hybrid and rrimport do what i asked for.  This does, somewhat atleast
address the problem.

As regards getting the real os.py I think the best possible way would
be to put all standard modules into a package called std or something.
This need not necessarily require a package structure for standard
modules.  import could simply be modified to see if someone has asked
for std.foo and it will only provide the os.py that is found on
sys.path.  Of course a terrorist package could stick in a nasty .pth
file and expose its os.py as the _real_ one which is a problem but I
guess its unlikely someone would want to create such a bad package.

[snip]
 
    GMcM> And it's my contention that the problem this proposal
    GMcM> "fixes" is a symptom of a deeper problem - the fact that the
    GMcM> import namespace has a concept of "relative" but no way to
    GMcM> spell it.  The proposal makes some things work, and makes
    GMcM> others worse.

I understand but apart from concerns on speed (for which I really
haven't seen real examples) and concerns on issues on things like
exposing os.py, I really don't see anything becoming 'worse'.
However, I agree wholeheartedly that the import mechanism needs a
re-think and am willing to help in my limited capacity. :)
 
    GMcM> Prabhu objects to your numbers. Perhaps he's using the
    GMcM> standard xml package, and you're using _xmlplus? The latter
    GMcM> is vastly more complex.

Indeed, I am using the standard xml package.  I don't have the
_xmlplus package installed.  So I might be wrong but I really can't
believe that there is a 3.5 fold speed reduction.  I'd like to hear a
confirmation from Eric.  Anyway, is there some other large package
that you'd like me to test with.

prabhu


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Mon Nov 12 07:34:01 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Mon, 12 Nov 2001 13:04:01 +0530
Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism.
In-Reply-To: <3BEE5863.11711.7C1ED9A1@localhost>
References: <3BED6CB8.27669.78864436@localhost>
 <3BEE5863.11711.7C1ED9A1@localhost>
Message-ID: <15343.31593.765575.923037@monster.linux.in>

>>>>> "GMcM" == Gordon McMillan <gmcm@hypernet.com> writes:

    GMcM> [snip]

    >> I guess this was unclear.  My addition is extremely simple and
    >> does not do anything new.  Here is an illustration

    GMcM> And I wasn't clear. I wasn't complaining about your
    GMcM> implementation. I was complaining that the rules of import
    GMcM> are ill-specified. There are many import hacks in use today
    GMcM> that rely on undocumented features, some of which are
    GMcM> accidental.

    GMcM> When package support was introduced, no one knew how
    GMcM> packages would end up being used. I would hate to see Python
    GMcM> stuck supporting some of this crap under the guise of
    GMcM> "backwards compatibility".

Okay, so what do you propose?  What do you think is the best
solution?? import, rimport, rrimport??

prabhu

p.s. I am not on import-sig so please cc me in on replies.  Thanks.


From gmcm@hypernet.com  Mon Nov 12 14:06:28 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Mon, 12 Nov 2001 09:06:28 -0500
Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism.
In-Reply-To: <15343.31593.765575.923037@monster.linux.in>
References: <3BEE5863.11711.7C1ED9A1@localhost>
Message-ID: <3BEF9114.31305.80E44B37@localhost>

Prabhu wrote:
 
> Okay, so what do you propose?  What do you think is the best
> solution?? import, rimport, rrimport??

If I had access to Guido's time machine, I would probably 
change things so:
 import is "absolute import"
 rimport is "relative import"
 rrimport is "recursive relative import"

I don't, so I tend towards:
 aimport is "absolute import"
 rimport is "relative import"
 rrimport is "recursive relative import"
and
 import is 
    try: rimport
    except ImportError:
        aimport

I'm not delighted with these names (and avoid syntax debates 
like the plague), but they seem much clearer than grafting 
relative-path syntax onto package / module names. For one 
thing, all the familiar (to Windows & *nix users) punctuation 
characters are unusable. For another, mac reverses the 
spellings (so <path-sep><name> is relative and <name> is 
absolute).

My rather unreliable crystal ball tells me the number one 
objection to this will be "Why should I have to spell it out - I 
just want to import it". My reply to that would be the same as 
my reply to those who don't want to be bothered by having to 
choose a spelling for division: at the time you write the code, 
you know exactly what you want. When your code runs, we 
can only guess what you wanted.



- Gordon


From jim@interet.com  Mon Nov 12 14:52:57 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 12 Nov 2001 09:52:57 -0500
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
References: <3BED5AD8.24350.78407262@localhost>
Message-ID: <3BEFE249.371EA5FD@interet.com>

Gordon McMillan wrote:

> Currently, os.py in a package masks the real one from
> anywhere inside the package. This would extend that to

What???

When Python starts, it imports site.py which imports os.py.
So os.py gets loaded, and won't normally get re-loaded.
The os.py that gets loaded depends on sys.path.

So if os.py is in package1, it won't get loaded for "import os",
but it would get loaded for "import package1.os".  Are you saying
that "import package1.package2.os" will load package1/os.py?
I hope that "import os" will not load package1/os.py, will it?
Or am I totally confused.

Jim Ahlstrom


From jim@interet.com  Mon Nov 12 15:02:59 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Mon, 12 Nov 2001 10:02:59 -0500
Subject: [Import-sig] Re: [Python-Dev] Re: Proposal for a modified import mechanism.
References: <E162bVD-0001nY-00@mail.python.org> <3BED6E5B.8E142057@arakne.com> <033301c16a1a$59c1c690$c300a8c0@ericlaptop> <3BED9F9A.30F77DB3@arakne.com> <038301c16a73$70c5f790$c300a8c0@ericlaptop>
Message-ID: <3BEFE4A3.63DF1A21@interet.com>

eric wrote:

> (and see) 0% overhead for importing standard modules (by far the most
> common case).  Adding 10% overhead to importing a very large package
> with 10-15 nested sub-packages is just not a big deal.  The 350% cost I
> saw (noted in a response to Gordon) is a *huge* deal and would need to be
> solved (moving to C would help) before this became standard.

I have code which caches directory contents, and related benchmarks.
This might help, and could be combined with a new Python module
for importing packages, say, as a new method "importer" in __init__.py.
Please see python-dev, "Caching directory files in import.c".

JimA


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Mon Nov 12 15:42:05 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Mon, 12 Nov 2001 21:12:05 +0530
Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism.
In-Reply-To: <3BEF9114.31305.80E44B37@localhost>
References: <3BEE5863.11711.7C1ED9A1@localhost>
 <3BEF9114.31305.80E44B37@localhost>
Message-ID: <15343.60877.693648.637512@monster.linux.in>

>>>>> "GMcM" == Gordon McMillan <gmcm@hypernet.com> writes:

    GMcM> Prabhu wrote:
    >> Okay, so what do you propose?  What do you think is the best
    >> solution?? import, rimport, rrimport??

    GMcM> If I had access to Guido's time machine, I would probably
    GMcM> change things so: import is "absolute import" rimport is
    GMcM> "relative import" rrimport is "recursive relative import"

Is borrowing the time machine an option? <wink>

    GMcM> I don't, so I tend towards: aimport is "absolute import"
    GMcM> rimport is "relative import" rrimport is "recursive relative
    GMcM> import" and import is try: rimport except ImportError:
    GMcM> aimport

It sounds good but I dont see the point with introducing rimport and
rrimport.  Afterall, import is going to do rimport so maybe all we
need is:

import - "same old"
aimport - "absolute import"
rimport - "recursive relative import"

    GMcM> My rather unreliable crystal ball tells me the number one
    GMcM> objection to this will be "Why should I have to spell it out
    GMcM> - I just want to import it". My reply to that would be the
    GMcM> same as my reply to those who don't want to be bothered by
    GMcM> having to choose a spelling for division: at the time you
    GMcM> write the code, you know exactly what you want. When your
    GMcM> code runs, we can only guess what you wanted.

Makes sense.

prabhu


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Mon Nov 12 15:53:20 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Mon, 12 Nov 2001 21:23:20 +0530
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <3BEFE249.371EA5FD@interet.com>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
Message-ID: <15343.61552.414932.44100@monster.linux.in>

>>>>> "JCA" == James C Ahlstrom <jim@interet.com> writes:

    JCA> When Python starts, it imports site.py which imports os.py.
    JCA> So os.py gets loaded, and won't normally get re-loaded.  The
    JCA> os.py that gets loaded depends on sys.path.

    JCA> So if os.py is in package1, it won't get loaded for "import
    JCA> os", but it would get loaded for "import package1.os".  Are
    JCA> you saying that "import package1.package2.os" will load
    JCA> package1/os.py?  I hope that "import os" will not load
    JCA> package1/os.py, will it?  Or am I totally confused.

Ummm doing an 'import os' will import the package1/os.py and *not* the
standard one.  This will happen even though os.py was imported earlier
by site.py.  This is what Gordon was objecting to in the first place
and why he proposes using rimport, rrimport etc. to make things more
explicit.

prabhu


From jeremy@zope.com  Mon Nov 12 16:33:19 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Mon, 12 Nov 2001 11:33:19 -0500 (EST)
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <15343.61552.414932.44100@monster.linux.in>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
Message-ID: <15343.63951.137972.487852@slothrop.digicool.com>

>>>>> "PR" == Prabhu Ramachandran <prabhu@aero.iitm.ernet.in> writes:

  PR> Ummm doing an 'import os' will import the package1/os.py and
  PR> *not* the standard one.  This will happen even though os.py was
  PR> imported earlier by site.py.  This is what Gordon was objecting
  PR> to in the first place and why he proposes using rimport,
  PR> rrimport etc. to make things more explicit.

Of course, you can use the existing mechanism to do this: 'from
package1 import os'.  The use of an explicit name seems like the
clearest route when you have a package-local module that shadows a
top-level module -- no need to understand details of relative imports,
no question about what is intended by the code.

I haven't followed this thread closely.  Is there some reason that
explicit names in imports is not sufficient?

Jeremy



From fredrik@pythonware.com  Mon Nov 12 17:05:30 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Mon, 12 Nov 2001 18:05:30 +0100
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import  mechanism.
References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com>
Message-ID: <006a01c16b9d$c31fc040$ced241d5@hagrid>

James C. Ahlstrom wrote:
> > Currently, os.py in a package masks the real one from
> > anywhere inside the package. This would extend that to
> 
> What???
> 
> When Python starts, it imports site.py which imports os.py.
> So os.py gets loaded, and won't normally get re-loaded.

a python system doesn't necessarily load site.py, but
nevermind...

> So if os.py is in package1, it won't get loaded for "import os",
> but it would get loaded for "import package1.os".  Are you saying
> that "import package1.package2.os" will load package1/os.py?

no, but an "import os" from inside the "spam.py" module installed
as "package1.spam" will attempt to import "package1.os" before
it looks for "os" in the path.

consider this (somewhat simplified) directory structure:

    package1/__init__.py
    package1/spam.py
    package1/os.py

and this interpreter session:

$ python -vvS
Python 2.1.1
Type "copyright", "credits" or "license" for more information.
>>> import package1
import package1 # directory pythonware
# trying package1/__init__.py
# package1/__init__.pyc matches package1/__init__.py
import pythonware # precompiled from package1/__init__.pyc
>>> import package1.spam
# trying package1/spam.py
# package1/spam.pyc matches package1/spam.py
import pythonware.spam # precompiled from package1/spam.pyc
# trying package1/os.py
# package1/os.pyc matches package1/os.py
import pythonware.os # precompiled from package1/os.pyc

(note the last line)

> I hope that "import os" will not load package1/os.py, will it?

depends on who's calling it.

</F>



From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Mon Nov 12 18:20:33 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Mon, 12 Nov 2001 23:50:33 +0530
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <15343.63951.137972.487852@slothrop.digicool.com>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
Message-ID: <15344.4849.71643.868791@monster.linux.in>

>>>>> "JH" == Jeremy Hylton <jeremy@zope.com> writes:

>>>>> "PR" == Prabhu Ramachandran <prabhu@aero.iitm.ernet.in> writes:
    PR> Ummm doing an 'import os' will import the package1/os.py and
    PR> *not* the standard one.  This will happen even though os.py
    PR> was imported earlier by site.py.  This is what Gordon was
    PR> objecting to in the first place and why he proposes using
    PR> rimport, rrimport etc. to make things more explicit.

    JH> Of course, you can use the existing mechanism to do this:
    JH> 'from package1 import os'.  The use of an explicit name seems
    JH> like the clearest route when you have a package-local module
    JH> that shadows a top-level module -- no need to understand
    JH> details of relative imports, no question about what is
    JH> intended by the code.

    JH> I haven't followed this thread closely.  Is there some reason
    JH> that explicit names in imports is not sufficient?

Yes indeed there is.  I've already explained my reasons twice.  Eric
also explained why this was important for Scipy.

Anyway, in short, its a big pain re-nesting packages.  Also for any
package that has a deep enough structure its a real pain accessing
packages.  

from pkg import subpkg is also not the best way to do imports. I
personally prefer import pkg.subpkg and I believe this is the
recommended way of doing imports.

prabhu


From jeremy@alum.mit.edu  Mon Nov 12 18:35:47 2001
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Mon, 12 Nov 2001 13:35:47 -0500 (EST)
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <15344.4849.71643.868791@monster.linux.in>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
Message-ID: <15344.5763.557118.787529@walden.zope.com>

>>>>> "PR" == Prabhu Ramachandran <prabhu@aero.iitm.ernet.in> writes:
>>>>> "JH" == Jeremy Hylton <jeremy@zope.com> writes:

  JH> I haven't followed this thread closely.  Is there some reason
  JH> that explicit names in imports is not sufficient?

  PR> Yes indeed there is.  I've already explained my reasons twice.
  PR> Eric also explained why this was important for Scipy.

I've gone back through the messages on python-dev, but don't seem a
clear summary of the issues that lead to your proposed change.  The
best I can come up with are: 1) packages that are re-nested are a pain
and 2) complex package structures also cause problems.  Eric has a
specific set of issues with SciPy that involve packages that are
developed and used externally but also included in SciPy.

I have had a hard time trying to figure out precisely what the
problems are.

  PR> Anyway, in short, its a big pain re-nesting packages.  Also for
  PR> any package that has a deep enough structure its a real pain
  PR> accessing packages.

What does "re-nesting" mean?  It get the impression you mean
putting one package inside another after it was developed and pacakged
for use as a top-level package.  If so, it doesn't seem like a problem
that occurs that often, right?  I'd be hesitant to add features to the
import mechanism to cater to an infrequent case.

I'd rather see the imports be explicit "import root.a.b.c" than
"import b.c".  Then re-nesting requires all the import statements to
be edited.  It's more typing and might even require a simple script to
do search-and-replace, but it doesn't sound like a prohibitive
burden. 

I expect there is more to the issue than just wanting to avoid some
extra typing.  A short PEP that describes the specific problems being
solved and discussing alternatives would help.

  PR> from pkg import subpkg is also not the best way to do imports. I
  PR> personally prefer import pkg.subpkg and I believe this is the
  PR> recommended way of doing imports.

Why do you think this is the recommended way of doing imports?  I use
both in my code and haven't been able to come up with a clear
rationale for doing one or the other.  The from ... import form seems
useful when the name of the package/module is long or when it's only
one or two names I'm using.

Jeremy



From gmcm@hypernet.com  Mon Nov 12 22:17:19 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Mon, 12 Nov 2001 17:17:19 -0500
Subject: [Import-sig] Re: Proposal for a modified import  mechanism.
In-Reply-To: <15344.5763.557118.787529@walden.zope.com>
References: <15344.4849.71643.868791@monster.linux.in>
Message-ID: <3BF0041F.517.82A5AD5B@localhost>

Jeremy wrote:

[relative and recursively-relative imports]

> I'd rather see the imports be explicit "import root.a.b.c" than
> "import b.c".  Then re-nesting requires all the import statements
> to be edited.  It's more typing and might even require a simple
> script to do search-and-replace, but it doesn't sound like a
> prohibitive burden. 

As a (minor) data point, if "b.c" is resolved as a relative import, 
it will be faster than the absolute form ("import a.b.c"). 

Having re-arranged a number of packages, I have some 
sympathy for Prabhu's complaint. OTOH, this is a feature 
which only helps package authors (not package users, who 
are likely to have a somewhat harder time finding their way 
around the package).

[Prabhu]
>   PR> from pkg import subpkg is also not the best way to do
>   imports. I PR> personally prefer import pkg.subpkg and I
>   believe this is the PR> recommended way of doing imports.

[Jeremy]
> Why do you think this is the recommended way of doing imports?  I
> use both in my code and haven't been able to come up with a clear
> rationale for doing one or the other.  The from ... import form
> seems useful when the name of the package/module is long or when
> it's only one or two names I'm using.

When you have circular imports, someone must use the 
"import a.b.c" form. This can show up in some surprising 
ways, especially when the package in question desparately 
needs re-arranging <wink>.

- Gordon


From barry@zope.com  Mon Nov 12 22:32:27 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 12 Nov 2001 17:32:27 -0500
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
Message-ID: <15344.19963.181812.444906@anthem.wooz.org>

>>>>> "JH" == Jeremy Hylton <jeremy@zope.com> writes:

    JH> I'd rather see the imports be explicit "import root.a.b.c"
    JH> than "import b.c".  Then re-nesting requires all the import
    JH> statements to be edited.  It's more typing and might even
    JH> require a simple script to do search-and-replace, but it
    JH> doesn't sound like a prohibitive burden.

Note that applications can achieve the same thing without editing code
by doing sys.path manipulations.

    JH> I expect there is more to the issue than just wanting to avoid
    JH> some extra typing.  A short PEP that describes the specific
    JH> problems being solved and discussing alternatives would help.

Indeed.  We've been here before (perhaps, several "befores" :).  Every
time this comes up I get the feeling like there are easy ways to
accomplish what you want if you think of the problem differently, or
I'm missing something fundamental about the problem, and/or the
problem has never been specified identified, or people are trying to
solve too many problems at once.

Are the needs of application authors different than library authors?

-Barry


From michel@zope.com  Mon Nov 12 23:10:18 2001
From: michel@zope.com (Michel Pelletier)
Date: Mon, 12 Nov 2001 15:10:18 -0800
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com> <15344.19963.181812.444906@anthem.wooz.org>
Message-ID: <3BF056DA.74724E5C@zope.com>

"Barry A. Warsaw" wrote:
> 
> Are the needs of application authors different than library authors?

This is the best place to start, almost everyone on this list plays both
roles to one degree or another.  I've read Prabhu's emails and I
understand his problem.  He's explained it a couple of times, but in
general users and their needs have been unclear which I think spawned
most of this discussion.

I'm actually sort of interested in more about the idea of 'looking up'
for packages that Prabhu mentioned and think it could be very useful. 
In Zope we call this "acquisition" and we use this pattern many, many
times to override general site  policies and objects with more specific
ones the farther "down" you go in the object heirarchy.  This not only
gives us a nice customization model, but it also gives us a nice
delegation model, ie, those responsible on high (that's all of you) can
dictate what is and is not the standard library "policy" and users below
you (that's me and all the other lusers) can specificly override that
mandate at a lower level without interfering with other users.

-Michel


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Tue Nov 13 05:08:34 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Tue, 13 Nov 2001 10:38:34 +0530
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <15344.5763.557118.787529@walden.zope.com>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
Message-ID: <15344.43730.821724.769885@monster.linux.in>

>>>>> "JH" == Jeremy Hylton <jeremy@zope.com> writes:

>>>>> "PR" == Prabhu Ramachandran <prabhu@aero.iitm.ernet.in> writes:

    JH> I haven't followed this thread closely.  Is there some reason
    JH> that explicit names in imports is not sufficient?

    PR> Yes indeed there is.  I've already explained my reasons twice.
    PR> Eric also explained why this was important for Scipy.

[snip]

    JH> I have had a hard time trying to figure out precisely what the
    JH> problems are.

I think you got it mostly right. Let me try to elaborate on it.

  (1) Re-nesting a package is a pain.  What I mean by re-nesting is
  that say I have a package, A, that is separate (and that has its own
  sub packages) and now I want it as part of another package, B.  Lets
  further suppose that the module which re-nests the package, B,
  tracks the development of A and keeps their copy updated.  In this
  case A is developed as a standalone package and B adds something to
  it that A cannot/refuses to use.  With the current approach B would
  be forced to modify A every time A changes in some significant way
  simply because A was re-nested.  Yes, this is contrived but such
  situations do occur. 

  To make things clearer.  My main objection is that the name of a
  package when one imports it depends on its parent packages name.
  This is IMHO absurd.

       foo/
	   sub/
	   sub1/

  From sub1 if you had to import anything from sub you'd have to do
  import foo.sub.module.  So if foo is now part of something else -
  you have to change all references to foo.

  (2) If you have a complex package with more than 2-3 nested sub
  directories it becomes a huge pain to use clean import statements
  and not have to type long lines just to get to different modules.

  (3) If you argue that import must always do only absolute imports
  then why are sibling packages allowed?  i.e. if there are two
  modules in the same directory Python currently allows one to import
  them with a relative name rather than an absolute foo.sub.pkg kind
  of name.  If this is valid, then its natural to expect that
  searching also be done in the local package structure.

  (4) Yes, its possible re-factoring code but sometimes this can be a
  pain if you have a CVS tree and you want to re-organize your package
  structure.  Bernhard Herzog posted a solution for my specific
  problem, so that really is not the issue.  In my case the current
  cvsroot for my sources is this:

  cvs.sourceforge.net/cgi-bin/viewcvs.cgi/mayavi/mayavi/

  and Bernhard's solution would create a directory structure like so:

  cvs.sourceforge.net/cgi-bin/viewcvs.cgi/mayavi/mayavi/mayavi/

  Which is pretty crazy if you ask me, its bad enough as it is. :)
  this will solve my particular problem but is dirty.

    JH> What does "re-nesting" mean?  It get the impression you mean
    JH> putting one package inside another after it was developed and
    JH> pacakged for use as a top-level package.  If so, it doesn't
    JH> seem like a problem that occurs that often, right?  I'd be
    JH> hesitant to add features to the import mechanism to cater to
    JH> an infrequent case.

I've had about 4 others mailing me about their related problems.  So I
wouldn't classify this as a rare problem that can be safely ignored.

    JH> I'd rather see the imports be explicit "import root.a.b.c"
    JH> than "import b.c".  Then re-nesting requires all the import
    JH> statements to be edited.  It's more typing and might even
    JH> require a simple script to do search-and-replace, but it
    JH> doesn't sound like a prohibitive burden.

It all depends.  I think Eric explained his position pretty clearly.
I'm convinced that Python's import structure needs improvement.

    JH> I expect there is more to the issue than just wanting to avoid
    JH> some extra typing.  A short PEP that describes the specific
    JH> problems being solved and discussing alternatives would help.

Well, its all about convenience anyway - if not we'd all be talking to
computers in binary! <wink> Why do we need 'high-level' programming
languages?  Yes, I'm digressing into te philosophy of computing but I
dont think syntactic sugar is something to be ignored because its
silly.

    PR> from pkg import subpkg is also not the best way to do
    PR> imports. I personally prefer import pkg.subpkg and I believe
    PR> this is the recommended way of doing imports.

    JH> Why do you think this is the recommended way of doing imports?
    JH> I use both in my code and haven't been able to come up with a
    JH> clear rationale for doing one or the other.  The from
    JH> ... import form seems useful when the name of the
    JH> package/module is long or when it's only one or two names I'm
    JH> using.

Well, the Python howto explains it much better than I could hope to:

    http://py-howto.sourceforge.net/doanddont/node8.html

Since re-loading packages is important for me, I prefer using plain
imports.

prabhu


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Tue Nov 13 05:20:03 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Tue, 13 Nov 2001 10:50:03 +0530
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <15344.19963.181812.444906@anthem.wooz.org>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
 <15344.19963.181812.444906@anthem.wooz.org>
Message-ID: <15344.44419.359241.47222@monster.linux.in>

>>>>> "BAW" == Barry A Warsaw <barry@zope.com> writes:

    JH> I'd rather see the imports be explicit "import root.a.b.c"
    JH> than "import b.c".  Then re-nesting requires all the import
    JH> statements to be edited.  It's more typing and might even
    JH> require a simple script to do search-and-replace, but it
    JH> doesn't sound like a prohibitive burden.

    BAW> Note that applications can achieve the same thing without
    BAW> editing code by doing sys.path manipulations.

Its not the application that I'm concerned about - an application is
typically a single/few file(s) and editing them to suit things is
certainly not an issue.  But editing 100 files inside a package each
time the parent changes is nuts.

There is another way to get around this by manipulating __path__
inside a sub package.  But this leads to the same module being
imported several times.  This is what I use currently and its evil. :(

    JH> I expect there is more to the issue than just wanting to avoid
    JH> some extra typing.  A short PEP that describes the specific
    JH> problems being solved and discussing alternatives would help.

    BAW> Indeed.  We've been here before (perhaps, several "befores"
    BAW> :).  Every time this comes up I get the feeling like there
    BAW> are easy ways to accomplish what you want if you think of the

So do I need to write a PEP?  Is there some special formality/format I
need to keep in mind?

    BAW> problem differently, or I'm missing something fundamental
    BAW> about the problem, and/or the problem has never been
    BAW> specified identified, or people are trying to solve too many
    BAW> problems at once.

    BAW> Are the needs of application authors different than library
    BAW> authors?

I would think so.

prabhu


From barry@zope.com  Tue Nov 13 06:01:21 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Tue, 13 Nov 2001 01:01:21 -0500
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
 <15344.43730.821724.769885@monster.linux.in>
Message-ID: <15344.46897.223437.983547@anthem.wooz.org>

>>>>> "PR" == Prabhu Ramachandran <prabhu@aero.iitm.ernet.in> writes:

    PR>   (1) Re-nesting a package is a pain.  What I mean by
    PR> re-nesting is that say I have a package, A, that is separate
    PR> (and that has its own sub packages) and now I want it as part
    PR> of another package, B.

Why would you want to do that?  Why not just keep them separate
top-level packages that cooperate?  Or export A's names in B's
modules?  I think distutils helps out here because it's now easy to
install A in a way that B could just use, or add to.

FWIW, we knit things together as well, e.g. with StandaloneZODB.  It's
got a bunch of top-level packages that are treated as a single entity
via a figment of CVS's imagination.  So what if it installs a bunch of
separate top-level package names that aren't all treed under a single
package?
    
    PR> Lets further suppose that the module which re-nests the
    PR> package, B, tracks the development of A and keeps their copy
    PR> updated.

Okay.
    
    PR> In this case A is developed as a standalone package and B adds
    PR> something to it that A cannot/refuses to use.

Okay.
    
    PR> With the current approach B would be forced to modify A every
    PR> time A changes in some significant way simply because A was
    PR> re-nested.  Yes, this is contrived but such situations do
    PR> occur.

Why does B have to add packages to A's namespace?  Why can't the B
author simply use distutils to ensure that vanilla A is installed,
import the bits and pieces of A that you want to expose, overriding
what you want to change, and export an interface through B that
clients can use instead of A?  I.e. through the use of "from foo
import bar" and "from foo import bar as baz", you can present whatever
public interface you want, through B's namespace, and mimic as much or
as little of A's as you want.

    PR> Its not the application that I'm concerned about - an
    PR> application is typically a single/few file(s) and editing them
    PR> to suit things is certainly not an issue.

Well, not /all/ applications!
    
    JH> I expect there is more to the issue than just wanting to avoid
    JH> some extra typing.  A short PEP that describes the specific
    JH> problems being solved and discussing alternatives would help.

    BAW> Indeed.  We've been here before (perhaps, several "befores"
    BAW> :).  Every time this comes up I get the feeling like there
    BAW> are easy ways to accomplish what you want if you think of the

    PR> So do I need to write a PEP?  Is there some special
    PR> formality/format I need to keep in mind?

PEP 1 and PEP 9 are your guidelines to proper PEP form and procedure.

    BAW> Are the needs of application authors different than library
    BAW> authors?

    PR> I would think so.

That would be good to outline in your PEP then <wink>.

-Barry


From fredrik@pythonware.com  Tue Nov 13 06:44:41 2001
From: fredrik@pythonware.com (Fredrik Lundh)
Date: Tue, 13 Nov 2001 07:44:41 +0100
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import  mechanism.
References: <3BED5AD8.24350.78407262@localhost><3BEFE249.371EA5FD@interet.com><15343.61552.414932.44100@monster.linux.in><15343.63951.137972.487852@slothrop.digicool.com><15344.4849.71643.868791@monster.linux.in><15344.5763.557118.787529@walden.zope.com><15344.19963.181812.444906@anthem.wooz.org> <15344.44419.359241.47222@monster.linux.in>
Message-ID: <009501c16c0e$acd6e130$ced241d5@hagrid>

Prabhu Ramachandran wrote:
> Its not the application that I'm concerned about - an application is
> typically a single/few file(s) and editing them to suit things is
> certainly not an issue. But editing 100 files inside a package each
> time the parent changes is nuts.

I'd recommend that you don't put the above in your PEP.

changing the language to cater to package owners who
have more files than users (and don't appear to give a
damn about the few users they have) isn't just nuts, it's
total madness.

if you do this a lot, write a 20-line script to do it for you.

</F>



From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Tue Nov 13 07:39:31 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Tue, 13 Nov 2001 13:09:31 +0530
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import  mechanism.
In-Reply-To: <009501c16c0e$acd6e130$ced241d5@hagrid>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
 <15344.19963.181812.444906@anthem.wooz.org>
 <15344.44419.359241.47222@monster.linux.in>
 <009501c16c0e$acd6e130$ced241d5@hagrid>
Message-ID: <15344.52787.673648.792323@monster.linux.in>

>>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes:

    >> Its not the application that I'm concerned about - an
    >> application is typically a single/few file(s) and editing them
    >> to suit things is certainly not an issue. But editing 100 files
    >> inside a package each time the parent changes is nuts.

    FL> I'd recommend that you don't put the above in your PEP.

    FL> changing the language to cater to package owners who have more
    FL> files than users (and don't appear to give a damn about the
    FL> few users they have) isn't just nuts, it's total madness.

Ouch!  Yes, I might be mad but I guess you are taking my statement
totally out of context.

  (1) I was talking of _my_ particular application where I have a
  script that imports the package depending on whether the application
  is installed or frozen.  I wasn't talking of every Python
  application!

  (2) How does modifying the behaviour of import or better still
  adding a couple of new keywords (aimport, rimport etc.) that do
  things a little differently affect the all the users of Python?
  What we proposed will only affect the package in question and not
  every user in the system.  AFAIK, what was proposed would also not
  break backward compatibility in any way.

  (3) As to me not giving a 'damn' about the 'few users' that I may
  have, I think that is fairly untrue and all I have to say is that I
  wouldn't have considered posting to c.l.p* with this proposal if it
  wouldn't be of use to someone else.  It would have been very easy
  for me to have solved my problem and left things at that.

    FL> if you do this a lot, write a 20-line script to do it for you.

I'm afraid I already did better than that. <wink> I modified knee.py
to do exactly what I wanted and it does work.  Also, even without such
a hack I can get my package running, thanks.  However, the point is
that Gordon, Eric, some others and myself think that it would be nice
if the import mechanism were a little nicer to package developers.
The problem is not something that I am personally desperate for.  I'm
just trying to see if Python's import mechanism can be improved (if
necessary).  Flaming me is pointless. :)

prabhu


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Tue Nov 13 08:38:31 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Tue, 13 Nov 2001 14:08:31 +0530
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <3BF056DA.74724E5C@zope.com>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
 <15344.19963.181812.444906@anthem.wooz.org>
 <3BF056DA.74724E5C@zope.com>
Message-ID: <15344.56327.337652.24737@monster.linux.in>

>>>>> "MP" == Michel Pelletier <michel@zope.com> writes:

    MP> I'm actually sort of interested in more about the idea of
    MP> 'looking up' for packages that Prabhu mentioned and think it
    MP> could be very useful.  In Zope we call this "acquisition" and
    MP> we use this pattern many, many times to override general site
    MP> policies and objects with more specific ones the farther
    MP> "down" you go in the object heirarchy.  This not only gives us
    MP> a nice customization model, but it also gives us a nice
    MP> delegation model, ie, those responsible on high (that's all of
    MP> you) can dictate what is and is not the standard library
    MP> "policy" and users below you (that's me and all the other
    MP> lusers) can specificly override that mandate at a lower level
    MP> without interfering with other users.

Yes, this is a nicer and probably more convincing use/argument than I
had proposed.

prabhu


From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Tue Nov 13 08:56:45 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Tue, 13 Nov 2001 14:26:45 +0530
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <15344.46897.223437.983547@anthem.wooz.org>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
 <15344.43730.821724.769885@monster.linux.in>
 <15344.46897.223437.983547@anthem.wooz.org>
Message-ID: <15344.57421.641192.998739@monster.linux.in>

>>>>> "BAW" == Barry A Warsaw <barry@zope.com> writes:

>>>>> "PR" == Prabhu Ramachandran <prabhu@aero.iitm.ernet.in> writes:

    PR> (1) Re-nesting a package is a pain.  What I mean by re-nesting
[Re-nesting packages contrived example]

    BAW> Why would you want to do that?  Why not just keep them
    BAW> separate top-level packages that cooperate?  Or export A's
    BAW> names in B's modules?  I think distutils helps out here
    BAW> because it's now easy to install A in a way that B could just
    BAW> use, or add to.

Umm, that was a contrived example so might not be very sensible.  For
a more realistic one I think I'll pass the question on to Eric.  I
think Eric did mention his difficulty with SciPy here:

http://mail.python.org/pipermail/python-list/2001-November/071794.html

<snip>

    BAW> Why does B have to add packages to A's namespace?  Why can't
    BAW> the B author simply use distutils to ensure that vanilla A is
    BAW> installed, import the bits and pieces of A that you want to
    BAW> expose, overriding what you want to change, and export an
    BAW> interface through B that clients can use instead of A?
    BAW> I.e. through the use of "from foo import bar" and "from foo
    BAW> import bar as baz", you can present whatever public interface
    BAW> you want, through B's namespace, and mimic as much or as
    BAW> little of A's as you want.

Ture, its possible to do things and work around situations with the
current scheme.  I guess I need to come up with something that
definitively proves my point.  Will think about it.  Maybe Gordon has
a better/more convincing argument?  I think Michel Pelletier also had
a different point of view on this.

    PR> Its not the application that I'm concerned about - an
    PR> application is typically a single/few file(s) and editing them
    PR> to suit things is certainly not an issue.

    BAW> Well, not /all/ applications!

Indeed.  I guess I caused confusion here.  I was talking of my
particular application where I ran into problems with re-nesting and
too much typing I was referring to that.  I certainly don't intend
changing every single application when there is no need for that.
    
    PR> So do I need to write a PEP?  Is there some special
    PR> formality/format I need to keep in mind?

    BAW> PEP 1 and PEP 9 are your guidelines to proper PEP form and
    BAW> procedure.

Thanks.  Will look at them.

prabhu


From mal@lemburg.com  Tue Nov 13 09:23:58 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Tue, 13 Nov 2001 10:23:58 +0100
Subject: [Import-sig] Re: Proposal for a modified import mechanism.
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com> <15344.19963.181812.444906@anthem.wooz.org> <3BF056DA.74724E5C@zope.com>
Message-ID: <3BF0E6AE.89CD9B9E@lemburg.com>

Michel Pelletier wrote:
> 
> "Barry A. Warsaw" wrote:
> >
> > Are the needs of application authors different than library authors?
> 
> This is the best place to start, almost everyone on this list plays both
> roles to one degree or another.  I've read Prabhu's emails and I
> understand his problem.  He's explained it a couple of times, but in
> general users and their needs have been unclear which I think spawned
> most of this discussion.

Well, I've been there and done that. I also proposed an (implicit) 
relative import scheme some years ago, because I thought it might be
worthwhile having to sort out the problems with naming conflicts,
local installs etc.

In the meantime I found that I can work well without
relative imports by being careful about the import statements
I use in software using the packages and in modules of the packages
themselves.

IMHO, it's a much better strategy to think about defining a
proper absolute package name structure and using it throughout
your code. Given such a structure, the need for relative imports
which are hard to debug and can cause a whole variety of
problems (e.g. pickled objects still carry the absolute package
name, modules can be overridden without the user realizing this,
etc.).
 
> I'm actually sort of interested in more about the idea of 'looking up'
> for packages that Prabhu mentioned and think it could be very useful.
> In Zope we call this "acquisition" and we use this pattern many, many
> times to override general site  policies and objects with more specific
> ones the farther "down" you go in the object heirarchy.  This not only
> gives us a nice customization model, but it also gives us a nice
> delegation model, ie, those responsible on high (that's all of you) can
> dictate what is and is not the standard library "policy" and users below
> you (that's me and all the other lusers) can specificly override that
> mandate at a lower level without interfering with other users.

Then you probably also know that debugging an acqusition scenario
can become a nightmare, simple due to the fact that each and
every attribute has to be tracked to where it's defined in the
acquisition hierarchy in order to tell where the problem originated.

Acquisition is very nice if it works, but way to complicated for
causal users manage in case something goes wrong. The same
goes for relative imports which are essentially the same thing
w/r to module lookups in packages.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Consulting & Company:                           http://www.egenix.com/
Python Software:                        http://www.lemburg.com/python/




From barry@zope.com  Tue Nov 13 14:59:14 2001
From: barry@zope.com (Barry A. Warsaw)
Date: Tue, 13 Nov 2001 09:59:14 -0500
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
 <15344.43730.821724.769885@monster.linux.in>
 <15344.46897.223437.983547@anthem.wooz.org>
 <15344.57421.641192.998739@monster.linux.in>
Message-ID: <15345.13634.801472.695445@anthem.wooz.org>

>>>>> "PR" == Prabhu Ramachandran <prabhu@aero.iitm.ernet.in> writes:

    PR> Umm, that was a contrived example so might not be very
    PR> sensible.  For a more realistic one I think I'll pass the
    PR> question on to Eric.  I think Eric did mention his difficulty
    PR> with SciPy here:

    PR> http://mail.python.org/pipermail/python-list/2001-November/071794.html

I'm not wholly unsympathetic to the problem, but I'm trying to give
some pushback because of just the reason Michel gives.  Even though I
can think of my own cool uses of an "acquisitional import", I think
Python should be really careful here.  One of the deep problems with
implicit acquisition is that you often don't know where something
really comes from.  I'm worried that building this into the import
mechanism will make it harder to figure out where something comes
from.  Explicit is better than implicit.

But having said all that, it's clear to me that we've mined this vein
enough on the list.  It's PEP time!

-Barry


From jeremy@zope.com  Tue Nov 13 17:08:33 2001
From: jeremy@zope.com (Jeremy Hylton)
Date: Tue, 13 Nov 2001 12:08:33 -0500 (EST)
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <15344.43730.821724.769885@monster.linux.in>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
 <15344.43730.821724.769885@monster.linux.in>
Message-ID: <15345.21393.28230.859650@slothrop.digicool.com>

>>>>> "PR" == Prabhu Ramachandran <prabhu@aero.iitm.ernet.in> writes:

  PR> (1) Re-nesting a package is a pain.  What I mean by re-nesting
  PR>       is
  PR>   that say I have a package, A, that is separate (and that has
  PR>   its own sub packages) and now I want it as part of another
  PR>   package, B.  Lets further suppose that the module which
  PR>   re-nests the package, B, tracks the development of A and keeps
  PR>   their copy updated.  In this case A is developed as a
  PR>   standalone package and B adds something to it that A
  PR>   cannot/refuses to use.  With the current approach B would be
  PR>   forced to modify A every time A changes in some significant
  PR>   way simply because A was re-nested.  Yes, this is contrived
  PR>   but such situations do occur.

I think we need to analyze the requirements here a lot more
carefully.  The specific question is what the packaging requirements
are for A and B.  I would like to find a solution for the problems
that lead to re-nesting, rather than trying to figure out how to make
re-nesting easier.

Jeremy



From Prabhu Ramachandran <prabhu@cyberwaveindia.com>  Tue Nov 13 18:07:24 2001
From: Prabhu Ramachandran <prabhu@cyberwaveindia.com> (Prabhu Ramachandran)
Date: Tue, 13 Nov 2001 23:37:24 +0530
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
In-Reply-To: <15345.13634.801472.695445@anthem.wooz.org>
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
 <15344.43730.821724.769885@monster.linux.in>
 <15344.46897.223437.983547@anthem.wooz.org>
 <15344.57421.641192.998739@monster.linux.in>
 <15345.13634.801472.695445@anthem.wooz.org>
Message-ID: <15345.24924.396195.958560@monster.linux.in>

>>>>> "BAW" == Barry A Warsaw <barry@zope.com> writes:

[on the suggested package import improvements]

    BAW> I'm not wholly unsympathetic to the problem, but I'm trying
    BAW> to give some pushback because of just the reason Michel
    BAW> gives.  Even though I can think of my own cool uses of an
    BAW> "acquisitional import", I think Python should be really
    BAW> careful here.  One of the deep problems with implicit
    BAW> acquisition is that you often don't know where something
    BAW> really comes from.  I'm worried that building this into the
    BAW> import mechanism will make it harder to figure out where
    BAW> something comes from.  Explicit is better than implicit.

    BAW> But having said all that, it's clear to me that we've mined
    BAW> this vein enough on the list.  It's PEP time!

Great!  I'm very happy and thankful that you folks have been so
patient with me and are listening to this stuff.  

I have a few other things to wind up this week, so I'll try starting
to write the PEP this weekend.  I'll first run it by Eric and maybe
pester Gordon a bit before I pass it on to the experts.

Thanks again!

prabhu


From jim@interet.com  Thu Nov 15 14:27:18 2001
From: jim@interet.com (James C. Ahlstrom)
Date: Thu, 15 Nov 2001 09:27:18 -0500
Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import
 mechanism.
References: <3BED5AD8.24350.78407262@localhost>
 <3BEFE249.371EA5FD@interet.com>
 <15343.61552.414932.44100@monster.linux.in>
 <15343.63951.137972.487852@slothrop.digicool.com>
 <15344.4849.71643.868791@monster.linux.in>
 <15344.5763.557118.787529@walden.zope.com>
 <15344.43730.821724.769885@monster.linux.in>
 <15344.46897.223437.983547@anthem.wooz.org>
 <15344.57421.641192.998739@monster.linux.in>
 <15345.13634.801472.695445@anthem.wooz.org> <15345.24924.396195.958560@monster.linux.in>
Message-ID: <3BF3D0C6.1892D49D@interet.com>

Prabhu Ramachandran wrote:

> I have a few other things to wind up this week, so I'll try starting
> to write the PEP this weekend.  I'll first run it by Eric and maybe
> pester Gordon a bit before I pass it on to the experts.

Please note that I am currently coding PEP 273

    http://python.sourceforge.net/peps/pep-0273.html

which also deals with imports.  My changes are mostly
confined to import.c.  If you plan to change import.c
we need to coordinate.  My changes have nothing to do
with the way packages are imported.  The existing
mechanism is extended to zip archives.

Jim Ahlstrom


From eo1426@ezlink.com.tw  Tue Nov 27 12:09:46 2001
From: eo1426@ezlink.com.tw (¾_Ãv)
Date: Tue, 27 Nov 2001 20:09:46 +0800
Subject: [Import-sig] TOP Computer Tables, Office Chair, Chair Components Manufacturer
Message-ID: <E168hbO-0004sn-00@mail.python.org>

<html>
<head>
<title>Everorient Enterprise Co. Ltd</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>

<body bgcolor="#FFFFFF" text="#000000" background="http://www.ezlink.com.tw/e-mail/everor/bg.gif" marginheight="0" topmargin="0" marginwidth="0" leftmargin="0">
<div align="center"><object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=5,0,0,0" width="673" height="100">
    <param name=movie value="http://www.ezlink.com.tw/e-mail/everor/title.swf">
    <param name=quality value=high>
    <embed src="title.swf" quality=high pluginspage="http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version=ShockwaveFlash" type="application/x-shockwave-flash" width="673" height="100">
    </embed></object><br>
  <img src="http://www.ezlink.com.tw/e-mail/everor/hor.gif" width="750" height="12"><br>
  <br>
</div>
  
<table width="90%" border="0" cellspacing="0" cellpadding="0" align="center">
  <tr> 
    <td><b>Everorient Enterprise Co. Ltd ,</b> founded in 1984, is the pioneer 
      and leader of computer tables, office chair and chair components in Taiwan. 
      Our products are marketing worldwide with recognition as quality supplier 
      from buyers as they are certified and conformed to <i><b>SGS</b></i> and 
      <i><b>ISO9002</b></i> quality standard. Our company products are mostly 
      exported to Europe, South America, Canada and USA. We always insist our 
      company's principle is superior quality, reasonable price, prompt delivery 
      to our customers, so we surely you can be pretty facile to succeed in business 
      among the competitive market.<br>
      Enclosed please find some photos of our products for your reference. Please 
      try to find any items you will be interested in and if you have any further 
      questions, please feel free to contact us. We are pleasured and glad to 
      serve you. 
      <p>Thanks for your kind attention and looking forward to hearing from you 
        soon.<br>
      </p>
    </td>
    <td valign="top"> 
      <table cellpadding="0" cellspacing="0" border="0" width="95%">
        <tr> 
          <td rowspan="3"><img src="http://www.ezlink.com.tw/e-mail/everor/page_0.jpg" width="7" height="167"></td>
          <td><img src="http://www.ezlink.com.tw/e-mail/everor/page_1.jpg" width="90" height="8"></td>
          <td rowspan="3"><img src="http://www.ezlink.com.tw/e-mail/everor/page_2.jpg" width="7" height="167"></td>
        </tr>
        <tr> 
          <td><object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=5,0,0,0" width="90" height="150">
              <param name=movie value="http://www.ezlink.com.tw/e-mail/everor/list.swf">
              <param name=quality value=high>
              <embed src="http://www.ezlink.com.tw/e-mail/everor/list.swf" quality=high pluginspage="http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version=ShockwaveFlash" type="application/x-shockwave-flash" width="90" height="150">
              </embed></object></td>
        </tr>
        <tr> 
          <td><img src="http://www.ezlink.com.tw/e-mail/everor/page_4.jpg" width="90" height="9"></td>
        </tr>
      </table>
      <div align="right"></div>
      <div align="right"></div>
    </td>
  </tr>
</table>
<br>
<div align="center"><img src="http://www.ezlink.com.tw/e-mail/everor/hor.gif" width="750" height="12"><br>
  <br>
</div>
<table border="1" cellspacing="0" cellpadding="0" align="center">
  <tr> 
    <td rowspan="2"><img src="http://www.ezlink.com.tw/e-mail/everor/do01.jpg" width="128" height="199"></td>
    <td><object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=5,0,0,0" width="412" height="100">
        <param name=movie value="http://www.ezlink.com.tw/e-mail/everor/list2.swf">
        <param name=quality value=high>
        <embed src="http://www.ezlink.com.tw/e-mail/everor/list2.swf" quality=high pluginspage="http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version=ShockwaveFlash" type="application/x-shockwave-flash" width="412" height="100">
        </embed></object></td>
    <td rowspan="2"><img src="http://www.ezlink.com.tw/e-mail/everor/do02.jpg" width="127" height="199"></td>
  </tr>
  <tr> 
    <td>
      <table border="1" cellspacing="0" cellpadding="0">
        <tr> 
          <td><img src="http://www.ezlink.com.tw/e-mail/everor/do03.jpg" width="80" height="93"></td>
          <td><img src="http://www.ezlink.com.tw/e-mail/everor/do04.jpg" width="80" height="93"></td>
          <td><img src="http://www.ezlink.com.tw/e-mail/everor/do05.jpg" width="80" height="93"></td>
          <td><img src="http://www.ezlink.com.tw/e-mail/everor/do06.jpg" width="80" height="93"></td>
          <td><img src="http://www.ezlink.com.tw/e-mail/everor/do07.jpg" width="80" height="93"></td>
        </tr>
      </table>
    </td>
  </tr>
</table>
<br>
<table width="90%" border="0" cellpadding="0" cellspacing="0" align="center">
  <tr>
    <td>
      <p><b>Best regards<br>
        International Trade Manger:<br>
        <font face="ParkAvenue BT">David Liao</font><br>
        Everorient Enterprise Co., Ltd.</b></p>
      <p><b>CONTACT US:<br>
        Everorient Enterprise Co., Ltd.<br>
        ADDRESS: P.O.Box 3-36, Shulin city, Taipei Hsien, Taiwan<br>
        Tel : <i>886-2-8684-2955</i><br>
        Fax :<i> 886-2-8686-8118</i><br>
        E-mail:<a href="mailto:ever336@ms2.hinet.net"><i>ever336@ms2.hinet.net</i></a><br>
        Homepage:<a href="http://www.everorient.com.tw"><i>http://www.everorient.com.tw</i></a></b><a href="http://www.everorient.com.tw"><br>
        </a></p>
      <p align="center"></p>
</td>
  </tr>
</table>
</body>
</html>