From jan.kanis at phil.uu.nl  Thu Feb  1 01:02:45 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Thu, 01 Feb 2007 01:02:45 +0100
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45C04D91.8030906@onego.ru>
References: <20070130142937.5A38.JCARLSON@uci.edu> <op.tmzsxtjhd64u53@e500>
	<20070130173337.5A4D.JCARLSON@uci.edu> <45C04D91.8030906@onego.ru>
Message-ID: <op.tm1mevbcd64u53@e500>

On Wed, 31 Jan 2007 09:04:33 +0100, Roman Susi <rnd at onego.ru> wrote:

> Josiah Carlson wrote:
>> "Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
>>
>>> On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson <jcarlson at uci.edu>
>>> wrote:
>>>
>>>> Roman Susi <rnd at onego.ru> wrote:
>>>>
>>>>>    def foo(x, y, z, bar=, qux=):
>>>>>        if baz is Missing:
>>>>>            baz = []
>>>>>        #code
>>>>>
>>>>> at least, it doesn't require decorators, is backward compatible
>>>>> (hopefully no grammar conflicts in there), reads as English.
>>>>
>>>> The above with a missing value for a default *is not* backwards
>>>> compatible with previous Pythons.  New syntax is, by definition, not
>>>> backwards compatible.
>>>>
>>>> - Josiah
>>>
>>> As a matter of fact, backward-compatible syntax changes are certainly
>>> possible. (ever wondered how C++ got it's syntax?) Any valid current
>>> python is still going to behave exactly the same if this syntax were  
>>> to be
>>> accepted. Talking about backward compatibility, I think it is safe to
>>> ignore any text files that don't get accepted by the python  
>>> interpreter.
>>> This syntax change would certainly not break any existing production
>>> python code.
>>> (note: the above statements do not entail in any way that I am in  
>>> favour
>>> of this syntax change)
>>
>>
>> Fowards compatible then.  That is to say, writing for the new proposed
>> system will break compatibility with older Pythons.  On the other hand,
>> using currently-available language syntax and semantics is compatible
>> among the entire range of Pythons available today.
>
>> From wikipedia:
>
>
> "... a product is said to be backward compatible (or downward
> compatible) when it is able to take the place of an older product, by
> interoperating with other products that were designed for the older
> product."
>
> "Forward compatibility (sometimes confused with extensibility) is the
> ability of a system to accept input intended for later versions of  
> itself."
>
>
> So, the right term is that Python 3000 will be in the syntactic aspect
> we discuss backward compatible because it will accept old syntax too.
>
>
> Regards,
> Roman
>
>

Forward compatible?? There are exactly zero people on python-dev who care  
about that. What people care about is that a piece of python code written  
at the time of 2.4 will still run the same on python 2.5. That's backward  
compatibility. Nobody cares that a piece of python code using syntax  
constructs that were new in 2.5 won't run on python 2.4. If you want that,  
then don't use the new constructs. New syntax constructs get added to the  
language on just about every major revision, no python major version is  
forward compatible with a higher major version.
Python 3.0 is specifically created to allow the backward incompatible  
changes that people deem nescessary, so neither forward nor backward  
compatibility matter in that respect. (well, except that it shouldn't  
become a superhuman task to convert 2.x code to run on python 3.0)

- Jan


From rrr at ronadam.com  Thu Feb  1 01:18:08 2007
From: rrr at ronadam.com (Ron Adam)
Date: Wed, 31 Jan 2007 18:18:08 -0600
Subject: [Python-ideas] Import and '..', '../..' in serach path.
In-Reply-To: <45C11C19.5030703@ronadam.com>
References: <45BF2D86.2080801@ronadam.com>		<bbaeab100701301132l5623625chb203d47d55e10f0d@mail.gmail.com>		<45BFACDD.5040602@ronadam.com>		<bbaeab100701301421y175d2d04ha4d445489d88203@mail.gmail.com>		<45C0185E.2090408@ronadam.com>	<bbaeab100701311148l2c1d8b41qc1bc3f35b405d857@mail.gmail.com>
	<45C11C19.5030703@ronadam.com>
Message-ID: <45C131C0.30208@ronadam.com>

Ron Adam wrote:

> What is needed is to append the actual root package location to the front os 
> sys.path. If absolute imports are used for intra-package imports, then there is 
> not chance of shadowing the library by doing this.

Correction, it's still possible to shadow the library.  At least in 2.5.  Maybe
after 2.7 it won't.  But it doesn't make anything worse.


A question that may be off topic here but is somewhat related.

How can I tell the difference when running a script with/without:

    from __future__ import absolute_import

It doesn't seem to make any difference on my computer.

Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win
32


Ron





From jcarlson at uci.edu  Thu Feb  1 01:40:08 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Wed, 31 Jan 2007 16:40:08 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <op.tm1mevbcd64u53@e500>
References: <45C04D91.8030906@onego.ru> <op.tm1mevbcd64u53@e500>
Message-ID: <20070131162000.5A70.JCARLSON@uci.edu>


"Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
> Forward compatible?? There are exactly zero people on python-dev who care  
> about that. What people care about is that a piece of python code written  
> at the time of 2.4 will still run the same on python 2.5. That's backward  
> compatibility. Nobody cares that a piece of python code using syntax  
> constructs that were new in 2.5 won't run on python 2.4. If you want that,  
> then don't use the new constructs. New syntax constructs get added to the  
> language on just about every major revision, no python major version is  
> forward compatible with a higher major version.

There are many developers who concern themselves with writing code that
will work on version x and x+1 of Python.  There are also people who
like to use "bleeding edge" features while also using an older Python.
If there weren't, then 'from __future__ import generators' wouldn't have
existed in Python 2.2, 'from __future__ import floor_division' wouldn't
exist in Python 2.3+, 'from __future__ import nested_scopes' wouldn't
have existed in Python 2.1 or 2.2 (I can't remember), etc.

Yes, there is new syntax with each later Python that won't work with
previous Pythons, but with the vast majority of syntax changes that have
come down (including the with statement in 2.5), the semantics of the
new syntax are defined by using the previously existing language.  In
this case, the 'def foo(x=):' has a translation into current Python, but
it turns out that the translation is *exactly what you should be doing
today* if you care about correctness.


> Python 3.0 is specifically created to allow the backward incompatible  
> changes that people deem nescessary, so neither forward nor backward  
> compatibility matter in that respect. (well, except that it shouldn't  
> become a superhuman task to convert 2.x code to run on python 3.0)


The reason I concerned myself with this in regards to the 'def foo(x=)'
proposal is because the ultimate point of this is to make writing code
*less* buggy.  However, code can be *less* buggy today through the
proper application of an if statement or conditional expression.

Yes, Py3k does allow for incompatible syntax.  But the syntax changes
aren't going to be "becase we can", they are going to be "because this
way is better".  Leaving out an argument isn't better (in the case of
the 'def foo(x=):' syntax); it's ambiguous at best, confusing and
worthless at worst.

We can argue all day about writing compatible code, breaking
compatibility, etc., but the fact remains: no developer who has any pull
in python-3000 or python-dev has come out in favor of changing default
argument semantics in this thread.


 - Josiah



From jan.kanis at phil.uu.nl  Thu Feb  1 04:33:09 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Thu, 01 Feb 2007 04:33:09 +0100
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <3cdcefb80701300748y3c92d544g6414bb5438d28305@mail.gmail.com>
References: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
	<op.tmsemwqxd64u53@e500> <20070126212657.5A06.JCARLSON@uci.edu>
	<op.tmyxw9g2d64u53@aidwe2058w.minlnv.agro.nl>
	<3cdcefb80701300748y3c92d544g6414bb5438d28305@mail.gmail.com>
Message-ID: <op.tm1v5jzwd64u53@e500>

(my response is a bit late, I needed some time to come up with a good  
answer to your objections)

On Tue, 30 Jan 2007 16:48:54 +0100, Greg Falcon <veloso at verylowsodium.com>  
wrote:

> On 1/30/07, Jan Kanis <jan.kanis at phil.uu.nl> wrote:
>> On the other hand, are there really any good reasons to choose the  
>> current
>> semantics of evaluation at definition time?
>
> While I sympathize with the programmer that falls for this common
> Python gotcha, and would not have minded if Python's semantics were
> different from the start (though the current behavior is cleaner and
> more consistent), making such a radical change to such a core part of
> the language semantics now is a very bad idea for many reasons.

It would be a py 3.0 change. Other important stuff is going to change as  
well. This part of python is IMO not that much part of the core that it  
can't change at all. Especially since the overwhelming majority of all  
uses of default args have immutable values, so their behaviour isn't going  
to change anyway. (judging by the usage in the std lib.)
Things like list comprehension and generators were a much greater change  
to python, drastically changing the way an idiomatic python program is  
written. They were added in 2.x because they could be implementen backward  
compatible. With python 3.0, backward compatibility isn't so important  
anymore. The whole reason for python 3.0's existance is to fix backward  
incompatible stuff.

>> What I've heard basically
>> boils down to two arguments:
>> - "let's not change anything", i.e. resist change because it is change,
>> which I don't think is a very pythonic argument.
>
> The argument here is not "let's not change anything because it's
> change," but rather "let's not break large amounts of existing code
> without a very good reason."  As has been stated here by others,
> making obsolete a common two-line idiom is not a compelling enough
> reason to do so.

py3k is going to break large ammounts of code anyway. This pep certainly  
won't break the most of it. And there's gonna be an automatic py2 -> py3  
refactoring tool, that can catch any possible breakage from this pep as  
well.

> Helping out beginning Python programmers, while well-intentioned,
> doesn't feel like enough of a motivation either.  Notice that the main
> challenge for the novice programmer is not to learn how default
> arguments work -- novices can learn to recognize and write the idiom
> easily enough -- but rather to learn how variables and objects work in
> general.
[snip]
> At some point in his Python career, a novice is going to have to
> understand why b "changed" but d didn't.  Fixing the default argument
> "wart" doesn't remove the necessity to understand the nature of
> mutable objects and variable bindings in Python; it just postpones the
> problem.  This is a fact worth keeping in mind when deciding whether
> the sweeping change in semantics is worth the costs.

The change was never intended to prevent newbies from learning about  
pythons object model. There are other ways to do that. But keeping a  
'wart' because newbies will learn from it seems like really bad reasoning,  
language-design wise.

>> - Arguments based on the assumption that people actually do make lots of
>> use of the fact that default arguments are shared between function
>> invocations, many of which will result in (much) more code if it has to  
>> be
>> transformed to using one of the alternative idioms. If this is true, it  
>> is
>> a valid argument. I guess there's still some stdlib grepping to do to
>> decide this.
>
> Though it's been decried here as unPythonic, I can't be the only
> person who uses the idiom
> def foo(..., cache={}):
> for making a cache when the function in question does not rise to the
> level of deserving to be a class object instead.  I don't apologize
> for finding it less ugly than using a global variable.

How often do you use this compared to the x=None idiom?

This idiom is really going to be the only idiom that's going to break.  
There are many ways around it, I wouldn't mind an @cache(var={}) decorator  
somewhere (perhaps in the stdlib). These kind of things seem to be exactly  
what decorators are good at.

> I know I'm not the only user of the idiom because I didn't invent it
> -- I learned it from the Python community.  And the fact that people
> have already found usages of the current default argument behavior in
> the standard library is an argument against the "unPythonic" claim.
>
> I'm reminded of GvR's post on what happened when he made strings
> non-iterable in a local build (iterable strings being another "wart"
> that people thought needed fixing):
> http://mail.python.org/pipermail/python-3000/2006-April/000824.html

In that thread, Guido is at first in favour of making strings  
non-iterable, one of the arguments being that it sometimes bites people  
who expect e.g. a list of strings and get a string. He decides not to make  
the change because there appear to be a number of valid use cases that are  
hard to change, and the number of people actually getting bitten by it is  
actually quite small. (To support that last part, note for example that  
none of the 'python problems' pages listed in the pep talk about string  
iteration while all talk about default arguments, some with dire warnings  
and quite a bit of text.)
In the end, the numbers are going to be important. There seems to be only  
a single use case in favour of definition time semantics for default  
variables (caching), which isn't very hard to do in a different way.  
Though seasoned python programmers don't get bitten by default args all  
the time, they have to work around it all the time using =None.

If it turns out that people are actually using caching and other idioms  
that require definition time semantics all the time, and the =None idiom  
is used only very rarely, I'd be all in favour of rejecting this pep.

>
>> So, are there any _other_ arguments in favour of the current semantics??
>
> Yes.  First, consistency.

[factoring out the first argument into another email. It's taking me some  
effort to get my head around the early/late binding part of the generator  
expressions pep, and the way you find an argument in that. As far as I  
understand it currently, either you or I do not understand that part of  
the pep correctly. I'll try to get this mail out somewhere tomorrow]

> Second, the a tool can't fix all usages of the old idiom.  When things
> break, they can break in subtle or confusing ways.  Consider my module
> "greeter":
>
> == begin greeter.py ==
> import sys
> def say_hi(out = sys.stdout):
>  print >> out, "Hi!"
> del sys # don't want to leak greeter.sys to the outside world
> == end greeter.py ==
>
> Nothing I've done here is strange or unidiomatic, and yet your
> proposed change breaks it, and it's unclear how an automated tool
> should fix it.

Sure this can be fixed by a tool:

import sys
@caching(out = sys.stdout)
def say_hi(out):
	print >> out, "Hi!"
del sys

where the function with the 'caching' wrapper checks to see if an argument  
for 'out' is provided, or else provides it itself. The caching(out =  
sys.stdout) is actually a function _call_, so it's sys.stdout gets  
evaluated immediately.

possible implementation of caching:

def caching(**cachevars):
	def inner(func):
		def wrapper(**argdict):
			for var in cachevars:
				if not var in argdict:
					argdict[var] = cachevars[var]
			return func(**argdict)
		return wrapper
	return inner

Defining a decorator unfortunately requires three levels of nested  
functions, but apart from that the thing is pretty straightforward, and it  
only needs to be defined once to use on every occasion of the caching  
idiom.
It doesn't currently handle positional vars, but that can be added.

> What's worse about the breakage is that it doesn't
> break when greeter is imported,

That's true of any function with a bug in it. Do you want to abandon  
functions alltogether?

> or even when greeter.say_hi is called
> with an argument.

Currently for people using x=None, if x=None <calculate default value>,  
this difference is a branch in the code. That's why you need to test _all_  
possible branches in your unit test. Analagously you need to test all  
combinations of arguments if you want to catch as many bugs as possible.

> It might take a while before getting a very
> surprising error "global name 'sys' is not defined".


However, your greeter module actually has a slight bug. What if I do this:

import sys, greeter
sys.stdout = my_output_proxy()
greeter.say_hi()


Now say_hi() still uses the old sys.stdout, which is most likely not what  
you want. If greeter were implemented like this:

import sys as _sys
def say_hi(out = _sys.stdout):
	print >> out, "Hi!"


under the proposed semantics, it would all by itself do a late binding of  
_sys.stdout, so when I change sys.stdout somewhere else, say_hi uses the  
new stdout.
Deleting sys in order not to 'leak' it to any other module is really not  
useful. Everybody knows that python does not actually enforce  
encapsulation, nor does it provide any kind of security barriers between  
modules. So if some other module wants to get at sys it can get there  
anyway, and if you want to indicate that sys isn't exporters and greeter's  
sys shouldn't be messed around with, the renaming import above does that  
just fine.

> Third, the old idiom is less surprising.
>
> def foo(x=None):
>  if x is None:
>    x=<some_expr>
>
> <some_expr> may take arbitrarily long to complete.  It may have side
> effects.  It may throw an exception.  It is evaluated inside the
> function call, but only evaluated when the default value is used (or
> the function is passed None).
>
> There is nothing surprising about any of that.  Now:
>
> def foo(x=<some_expr>):
>  pass
>
> Everything I said before applies.  The expression can take a long
> time, have side effects, throw an exception.  It is conditionally
> evaluated inside the function call.
>
> Only now, all of that is terribly confusing and surprising (IMO).

Read the "what's new in python 3.0" (assuming the pep gets incorporated,  
of course).
Exception tracebacks and profiler stats will point you at the right line,  
and you will figure it out. As you said above, all of it is true under the  
current =None idiom, so there are no totally new ways in which a program  
can break. If you know the ways current python can break (take too long,  
unwanted side effects, exceptions) you will figure it out in the new  
version.
Anyway, many python newbies consider it confusing and surprising that an  
empty list default value doesn't stay empty, and all other pythoneers have  
to work around it a lot of times. It will be a pretty unique python  
programmer whose program will break in ways mentioned above by the default  
expression being evaluated at call time, and wouldn't have broken under  
python's current behaviour, and who isn't able to figure out what happened  
in a reasonable amout of time. So even if your argument holds, it will  
still be a net win to accept the pep.

>
>
> Greg F
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas

- Jan


From cvrebert at gmail.com  Thu Feb  1 07:27:22 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Wed, 31 Jan 2007 22:27:22 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BFA129.1030805@onego.ru>
References: <45BCF804.7040207@gmail.com>
	<45BDA47F.4040603@onego.ru>	<45BED64D.10801@gmail.com>
	<45BFA129.1030805@onego.ru>
Message-ID: <45C1884A.20000@gmail.com>

Added to PEP draft. Thanks.
- Chris Rebert

Roman Susi wrote:
> Chris Rebert wrote:
>> Roman Susi wrote:
>>
>>> Hello!
>>>
>>> I'd liked to say outright that this bad idea which complicates matters
> 
> [skip]
> 
>>> P.S. However, I may be wrong. In that case my syntax suggestion would
>>> be this:
>>>
>>> def foo(non_const or []):
>>>    ...
>>>
>>> where [] is executed at runtime BECAUSE at def time non_const is
>>> somehow True and that is enough to leave [] alone.
>>> I have not checked, but I believe it is backward compatible.
>>> Anyway, could you summarize both contr-argument and this syntax
>>> proposal in the PEP?
>>
>> I don't quite understand exactly how this would work and would like more
>> details on it, but once you've explained it, of course I'd be happy to
>> include it in the next draft.
> 
> Simple.
> 
> 
> def foo(non_const or []):
>     ...
> 
> is equivalent to
> 
> def foo(non_const=None):
>     if non_const is None:
>         none_const = []
>     ...
> 
> 
> And this will be as before:
> 
> def foo(non_const=[]):
>     ...
> 
> Also, I thing that programmers should not use subtle difference between
> None and other False values, so something like
> 
> def foo(non_const=None):
>     non_const = none_const or []
> 
> is also valid.
> 
> Another approach (if you want to pursue the feature) could be
> complication to name binding protocol.
> 
> a = []
> 
> will be as before, but default value assignment could trigger some extra
> method. So, you can explicitly regulate your instance reaction to
> default-value assignment:
> 
> class MyMutableList:
>     ...
>     def __default__(self, old_default):
>         return old_default.copy()
> 
> 
> 
> 
> Regards,
> Roman
> 
> 
>> - Chris Rebert
>>
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
> 


From tal.no.no.spam at gmail.com  Fri Feb  2 12:22:53 2007
From: tal.no.no.spam at gmail.com (Tal Einat)
Date: Fri, 2 Feb 2007 13:22:53 +0200
Subject: [Python-ideas] File system API
Message-ID: <cd8c55c90702020322g3bc4545hd194fef986ee1c90@mail.gmail.com>

I'm proposing a general file-system API (for Python of course).
Basically, that's it.


I've been wondering how many thousands of times this has been
discussed, but an hour's worth of searching the net actually gave very
little on this subject. Most other proposals I've seen started far,
far further into developing the idea, and I think it's worth giving it
a fresh start from the beginning.

Also, most proposals centered around either creating efficient fake
file-systems, simplifying working with paths, or unifying
functionality currently spread between several Python modules. I
believe these can be achieved, and more - but let's first think what
other goals can be achieved, which goals conflict with each other, and
which are the most important.


Just for a taste: The OS's file system would be an instance of a class
implementing this API. Other examples of classes which would implement
this API: FTP implementations, file system proxies in RPC frameworks,
and "fake" file systems for testing purposes. (This last example -
testing - is where most of the discussion I've seen on this subject
was centered.) Surely more uses can be found - I think we should think
through the uses cases before thinking of design.

Personally, my need is for a remote file-system proxy - my application
runs multiple remote servers, and I need to work with a virtual file
system on each server. Currently the code for this is quite horrid
(low-level hackery); with an API for file-systems I would just need to
wrap the local file-system with a proxy class and be done.

Obviously I have some initial ideas concerning design, but I haven't
hacked at file-systems much yet, and I'd like to hear what more
experienced people have to say.


So let's leave out design and implementation ideas for later, and focus on:
* What would be the uses for this (e.g. file-system proxies)
* What would be the benefits and drawbacks
* Is there any reason why this would not be possible
* Is this a good idea at all (or are we better off without it)
* How important do we think this is (for the betterment of Python)


- Tal Einat
reduce(lambda m,x:[m[i]+s[-1] for i,s in enumerate(sorted(m))],
      [[chr(154-ord(c)) for c in '.&-&,l.Z95193+179-']]*18)[3]


From eopadoan at altavix.com  Fri Feb  2 12:37:00 2007
From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan)
Date: Fri, 2 Feb 2007 09:37:00 -0200
Subject: [Python-ideas] File system API
In-Reply-To: <dea92f560702020336t2aee1f51sa0f23e816cb13aed@mail.gmail.com>
References: <cd8c55c90702020322g3bc4545hd194fef986ee1c90@mail.gmail.com>
	<dea92f560702020336t2aee1f51sa0f23e816cb13aed@mail.gmail.com>
Message-ID: <dea92f560702020337x158dd14dk9bd5dc3a457cadb6@mail.gmail.com>

I think that there aready exists a  proposal for an Abstract FS Layer
for Python somewere.

--
EduardoOPadoan (eopadoan->altavix::com)
Bookmarks: http://del.icio.us/edcrypt
Blog: http://edcrypt.blogspot.com
Jabber: edcrypt at jabber dot org
ICQ: 161480283
GTalk: eduardo dot padoan at gmail dot com
MSN: eopadoan at altavix dot com


From eopadoan at altavix.com  Fri Feb  2 13:17:20 2007
From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan)
Date: Fri, 2 Feb 2007 10:17:20 -0200
Subject: [Python-ideas] File system API
In-Reply-To: <dea92f560702020416s7e25f5efs689c61ae0e8c5386@mail.gmail.com>
References: <7afdee2f0702020402h55e605cv6b591cafe91bf2b4@mail.gmail.com>
	<dea92f560702020416s7e25f5efs689c61ae0e8c5386@mail.gmail.com>
Message-ID: <dea92f560702020417i425b1443o3fa9aaffbf03e953@mail.gmail.com>

> > I think that there aready exists a proposal for an Abstract FS Layer
> > for Python somewere.
>
> I haven't been able to find any mention of it. Maybe you could point me in
> the right direction?
>
> - Tal
>
http://wiki.python.org/moin/CodingProjectIdeas/FileSystemVirtualization

Another related reference is the recently announced UniPath module:
http://sluggo.scrapping.cc/python/unipath/Unipath-current/README.html


-- 
EduardoOPadoan (eopadoan->altavix::com)
Bookmarks: http://del.icio.us/edcrypt
Blog: http://edcrypt.blogspot.com
Jabber: edcrypt at jabber dot org
ICQ: 161480283
GTalk: eduardo dot padoan at gmail dot com
MSN: eopadoan at altavix dot com


From tal.no.no.spam at gmail.com  Fri Feb  2 14:41:38 2007
From: tal.no.no.spam at gmail.com (Tal Einat)
Date: Fri, 2 Feb 2007 15:41:38 +0200
Subject: [Python-ideas] File system API
In-Reply-To: <dea92f560702020336t2aee1f51sa0f23e816cb13aed@mail.gmail.com>
References: <cd8c55c90702020322g3bc4545hd194fef986ee1c90@mail.gmail.com>
	<dea92f560702020336t2aee1f51sa0f23e816cb13aed@mail.gmail.com>
Message-ID: <cd8c55c90702020541s746bb39xc959a063d4e6a4fe@mail.gmail.com>

On 2/2/07, Eduardo EdCrypt O. Padoan <eopadoan at altavix.com> wrote:

> I think that there aready exists a  proposal for an Abstract FS Layer
> for Python somewere.
>

I haven't really seen a proposal for such a thing.

itools.vfs is the only actualy file-system API, or file-system abstraction
layer, I know of. It is going more or less in the directions I am currently
thinking of. But the fact is that it has not gained popularity makes me
think that maybe the direction needs to be better thought out. And it's not
really a proposal as much as a partial implementation, created mostly for
the needs of the other itools modules.

Most other proposals had to do with handling file-systems paths, and while
some also mixed in some functionality for reading/writing files, browsing
directory trees, etc., they do not provide a layer of abstraction for
file-systems and file-system-like entities.

This is why I think there is room for such an initiative.

- Tal Einat
reduce(lambda m,x:[m[i]+s[-1] for i,s in enumerate(sorted(m))],
       [[chr(154-ord(c)) for c in '.&-&,l.Z95193+179-']]*18)[3]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070202/a7c3ef6c/attachment.html>

From jcarlson at uci.edu  Fri Feb  2 18:57:02 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 02 Feb 2007 09:57:02 -0800
Subject: [Python-ideas] File system API
In-Reply-To: <cd8c55c90702020541s746bb39xc959a063d4e6a4fe@mail.gmail.com>
References: <dea92f560702020336t2aee1f51sa0f23e816cb13aed@mail.gmail.com>
	<cd8c55c90702020541s746bb39xc959a063d4e6a4fe@mail.gmail.com>
Message-ID: <20070202093758.5AB0.JCARLSON@uci.edu>


"Tal Einat" <tal.no.no.spam at gmail.com> wrote:
> On 2/2/07, Eduardo EdCrypt O. Padoan <eopadoan at altavix.com> wrote:
> 
> > I think that there aready exists a  proposal for an Abstract FS Layer
> > for Python somewere.
> >
> 
> I haven't really seen a proposal for such a thing.
> 
> itools.vfs is the only actualy file-system API, or file-system abstraction
> layer, I know of. It is going more or less in the directions I am currently
> thinking of. But the fact is that it has not gained popularity makes me
> think that maybe the direction needs to be better thought out.

Or maybe it's because not that many people need a file-system API?  I
would imagine that the needs of most people lie within the realm of
manipulating the local filesystem, and when remote access is necessary,
they choose some set of remote protocols they want to use to access
remote filesystems, and just use the relevant client protocols for that.


 - Josiah



From grosser.meister.morti at gmx.net  Fri Feb  2 19:08:03 2007
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Fri, 02 Feb 2007 19:08:03 +0100
Subject: [Python-ideas] File system API
In-Reply-To: <20070202093758.5AB0.JCARLSON@uci.edu>
References: <dea92f560702020336t2aee1f51sa0f23e816cb13aed@mail.gmail.com>	<cd8c55c90702020541s746bb39xc959a063d4e6a4fe@mail.gmail.com>
	<20070202093758.5AB0.JCARLSON@uci.edu>
Message-ID: <45C37E03.7050708@gmx.net>

Josiah Carlson schrieb:
> "Tal Einat" <tal.no.no.spam at gmail.com> wrote:
>> On 2/2/07, Eduardo EdCrypt O. Padoan <eopadoan at altavix.com> wrote:
>>
>>> I think that there aready exists a  proposal for an Abstract FS Layer
>>> for Python somewere.
>>>
>> I haven't really seen a proposal for such a thing.
>>
>> itools.vfs is the only actualy file-system API, or file-system abstraction
>> layer, I know of. It is going more or less in the directions I am currently
>> thinking of. But the fact is that it has not gained popularity makes me
>> think that maybe the direction needs to be better thought out.
> 
> Or maybe it's because not that many people need a file-system API?  I
> would imagine that the needs of most people lie within the realm of
> manipulating the local filesystem, and when remote access is necessary,
> they choose some set of remote protocols they want to use to access
> remote filesystems, and just use the relevant client protocols for that.
> 
> 
>  - Josiah

And with KDE bindings for python you have KIO in python and therefore file:/ zip:/ tar:/ (s)ftp:/
http(s):/ fish:/ (rocks!) and many others. :)

Yes, it's dependent on KDE and KDE is not yet available for all platforms python is. But as sayed
before, there is no big need for such an API (though it would be very nice to have one).

panzi


From cvrebert at gmail.com  Sat Feb  3 03:10:22 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Fri, 02 Feb 2007 18:10:22 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <43aa6ff70701292122o6248a9afi3a139a106fe717f5@mail.gmail.com>
References: <45BCF804.7040207@gmail.com>	
	<43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com>	
	<45BECF7B.6090503@gmail.com>
	<43aa6ff70701292122o6248a9afi3a139a106fe717f5@mail.gmail.com>
Message-ID: <45C3EF0E.4050909@gmail.com>

Collin Winter wrote:
> On 1/29/07, Chris Rebert <cvrebert at gmail.com> wrote:
>> Collin Winter wrote:
>> > On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
>> > Syntax changes are a huge stick to wield against such a small problem.
>> > I realize the boilerplate is annoying, but Tomer has pointed out a
>> > number of decorator-based solutions [1] that could easily be adapted
>> > to any unusual needs you have.
>>
>> One of his decorators assumes that the default value merely needs to be
>> copied across calls. However, as the PEP says, there are cases where
>> this isn't sufficient or just doesn't work.
> 
> Tomer wasn't proposing that any of those decorators be included in the
> standard library. Those cases where one of decorators isn't sufficient
> as written? Modify it, use it in your code, post it to the cookbook.
> Not every n-line function should be turned into syntax.

Right, but for something used so frequently, should you really have to 
look through N decorator recipes, figure out which one you want, and 
then modify it, as opposed to a change in semantics that obviates the 
need for such decorators altogether?

>> His second decorator
>> involves using lambdas (ick!), and doesn't allow you to refer to other
>> arguments in determining the default value (think 'self' as a hint for
>> how this could be useful).
> 
> You can't do that as things are, so that's not a strike against the
> decorator-based approach.

Valid point. It's actually more pro-reevaluation than it is anti-decorator.

>> > but how do you intend to capture these "default argument expressions"?
>> > Emit additional bytecode for functions making use of these special
>> > default arguments?
>>
>> The "Reference Implementation" section of the PEP discusses this. I
>> don't personally know Python's internals, but I imagine this proposal
>> would just change how default arguments are compiled and might entail
>> some changes to the interpreter's argument-processing machinery.
> 
> As you work on this, think about how you'll explain the new semantics
> in the documentation. Rewriting http://docs.python.org/ref/calls.html
> and http://docs.python.org/ref/function.html -- paying particular
> attention to paragraphs 2-5 in the former -- would be a good start.

Sweet! Thanks for the link.

- Chris Rebert


From jcarlson at uci.edu  Sat Feb  3 04:08:44 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 02 Feb 2007 19:08:44 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45C3EF0E.4050909@gmail.com>
References: <43aa6ff70701292122o6248a9afi3a139a106fe717f5@mail.gmail.com>
	<45C3EF0E.4050909@gmail.com>
Message-ID: <20070202190104.5AC2.JCARLSON@uci.edu>


Chris Rebert <cvrebert at gmail.com> wrote:
> Collin Winter wrote:
> > Tomer wasn't proposing that any of those decorators be included in the
> > standard library. Those cases where one of decorators isn't sufficient
> > as written? Modify it, use it in your code, post it to the cookbook.
> > Not every n-line function should be turned into syntax.
> 
> Right, but for something used so frequently, should you really have to 
> look through N decorator recipes, figure out which one you want, and 
> then modify it, as opposed to a change in semantics that obviates the 
> need for such decorators altogether?

Ahh, but the decorator recipes are more or less examples of how one
could go about removing the 1 additional if statement/conditional
expression.  Obviously you don't *need* to look through the decorators,
because the 1 line if statement or conditional expression solves them
all!  Really, you could think of the decorator variants as anti-patterns,
because they seek to complicate what is trivial to solve, which isn't an
issue in real world code.


 - Josiah



From cvrebert at gmail.com  Sat Feb  3 06:21:12 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Fri, 02 Feb 2007 21:21:12 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <fb6fbf560701292319l4b42f0b4nc33da1eb1b00ce96@mail.gmail.com>
References: <45BCF804.7040207@gmail.com>	
	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>	
	<45BED309.10301@gmail.com>
	<fb6fbf560701292319l4b42f0b4nc33da1eb1b00ce96@mail.gmail.com>
Message-ID: <45C41BC8.3050304@gmail.com>

Apologies in advance for the lateness of my reply. I missed a few emails 
in all the shuffle.

Jim Jewett wrote:
> On 1/30/07, Chris Rebert <cvrebert at gmail.com> wrote:
>> Jim Jewett wrote:
>> > On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> 
>> >> Naive programmers desiring mutable default arguments
>> >> often make the mistake of writing the following:
> 
>> >>      def foo(mutable=some_expr_producing_mutable):
>> >>          #rest of function
> 
>> > Yes, it is an ugly gotcha, but so is the alternative.
>> > If it is changed, then just as many naive programmers (though perhaps
>> > not exactly the same ones) will make the opposite mistake -- and so
>> > will some experienced programmers who are used to current semantics.
> 
>> Anyone (ab)using the current semantics with the above construct is or
>> ought to be aware that such usage is unpythonic.
> 
> Are you worried about naive programmers or not?
[snip]
 >> Additionally, I see
 >> making the opposite mistake as usually only maybe causing a performance
 >> problem, as opposed to causing the program to work incorrectly.
 >
 > That is true only when the argument is a cache.
 > Generally, resetting the storage will make the program work
 > incorrectly, but the program won't fail on the first data tested --
 > which means the bug is less likely to be caught.
 >

I am worried about naive programmers. It seems that the opposite mistake 
(thinking arguments will be evaluated only once, at def-time, rather 
than at each call) won't have non-performance related effects a good bit 
of the time. For many of the remaining cases, the person probably 
is/should be using a constant instead of a default argument. Yes, there 
will be some cases where badness could happen. However, (1) this is 
intended for Py3k, so there will already be a bunch of changes for 
newbies to learn, (2) the chance of weirdness happening is a good bit 
lower compared to the current state of mutable default arguments.

>> Also, due to the
>> hairiness/obscurity of the construct, hardly anyone uses it, so the
>> impact of removing it should be relatively minor.
> 
> I just did found 181 such uses in my own installation.  29 were either
> 3rd-party or test code, but that still leaves over 150 uses in the
> standard library.  It is possible that some of them would also work
> with your semantics, or may even be bugs today -- but you would have
> to go through them one-by-one as part of the PEP process.

I have a script to gather statistics on default argument usage in the 
standard library 90% done. Its results will be included in the next PEP 
draft.

>>      There are currently few, if any, known good uses of the current
>> behavior of mutable default arguments.  The most common one is to
>> preserve function state between calls.  However, as one of the 
>> lists [2] comments, this purpose is much better served by decorators, 
>> classes, or(though less preferred) global variables.
> 
> I disagree.  This is particularly wrong for someone coming from a
> functional background.
> 
>> I assume you disagree with the "purpose is much better served by
>> decorators, classes, or local variables" part as opposed to the "default
>> mutable arguments with current semantics have few good uses" part.
>> Please correct me if I'm in error.
> 
> I disagree with both.  There are many uses for preserving function
> state between calls.  C uses static local variables.   Object Oriented
> languages often use objects.

First, I was referring to using default args for caching being evil, not 
mutable default arguments in general. Just wanted to clarify in case you 
misinterpreted me.
True, there are many uses for preserving function state between calls, 
but how much better/clearer is doing so using default arguments as 
opposed to objects or other approaches? It's an aesthetic decision. 
IMHO, the other ways seem better. YMMV.

>> > A class plus an instantiation seems far too heavyweight for what ought
>> > to be a simple function.  I'm not talking (only) about the runtime;
>> > the number of methods and lines of code is the real barrier for me.
>> > I'll sometimes do it (or use a global) anyhow, but it feels wrong.  If
>> > I had felt that need more often when I was first learning python (or
>> > before I knew about the  __call__ workaround), I might have just
>> > written the language off as no less bloated than java.
> 
>> I'm sorry, but when you have a function sharing state between calls,
>> that just screams to me that you should make it a method of an object
> 
> That's because you drank the OO koolaid.  Python tries not to enforce
> any particular programming style.  Its object model happens to be
> pretty good, but there are still times when OO isn't the answer.

Well, it's a trade-off: support procedural/functional programming a bit 
more, or make default arguments a bit more intuitive and obsolete the 
'foo=None; if foo is None' idiom.

>> If performance is your concern, the decorator
>> version might perform better (I don't really know), or in the extreme
>> case, you could use a global variable, which is definitely faster.
> 
> I think there is something wrong with your benchmark.

Sorry, yeah, you're right. I was being a little fast and loose. But the 
point was that there are (at the price of some elegance) ways to 
optimize such situations even more.

> That said, speed is *not* my concern -- code size is.  If I have to
> add several lines of boilerplate, the code becomes much less readable.
> That is roughly the same justification you have for not liking the
> "if arg is None:  arg=foo()" idiom.  The difference is that you're
> suggesting adding far more than a line or two.

True, however I'm willing to bet that the boilerplate for evaluating 
default arguments only once won't be needed nearly as much, as such 
situations requiring such behavior is not too common. The statistics 
should help show exactly how much mutable default arguments are 
exploited in the wild.

>> Alternatively, we could make the new
>> semantics the default and have syntax to use the old semantics via
>> 'once' or some other keyword. This does nicely emphasize that such
>> semantics would be purely for optimization reasons.
> 
> Using a mutable default is almost never for optimization reasons.

Actually, I was more thinking it might be used to stop immutable default 
arguments from being re-evaluated multiple times, if it turns out that 
re-evaluations really do have major effects on performance. Now that I 
think about it, I guess it could be used to get the old behavior for 
mutable default arguments.

- Chris Rebert


From rrr at ronadam.com  Sun Feb  4 20:26:27 2007
From: rrr at ronadam.com (Ron Adam)
Date: Sun, 04 Feb 2007 13:26:27 -0600
Subject: [Python-ideas] Packages and Import
Message-ID: <45C63363.2080100@ronadam.com>


After exploring this a bit further on comp.lang.python, I was able to organize 
these ideas better.  The more I thought about it, the more '+'s I found, and 
about the only '-'s I can think of is the work required to actually make a patch 
to do it.

It's also good to keep in mind that since most people still rely on the old 
relative import behavior, most people have not run into some of the issues I 
mention here.  But they will at some point.

I did mean to keep this short, but clarity won out. (At least it's clear to me, 
but that's an entirely subjective opinion on my part.)

Maybe someone will adopt this and make a real PEP out of it.  :-)

Cheers,
   Ron



PROPOSAL
========

Make pythons concept of a package, (currently an informal type), be stronger 
than that of the underlying file system search path and directory structure.


Where the following hold true in python 3.X, or when absolute_import behavior is 
imported from __future__ in python 2.X:


(1) Python first determines if a module or package is part of a package and then 
runs that module or package in the context of the package they belong to. (see 
items below)


(2)  import this_package.module
      import this_package.sub_package

If this_package is the same name as the current package, then do not look on 
sys.path. Use the location of this_package.


(3)  import other_package.module
      import other_package.sub_package

If other_package is a different name from the current package (this_package), 
then do not look in this_package and exclude searches in sys.path locations that 
are inside this_package including the current directory.


(4)  import module
      import package

Module and package are not in a package, so don't look in any packages, even 
this one or sys.path locations inside of packages.


(5) For behaviors other than these, like when you do actually want to run a 
module belonging to a package in a different context, a mechanism such as a 
command line switch, or a settable import attribute should be used.


MOTIVATION
==========

(A) Added reliability.

There will be much less chance of errors (silent or otherwise) due to 
path/import conflicts which are sometimes difficult to diagnose.

There may also be some added security benefits as well because it would much 
harder for someone to create a same named module or package and insert it by 
putting it on the path. Or by altering sys.path to do the same. [*]

[* - If this can happen there are probably more serious security issues, but not 
everyone has the most secure setup, so this point is still probably a good 
point. General reliable execution of modules is the first concern, this may be a 
side benefit of that.]


(B) Reduce the need for special checks and editing sys.path.

Currently some authors have edit sys.path or do special if os.path.exists() 
checks to ensure proper operations in some situations such as running tests. 
These suggestions would reduce the need for such special testing and modifications.


(D) Easier editing and testing.

While you are editing modules in a package, you could then run the module 
directly (as you can with old style relative imports) and still get the correct 
package-relative behavior instead of something else. (like an exception or wrong 
output). Many editors support running the file being edited, including idle. 
It's also can be difficult to write scripts for the editors to determine the 
correct context to run a module in.


(E) Consistency with from ... import ... relative imports.

A relative import also needs to find it's home package(s).  These suggestions 
are consistent with relative import needs and would also enable relative imports 
to work if a module is run directly from and editor (like idle) while editing 
it.  [*]

[* - Consistency isn't a major issue, but it's nice to have.]




(F) It would make things much easier for me.  ;-)

    (Insert "Me Too's" here.)



DISCUSSION
==========

(I)
Python has a certain minimalist quality where it tries to do a lot with a
minimum amount of resources. (Which I generally love.)  But in the case of 
packages, that might not be the best thing.  It is not difficult for python to 
detect if a module is located in a package.

With the move to explicit absolute/relative imports, it would make since if 
Python also were a little smarter in this area.  Packages are becoming used more 
often and so it may also be useful to formalize the concept of a package in a 
stronger way.


(II)
Many of the problems associated with imports are a side effect of using the OS's 
directory structure to represent a python "package" structure.  This creates 
some external dependence on the operating system that can effect how python 
programs run.  Some of these issues include:

    - Importing the wrong package or module.

    - Getting an error due to a package or module not being found.

    - Getting an error due to a package not being loaded or initialized first.

    - Having to run modules or packages within a very specific OS file context.

    - Needing a package location to be in the systems search path.

By making the concept of a package have priority over the OS's search path and 
directory structure, the dependence on the OS's environment is lessoned and it 
would insure a module runs in the correct context or give meaningful exceptions 
in more cases than presently.


(III)
If a package was represented as a combined single file.  Then the working 
directory would always be the package directory.  The suggestions presented here 
would have that effect and also reduce or eliminate most if not all of these 
problem situations.


(IV)
The suggested changes would change the precise meaning of an absolute import.

Given the following example of an un-dotted import:

     >>> import foo

The current meaning is:

     "A module or package that is located in sys.path or the current
       directory".

But maybe a narrower interpretation of "absolute import" would be better:

     "A module or package found in a specific package."

I believe that this latter definition is what most people will think of while 
editing their programs.  When dotted imports are used, the left most part of the 
name is always a top level package or module in this case.


(V) Requirements to be on the search path.

It is quite reasonable to have python modules and packages not in the search 
path.  Conversely it is not reasonable to require all python modules or packages 
to be in locations listed in sys.path.

While this isn't a true requirement, it is often put forth as a solution to some 
of the problems that occur with respect to imports.


(VI) Clearer errors messages.

In cases where a wrong module or package is imported you often get attribute 
exceptions further in the code.  These changes would move that up to the import 
statement because the wrong module would not be imported.


(VII) Setting a __package__ attribute.

Would it be a good idea to have a simple way for modules to determine parent 
packages, and their absolute locations?  Python could set these when it starts 
or imports a module.  That may make it easier to write alternate importers that 
are package aware.



PROBLEMS AND ISSUES:

   - Someone needs to make it happen.


I really can't think of any more than that.  But I'm sure there are some as most 
things like this are usually a trade off of something.





From jan.kanis at phil.uu.nl  Wed Feb  7 16:53:57 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Wed, 07 Feb 2007 16:53:57 +0100
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <999187ed0702010719o611243dcp8725f3b40000f538@mail.gmail.com>
References: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
	<op.tmsemwqxd64u53@e500> <20070126212657.5A06.JCARLSON@uci.edu>
	<op.tmyxw9g2d64u53@aidwe2058w.minlnv.agro.nl>
	<999187ed0701300736v517fd7a8r69eba959bc97d691@mail.gmail.com>
	<op.tm1tcigad64u53@e500>
	<999187ed0702010719o611243dcp8725f3b40000f538@mail.gmail.com>
Message-ID: <op.tndyf7uvd64u53@e500>

Other non-python stuff is taking my time, but I'll reply to this anyway.  
Better late than never (in this case).

On Thu, 01 Feb 2007 16:19:00 +0100, Lucio Torre <lucio.torre at gmail.com>  
wrote:

> On 1/31/07, Jan Kanis <jan.kanis at phil.uu.nl> wrote:
>> >
>> > Isnt the other argument the fact that always re evaluating will bring
>> > similar gotchas?
>>
>> IMO, it won't. Caching will have to be implemented in a different way,  
>> but
>> that wouldn't really be a gotcha (except for python 2.x programmers who
>> didn't read the "what's new in python 3.0")
>>
>
> Maybe is better to annoy newbies that didnt read the python FAQ than
> seasoned python 2.x programmers! :)

That statement I can agree with, but it isn't really applicable here. The  
applicable statement reads: "Maybe it's better to annoy newbies that  
didn't read the python FAQ than python 2.x programmers who don't know  
anything about 3.0 and assume it's just another new version of python."
And I don't agree with that one. :)

>>  If the
>> function depends on other state I'd imagine that most of the time you'd
>> want a change in state to be reflected in the functions default value.
>>
>> example:
>> def foo(x = exec_sql("SELECT someval FROM defaults_table WHERE
>> some_expr")):
>>     # do something
>>
>
> i was thinking more in the lines of the case when you dont want the
> default to change because something has changed.

Then don't make the default depend on something you're going to change.

Well, what this is really about is about closure vars changing when the  
outer scope changes them instead of the value of the closure vars being  
captured by the new lexical scope. This has been proposed before (to  
change this behaviour on closure variables in general) and rejected.  
(though I am in favour of it)
Python is a late binding language in just about everything, and that is  
considered to be a good thing. The point you're making is exactly the  
early-vs-late binding discussion.

> I still think this would be optimizing towards the case where you
> should put the code in the body. What if i want a static default
> parameter, never to change, but i want to compute it at compile time?
> Should i need to use extra syntax for that? Isnt this case more common
> than the others? Something like:
>
> def foo( pixels = (640*480) ): pass
>
> Should overriding '*' change that default value? (can you do that in
> python3k? i think you cant in 2.x)

I'm not really sure if you can either. Anyway, the only cases I can think  
of when this makes sense is when there's a bug in '*' and you want to  
patch it, or if you're overloading '*' to work with other values, without  
changing the meaning for ints. In both of these cases it would be a good  
resp. neutral thing to late-bind '*'

> On the other hand, i think that learning about compile time and run
> time is a great thing and should be encouraged.

Yup. But let's not fill the language with gotchas for the newbies to find  
and learn from :)

>> > And having code that is invoked every time a function is called..
>> > shouldnt the place for this be the function body?
>>
>> The only reason to answer 'yes' is because it is what you're currently
>> used to.
>
> Im not only used to it, i like it :)

You'll get to like the new way :)

>> If I see a function with a default argument of [], I expect it to
>> use an empty list when I call it without the argument.
>>
>
> That is true. I hate that. But i still like it better than the other
> cases. This can be learnt easily and gives you the power to do
> everything the other solutions have. The other solutions require more
> syntax and more learning to achieve the same. Id love to find a
> solution for this, but this options dont seem like one.

Please note that I'm not in favour of any of the possible new syntaxes, I  
am only in favour of changing the semantics without new syntax (cf the  
pre-pep).
Since many newbie python programmers run into this, it seems that it is  
easier to learn that python evaluates the default expressions at calltime,  
because that is what the rest of python makes the newbies expect when they  
first try it. So for newbies, there would actually be less learning  
involved.
You will have to remember this change when you start using py 3.0, but  
this won't be the only thing to remember. And there are still several  
other ways to make some python code evaluate the expression at deftime.

>> Having code that is eval'd when the function is defined, shouldn't the
>> place for this be outside the function definition?
>
> i dont know, "def fname (...):" is executed when the code block is
> executed (creating the function object), and the body is executed when
> the function is called. So having a statement that has both compile
> time and run time effects is kind of messy.

As the body is part of the def statement, the def is already doing exactly  
that (being both compile/def time and run/call time). Consider the ...'s  
to be part of the function body in the new semantics. Instead of part of  
the code in a def being executed at deftime and part at calltime,  
everything is just run at calltime. There are some pretty large warnings  
in the docs about this distinction currently, they can be taken out.

> I hope im still making sense and not playing out of my league, but
> thats just my 2c.
>
> Lucio.


- Jan


From jcarlson at uci.edu  Wed Feb  7 18:46:20 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Wed, 07 Feb 2007 09:46:20 -0800
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <op.tndyf7uvd64u53@e500>
References: <999187ed0702010719o611243dcp8725f3b40000f538@mail.gmail.com>
	<op.tndyf7uvd64u53@e500>
Message-ID: <20070207093345.5AF9.JCARLSON@uci.edu>


"Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
> Other non-python stuff is taking my time, but I'll reply to this anyway.  
> Better late than never (in this case).
> 
> On Thu, 01 Feb 2007 16:19:00 +0100, Lucio Torre <lucio.torre at gmail.com>  
> wrote:
> 
> > On 1/31/07, Jan Kanis <jan.kanis at phil.uu.nl> wrote:
> >> >
> >> > Isnt the other argument the fact that always re evaluating will bring
> >> > similar gotchas?
> >>
> >> IMO, it won't. Caching will have to be implemented in a different way,  
> >> but
> >> that wouldn't really be a gotcha (except for python 2.x programmers who
> >> didn't read the "what's new in python 3.0")
> >>
> >
> > Maybe is better to annoy newbies that didnt read the python FAQ than
> > seasoned python 2.x programmers! :)
> 
> That statement I can agree with, but it isn't really applicable here. The  
> applicable statement reads: "Maybe it's better to annoy newbies that  
> didn't read the python FAQ than python 2.x programmers who don't know  
> anything about 3.0 and assume it's just another new version of python."
> And I don't agree with that one. :)
> 
> >>  If the
> >> function depends on other state I'd imagine that most of the time you'd
> >> want a change in state to be reflected in the functions default value.
> >>
> >> example:
> >> def foo(x = exec_sql("SELECT someval FROM defaults_table WHERE
> >> some_expr")):
> >>     # do something
> >>
> >
> > i was thinking more in the lines of the case when you dont want the
> > default to change because something has changed.
> 
> Then don't make the default depend on something you're going to change.
> 
> Well, what this is really about is about closure vars changing when the  
> outer scope changes them instead of the value of the closure vars being  
> captured by the new lexical scope. This has been proposed before (to  
> change this behaviour on closure variables in general) and rejected.  
> (though I am in favour of it)
> Python is a late binding language in just about everything, and that is  
> considered to be a good thing. The point you're making is exactly the  
> early-vs-late binding discussion.
> 
> > I still think this would be optimizing towards the case where you
> > should put the code in the body. What if i want a static default
> > parameter, never to change, but i want to compute it at compile time?
> > Should i need to use extra syntax for that? Isnt this case more common
> > than the others? Something like:
> >
> > def foo( pixels = (640*480) ): pass
> >
> > Should overriding '*' change that default value? (can you do that in
> > python3k? i think you cant in 2.x)
> 
> I'm not really sure if you can either. Anyway, the only cases I can think  
> of when this makes sense is when there's a bug in '*' and you want to  
> patch it, or if you're overloading '*' to work with other values, without  
> changing the meaning for ints. In both of these cases it would be a good  
> resp. neutral thing to late-bind '*'

No.  You cannot change the meaning of any operation on built-in types.
You can subclass the built-in types and add your own implementation of a
__magic__ method, or you can prefix your other operations with some
other type that does something else (X*640*480), but you can't
monkey-patch int to have a new __mul__ operation.

In Python, the functions that get called when an operation is called on
an object defined in C are "bound early", that is, they cannot be
changed during runtime (unless you write some C code to monkey about
with Python internals - which is a *bad* idea). The functions that get
called when an operation is called on an object defined in Python may be
"bound early", as is the case with properties and other descriptors on
new-style classes, but classic classes, generally speaking, can be
manipulated at will (though you sometimes need to be careful when adding
or changing new instance, class, and static methods).


> Yup. But let's not fill the language with gotchas for the newbies to find  
> and learn from :)

No one is advocating for *adding* gotchas to the language; this thread
is more or less a question of whether this particular default argument
gotcha is worth changing the language.


> You'll get to like the new way :)

And you will get used to and like the old way.


 - Josiah



From rrr at ronadam.com  Thu Feb  8 01:53:54 2007
From: rrr at ronadam.com (Ron Adam)
Date: Wed, 07 Feb 2007 18:53:54 -0600
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702071121w4e6b55e9g615a5284abfc3c2d@mail.gmail.com>
References: <45C63363.2080100@ronadam.com>
	<bbaeab100702071121w4e6b55e9g615a5284abfc3c2d@mail.gmail.com>
Message-ID: <45CA74A2.4060401@ronadam.com>

Brett Cannon wrote:
> On 2/4/07, Ron Adam <rrr at ronadam.com> wrote:
>>
>> After exploring this a bit further on comp.lang.python, I was able to 
>> organize
>> these ideas better.  The more I thought about it, the more '+'s I 
>> found, and
>> about the only '-'s I can think of is the work required to actually 
>> make a patch
>> to do it.
>>
>> It's also good to keep in mind that since most people still rely on 
>> the old
>> relative import behavior, most people have not run into some of the 
>> issues I
>> mention here.  But they will at some point.
>>
>> I did mean to keep this short, but clarity won out. (At least it's 
>> clear to me,
>> but that's an entirely subjective opinion on my part.)
>>
>> Maybe someone will adopt this and make a real PEP out of it.  :-)
>>
>> Cheers,
>>    Ron
>>
>>
>>
>> PROPOSAL
>> ========
>>
>> Make pythons concept of a package, (currently an informal type), be 
>> stronger
>> than that of the underlying file system search path and directory 
>> structure.
>>
> 
> So you mean make packages more of an official thing than just having a
> __path__ attribute on a module, right?

Currently in python 2.5, __path__ attributes are only in the imported package 
name spaces.  Running a module doesn't set a __path__ attribute, just the 
__file__ attribute.

It would be nice if __path__ were set on all modules in packages no matter how 
they are started.  The real name could be worked out by comparing __path__ and 
__file__ if someone needs that.  But I think it would be better to just go ahead 
and add a __realname__ attribute for when __name__ is "__main__".

__name__ == "__main__" can stay the same and still serve it's purpose to tell 
weather a script was started directly or imported.



>> Where the following hold true in python 3.X, or when absolute_import 
>> behavior is
>> imported from __future__ in python 2.X:
>>
>>
>> (1) Python first determines if a module or package is part of a 
>> package and then
>> runs that module or package in the context of the package they belong 
>> to. (see
>> items below)
>>
> 
> Don't quite follow this statement.  What do you mean by "runs" here?
> You mean when using runpy or something and having the name set to
> '__main__'?

Yes


>> (2)  import this_package.module
>>       import this_package.sub_package
>>
>> If this_package is the same name as the current package, then do not 
>> look on
>> sys.path. Use the location of this_package.
>>
> 
> Already does this (at least in my pure Python implementation).
> Searches are done on __path__ when you are within a package.

Cool! I don't think it's like that for the non-pure version, but it may do it 
that way if
"from __future__ import absolute_import" is used.

Are you setting __path__ for each module imported in a package too?


>> (3)  import other_package.module
>>       import other_package.sub_package
>>
>> If other_package is a different name from the current package 
>> (this_package),
>> then do not look in this_package and exclude searches in sys.path 
>> locations that
>> are inside this_package including the current directory.
>
> 
> This change would require importers to do more.  Since the absolute
> import semantics automatically make this kind of import start at the
> top-level (i.e., sys.path), each import for an entry on sys.path would
> need to be told what package it is currently in, check if it handles
> that package, and then skip it if it does have it.

I don't think it will be as hard as this.  See below.


> That seems like a lot of work that I know I don't want to have to
> implement for every importer I ever write.

Only getting the correct package location for the first module executed in the 
package will be a bit of work. (But not that much.) After that, it can be passed 
around.

Here's something I used recently to get the full dotted name without importing. 
It could also return the base package path as well.  You probably don't need the 
cache.  These could be combined and shortened further for just finding a root 
package location.


def path_type(path):
     """ Determine what kind of thing path is.

         Returns  ->  'module'|'package'|'dir'| None
     """
     if os.path.isfile(path) \
         and  (path[-3:] == '.py' or \
               path[-4:] in ('.pyw', '.pyc', '.pyd', '.pyo')):
         return 'module'
     if os.path.isdir(path):
         for end in ['', 'w', 'c', 'o']:
             if os.path.isfile(os.path.join(path, '__init__.py' + end)):
                 return 'package'
         return 'dir'

def dotted_name(path, cache={}):
     """ Get a full dotted module or package name from a path name.

         Returns  ->  fully qualified (dotted) name | None
     """
     if path in cache:
         return cache[path]
     if path_type(path) in ('package', 'module'):
         parent, name = os.path.split(path)
         name, _ = os.path.splitext(name)
         while 1:
             if path_type(parent) != 'package':
                 break
             parent, nextname = os.path.split(parent)
             name = '.'.join([nextname, name])
         cache[path] = name
         return name



lets.. see  (untested)

def package_path(path):
     """ Get the package location of a module.
     """
     package = None
     if path_type(path) in ('package', 'module'):
         parent, name = os.path.split(path)
         while 1:
             if path_type(parent) != 'package':
                 break
	    package = os.path.join(parent, name)
             parent, name = os.path.split(parent)
     return package



>> (4)  import module
>>       import package
>>
>> Module and package are not in a package, so don't look in any 
>> packages, even
>> this one or sys.path locations inside of packages.
>>
> 
> This is already done.  Absolute imports would cause this to do a
> shallow check on sys.path for the module or package name.

Great! 2 down.  Almost half way there.  :-)

But will it check the current directory if you run a module directly because 
currently it doesn't know if it's part of a package.  Is that correct?


>> (5) For behaviors other than these, like when you do actually want to 
>> run a
>> module belonging to a package in a different context, a mechanism such 
>> as a
>> command line switch, or a settable import attribute should be used.
>>
>>
>> MOTIVATION
>> ==========
>>
>> (A) Added reliability.
>>
>> There will be much less chance of errors (silent or otherwise) due to
>> path/import conflicts which are sometimes difficult to diagnose.
>>
> 
> Probably, but I don't know if the implementation complexity warrants
> worrying about this.  But then again how many people have actually
> needed to implement the import machinery.  =)  I could be labeled as
> jaded.

Well, I know it's not an easy thing to do.  But it's not finding the paths and 
or weather files are modules etc... that is hard.  From what I understand the 
hard part is making it work so it can be extended and customized.

Is that correct?


>> There may also be some added security benefits as well because it 
>> would much
>> harder for someone to create a same named module or package and insert 
>> it by
>> putting it on the path. Or by altering sys.path to do the same. [*]
>>
>> [* - If this can happen there are probably more serious security 
>> issues, but not
>> everyone has the most secure setup, so this point is still probably a 
>> good
>> point. General reliable execution of modules is the first concern, 
>> this may be a
>> side benefit of that.]
>>
>>
>> (B) Reduce the need for special checks and editing sys.path.
>>
>> Currently some authors have edit sys.path or do special if 
>> os.path.exists()
>> checks to ensure proper operations in some situations such as running 
>> tests.
>> These suggestions would reduce the need for such special testing and 
>> modifications.
>>
> 
> This might minimize some sys.path hacks in some instances, but it also
> complicates imports overall in terms of implementation and semantics.

I'm not sure why it would make it so much more complicated.  The contexts for 
which the imports are done will need to be done for cases of package imports, 
relative package imports, and modules in any case.  It's just a matter of 
determining which one to use from the start.  I guess I need to look into how 
pythons imports work in a little more detail.

> Where is point C?

Woops... I could make one up if you really want one.  ;-)


(It was moved elsewhere and I forgot to reletter.)


>> (D) Easier editing and testing.
>>
>> While you are editing modules in a package, you could then run the module
>> directly (as you can with old style relative imports) and still get 
>> the correct
>> package-relative behavior instead of something else. (like an 
>> exception or wrong
>> output). Many editors support running the file being edited, including 
>> idle.
>> It's also can be difficult to write scripts for the editors to 
>> determine the
>> correct context to run a module in.
>>
> 
> How is this directly solved, though?  You mentioned "running" a module
> as if it is in a package, but there is no direct explanation of how
> you would want to change the import machinery to pull this off.
> Basically you need a way to have either modules with the name __main__
> be able to get the canonical name for import purposes.  Or you need to
> leave __name__ alone and set some other global or something to flag
> that it is the __main__ module.

Leave __name__ alone, yes.  Add a __path__ attribute for all modules that is set 
to the base package location. Add a __realname__ attribute only to modules who's 
__name__ is set to '__main__'.

The import machinery could then use those to determine how to handle imports in 
that module.

Is that clearer?

If __path__ exists, then it's module in a package.
If __realname__ exists, then it was run as a script, but here's the actual name 
anyway.

If __name__ is '__main__' then do what scripts do when __name__ == '__main__'.



> Regardless, I am not seeing how you are proposing to go about solving
> this problem.

Discussing it is a good start to doing that,  isn't it?   ;-)


> I understand the desire to fix this __main__ issue with absolute
> imports and I totally support it, but I just need a more concrete
> solution in front of me (assuming I am not totally blind and it is
> actually in this doc).
> 
> -Brett

I only outlined the behavioral rules that need to be probably first agreed on. 
After that its a matter of writing it.  As I said above these behaviors are not 
the hard part, making it extendable in a nice clean way is.

Cheers,
Ron


From brett at python.org  Thu Feb  8 07:18:21 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 7 Feb 2007 22:18:21 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <45CA74A2.4060401@ronadam.com>
References: <45C63363.2080100@ronadam.com>
	<bbaeab100702071121w4e6b55e9g615a5284abfc3c2d@mail.gmail.com>
	<45CA74A2.4060401@ronadam.com>
Message-ID: <bbaeab100702072218o51bef3e3h74110fd8098eae05@mail.gmail.com>

On 2/7/07, Ron Adam <rrr at ronadam.com> wrote:
> Brett Cannon wrote:
> > On 2/4/07, Ron Adam <rrr at ronadam.com> wrote:
> >>
> >> After exploring this a bit further on comp.lang.python, I was able to
> >> organize
> >> these ideas better.  The more I thought about it, the more '+'s I
> >> found, and
> >> about the only '-'s I can think of is the work required to actually
> >> make a patch
> >> to do it.
> >>
> >> It's also good to keep in mind that since most people still rely on
> >> the old
> >> relative import behavior, most people have not run into some of the
> >> issues I
> >> mention here.  But they will at some point.
> >>
> >> I did mean to keep this short, but clarity won out. (At least it's
> >> clear to me,
> >> but that's an entirely subjective opinion on my part.)
> >>
> >> Maybe someone will adopt this and make a real PEP out of it.  :-)
> >>
> >> Cheers,
> >>    Ron
> >>
> >>
> >>
> >> PROPOSAL
> >> ========
> >>
> >> Make pythons concept of a package, (currently an informal type), be
> >> stronger
> >> than that of the underlying file system search path and directory
> >> structure.
> >>
> >
> > So you mean make packages more of an official thing than just having a
> > __path__ attribute on a module, right?
>
> Currently in python 2.5, __path__ attributes are only in the imported package
> name spaces.  Running a module doesn't set a __path__ attribute, just the
> __file__ attribute.
>

True.

> It would be nice if __path__ were set on all modules in packages no matter how
> they are started.

There is a slight issue with that as the __path__ attribute represents
the top of a package and thus that it has an __init__ module.  It has
some significance in terms of how stuff works at the moment.

>  The real name could be worked out by comparing __path__ and
> __file__ if someone needs that.  But I think it would be better to just go ahead
> and add a __realname__ attribute for when __name__ is "__main__".
>
> __name__ == "__main__" can stay the same and still serve it's purpose to tell
> weather a script was started directly or imported.
>

I think the whole __main__ thing is the wrong thing to be trying to
keep alive for this.  I know it would break things, but it is probably
better to come up with a better way for a module to know when it is
being executed or do denote what code should only be run when it is
executed.

>
>
> >> Where the following hold true in python 3.X, or when absolute_import
> >> behavior is
> >> imported from __future__ in python 2.X:
> >>
> >>
> >> (1) Python first determines if a module or package is part of a
> >> package and then
> >> runs that module or package in the context of the package they belong
> >> to. (see
> >> items below)
> >>
> >
> > Don't quite follow this statement.  What do you mean by "runs" here?
> > You mean when using runpy or something and having the name set to
> > '__main__'?
>
> Yes
>
>
> >> (2)  import this_package.module
> >>       import this_package.sub_package
> >>
> >> If this_package is the same name as the current package, then do not
> >> look on
> >> sys.path. Use the location of this_package.
> >>
> >
> > Already does this (at least in my pure Python implementation).
> > Searches are done on __path__ when you are within a package.
>
> Cool! I don't think it's like that for the non-pure version, but it may do it
> that way if
> "from __future__ import absolute_import" is used.
>

It does do it both ways, there is just a fallback on the classic
import semantics in terms of trying it both as a relative and absolute
import.  But I got the semantics from the current implementation so it
is not some great inspiration of mine.  =)

> Are you setting __path__ for each module imported in a package too?
>

No.  As I said above, having __path__ set has some special meaning in
how imports work at the moment.  It stays on packages and not modules
within packages.

>
> >> (3)  import other_package.module
> >>       import other_package.sub_package
> >>
> >> If other_package is a different name from the current package
> >> (this_package),
> >> then do not look in this_package and exclude searches in sys.path
> >> locations that
> >> are inside this_package including the current directory.
> >
> >
> > This change would require importers to do more.  Since the absolute
> > import semantics automatically make this kind of import start at the
> > top-level (i.e., sys.path), each import for an entry on sys.path would
> > need to be told what package it is currently in, check if it handles
> > that package, and then skip it if it does have it.
>
> I don't think it will be as hard as this.  See below.
>
>
> > That seems like a lot of work that I know I don't want to have to
> > implement for every importer I ever write.
>
> Only getting the correct package location for the first module executed in the
> package will be a bit of work. (But not that much.) After that, it can be passed
> around.
>
> Here's something I used recently to get the full dotted name without importing.
> It could also return the base package path as well.  You probably don't need the
> cache.  These could be combined and shortened further for just finding a root
> package location.
>
>
> def path_type(path):
>      """ Determine what kind of thing path is.
>
>          Returns  ->  'module'|'package'|'dir'| None
>      """
>      if os.path.isfile(path) \
>          and  (path[-3:] == '.py' or \
>                path[-4:] in ('.pyw', '.pyc', '.pyd', '.pyo')):
>          return 'module'
>      if os.path.isdir(path):
>          for end in ['', 'w', 'c', 'o']:
>              if os.path.isfile(os.path.join(path, '__init__.py' + end)):
>                  return 'package'
>          return 'dir'
>
> def dotted_name(path, cache={}):
>      """ Get a full dotted module or package name from a path name.
>
>          Returns  ->  fully qualified (dotted) name | None
>      """
>      if path in cache:
>          return cache[path]
>      if path_type(path) in ('package', 'module'):
>          parent, name = os.path.split(path)
>          name, _ = os.path.splitext(name)
>          while 1:
>              if path_type(parent) != 'package':
>                  break
>              parent, nextname = os.path.split(parent)
>              name = '.'.join([nextname, name])
>          cache[path] = name
>          return name
>
>
>
> lets.. see  (untested)
>
> def package_path(path):
>      """ Get the package location of a module.
>      """
>      package = None
>      if path_type(path) in ('package', 'module'):
>          parent, name = os.path.split(path)
>          while 1:
>              if path_type(parent) != 'package':
>                  break
>             package = os.path.join(parent, name)
>              parent, name = os.path.split(parent)
>      return package
>

Or you could have copied the code I wrote for the filesystem
importer's find_module method that already does this classification.
=)

Part of the problem of working backwards from path to dotted name is
that it might not import that way.  __path__ can be tweaked, importers
and loaders can be written to interpret the directory structure or
file names differently, etc.  Plus what about different file types
like .ptl files from Quixote?

>
>
> >> (4)  import module
> >>       import package
> >>
> >> Module and package are not in a package, so don't look in any
> >> packages, even
> >> this one or sys.path locations inside of packages.
> >>
> >
> > This is already done.  Absolute imports would cause this to do a
> > shallow check on sys.path for the module or package name.
>
> Great! 2 down.  Almost half way there.  :-)
>
> But will it check the current directory if you run a module directly because
> currently it doesn't know if it's part of a package.  Is that correct?
>

Absolute import semantics go straight to sys.path, period.

>
> >> (5) For behaviors other than these, like when you do actually want to
> >> run a
> >> module belonging to a package in a different context, a mechanism such
> >> as a
> >> command line switch, or a settable import attribute should be used.
> >>
> >>
> >> MOTIVATION
> >> ==========
> >>
> >> (A) Added reliability.
> >>
> >> There will be much less chance of errors (silent or otherwise) due to
> >> path/import conflicts which are sometimes difficult to diagnose.
> >>
> >
> > Probably, but I don't know if the implementation complexity warrants
> > worrying about this.  But then again how many people have actually
> > needed to implement the import machinery.  =)  I could be labeled as
> > jaded.
>
> Well, I know it's not an easy thing to do.  But it's not finding the paths and
> or weather files are modules etc... that is hard.  From what I understand the
> hard part is making it work so it can be extended and customized.
>
> Is that correct?
>

Yes.  I really think ditching this whole __main__ name thing is going
to be the only solid solution.  Defining a __main__() method for
modules that gets executed makes the most sense to me.  Just import
the module and then execute the function if it exists.  That allow
runpy to have the name be set properly and does away with import
problems without mucking with import semantics.  Still have the name
problem if you specify a file directly on the command line, though.

>
> >> There may also be some added security benefits as well because it
> >> would much
> >> harder for someone to create a same named module or package and insert
> >> it by
> >> putting it on the path. Or by altering sys.path to do the same. [*]
> >>
> >> [* - If this can happen there are probably more serious security
> >> issues, but not
> >> everyone has the most secure setup, so this point is still probably a
> >> good
> >> point. General reliable execution of modules is the first concern,
> >> this may be a
> >> side benefit of that.]
> >>
> >>
> >> (B) Reduce the need for special checks and editing sys.path.
> >>
> >> Currently some authors have edit sys.path or do special if
> >> os.path.exists()
> >> checks to ensure proper operations in some situations such as running
> >> tests.
> >> These suggestions would reduce the need for such special testing and
> >> modifications.
> >>
> >
> > This might minimize some sys.path hacks in some instances, but it also
> > complicates imports overall in terms of implementation and semantics.
>
> I'm not sure why it would make it so much more complicated.  The contexts for
> which the imports are done will need to be done for cases of package imports,
> relative package imports, and modules in any case.  It's just a matter of
> determining which one to use from the start.  I guess I need to look into how
> pythons imports work in a little more detail.
>
> > Where is point C?
>
> Woops... I could make one up if you really want one.  ;-)
>

No, that's okay.  =)

>
> (It was moved elsewhere and I forgot to reletter.)
>
>
> >> (D) Easier editing and testing.
> >>
> >> While you are editing modules in a package, you could then run the module
> >> directly (as you can with old style relative imports) and still get
> >> the correct
> >> package-relative behavior instead of something else. (like an
> >> exception or wrong
> >> output). Many editors support running the file being edited, including
> >> idle.
> >> It's also can be difficult to write scripts for the editors to
> >> determine the
> >> correct context to run a module in.
> >>
> >
> > How is this directly solved, though?  You mentioned "running" a module
> > as if it is in a package, but there is no direct explanation of how
> > you would want to change the import machinery to pull this off.
> > Basically you need a way to have either modules with the name __main__
> > be able to get the canonical name for import purposes.  Or you need to
> > leave __name__ alone and set some other global or something to flag
> > that it is the __main__ module.
>
> Leave __name__ alone, yes.  Add a __path__ attribute for all modules that is set
> to the base package location. Add a __realname__ attribute only to modules who's
> __name__ is set to '__main__'.
>

I don't like this idea of having one attribute have the same meaning
as another attribute.  I don't think a good backwards-compatible
solution is going to crop up.

> The import machinery could then use those to determine how to handle imports in
> that module.
>
> Is that clearer?
>

It is, but I don't like it.  =)

> If __path__ exists, then it's module in a package.
> If __realname__ exists, then it was run as a script, but here's the actual name
> anyway.
>
> If __name__ is '__main__' then do what scripts do when __name__ == '__main__'.
>
>
>
> > Regardless, I am not seeing how you are proposing to go about solving
> > this problem.
>
> Discussing it is a good start to doing that,  isn't it?   ;-)
>

Yep.

-Brett


From brett at python.org  Wed Feb  7 20:21:08 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 7 Feb 2007 11:21:08 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <45C63363.2080100@ronadam.com>
References: <45C63363.2080100@ronadam.com>
Message-ID: <bbaeab100702071121w4e6b55e9g615a5284abfc3c2d@mail.gmail.com>

On 2/4/07, Ron Adam <rrr at ronadam.com> wrote:
>
> After exploring this a bit further on comp.lang.python, I was able to organize
> these ideas better.  The more I thought about it, the more '+'s I found, and
> about the only '-'s I can think of is the work required to actually make a patch
> to do it.
>
> It's also good to keep in mind that since most people still rely on the old
> relative import behavior, most people have not run into some of the issues I
> mention here.  But they will at some point.
>
> I did mean to keep this short, but clarity won out. (At least it's clear to me,
> but that's an entirely subjective opinion on my part.)
>
> Maybe someone will adopt this and make a real PEP out of it.  :-)
>
> Cheers,
>    Ron
>
>
>
> PROPOSAL
> ========
>
> Make pythons concept of a package, (currently an informal type), be stronger
> than that of the underlying file system search path and directory structure.
>

So you mean make packages more of an official thing than just having a
__path__ attribute on a module, right?

>
> Where the following hold true in python 3.X, or when absolute_import behavior is
> imported from __future__ in python 2.X:
>
>
> (1) Python first determines if a module or package is part of a package and then
> runs that module or package in the context of the package they belong to. (see
> items below)
>

Don't quite follow this statement.  What do you mean by "runs" here?
You mean when using runpy or something and having the name set to
'__main__'?

>
> (2)  import this_package.module
>       import this_package.sub_package
>
> If this_package is the same name as the current package, then do not look on
> sys.path. Use the location of this_package.
>

Already does this (at least in my pure Python implementation).
Searches are done on __path__ when you are within a package.

>
> (3)  import other_package.module
>       import other_package.sub_package
>
> If other_package is a different name from the current package (this_package),
> then do not look in this_package and exclude searches in sys.path locations that
> are inside this_package including the current directory.
>

This change would require importers to do more.  Since the absolute
import semantics automatically make this kind of import start at the
top-level (i.e., sys.path), each import for an entry on sys.path would
need to be told what package it is currently in, check if it handles
that package, and then skip it if it does have it.

That seems like a lot of work that I know I don't want to have to
implement for every importer I ever write.

>
> (4)  import module
>       import package
>
> Module and package are not in a package, so don't look in any packages, even
> this one or sys.path locations inside of packages.
>

This is already done.  Absolute imports would cause this to do a
shallow check on sys.path for the module or package name.

>
> (5) For behaviors other than these, like when you do actually want to run a
> module belonging to a package in a different context, a mechanism such as a
> command line switch, or a settable import attribute should be used.
>
>
> MOTIVATION
> ==========
>
> (A) Added reliability.
>
> There will be much less chance of errors (silent or otherwise) due to
> path/import conflicts which are sometimes difficult to diagnose.
>

Probably, but I don't know if the implementation complexity warrants
worrying about this.  But then again how many people have actually
needed to implement the import machinery.  =)  I could be labeled as
jaded.

> There may also be some added security benefits as well because it would much
> harder for someone to create a same named module or package and insert it by
> putting it on the path. Or by altering sys.path to do the same. [*]
>
> [* - If this can happen there are probably more serious security issues, but not
> everyone has the most secure setup, so this point is still probably a good
> point. General reliable execution of modules is the first concern, this may be a
> side benefit of that.]
>
>
> (B) Reduce the need for special checks and editing sys.path.
>
> Currently some authors have edit sys.path or do special if os.path.exists()
> checks to ensure proper operations in some situations such as running tests.
> These suggestions would reduce the need for such special testing and modifications.
>

This might minimize some sys.path hacks in some instances, but it also
complicates imports overall in terms of implementation and semantics.

Where is point C?
>
> (D) Easier editing and testing.
>
> While you are editing modules in a package, you could then run the module
> directly (as you can with old style relative imports) and still get the correct
> package-relative behavior instead of something else. (like an exception or wrong
> output). Many editors support running the file being edited, including idle.
> It's also can be difficult to write scripts for the editors to determine the
> correct context to run a module in.
>

How is this directly solved, though?  You mentioned "running" a module
as if it is in a package, but there is no direct explanation of how
you would want to change the import machinery to pull this off.
Basically you need a way to have either modules with the name __main__
be able to get the canonical name for import purposes.  Or you need to
leave __name__ alone and set some other global or something to flag
that it is the __main__ module.

Regardless, I am not seeing how you are proposing to go about solving
this problem.

I understand the desire to fix this __main__ issue with absolute
imports and I totally support it, but I just need a more concrete
solution in front of me (assuming I am not totally blind and it is
actually in this doc).

-Brett


From stephenemslie at gmail.com  Thu Feb  8 11:44:20 2007
From: stephenemslie at gmail.com (stephen emslie)
Date: Thu, 8 Feb 2007 10:44:20 +0000
Subject: [Python-ideas] tab completion in pdb
In-Reply-To: <51f97e530701220357sdc7c1d8nab5dfebca2aae605@mail.gmail.com>
References: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com>
	<20070119173804.GA8679@panix.com>
	<51f97e530701220357sdc7c1d8nab5dfebca2aae605@mail.gmail.com>
Message-ID: <51f97e530702080244o204d6413sa8981fa24a3f4ffa@mail.gmail.com>

There has been some discussion about this off-list. Most of the
discussion has centered around what appropriate tab completion should
be in different cases (what should be offered in the completion
namespace and when).

pydb has been dealt with a number of these issues already (thanks
Rocky Bernstein).

I would like to continue the discussion here. Here's a quick summary:
 - when a line is blank, pdb commands and valid identifiers and
keywords should be included in the namespace.
 - if a line does not start with a pdb command, then it is a python
expression and can be completed by rlcompleter, with all valid
identifiers and keywords available in the namespace.
 - if a line does start with a pdb command then we should decide on
the best way to complete each pdb command.

for example:
" "[complete] -> all possible commands, identifiers and keywords
"b"[complete] -> "basestring  bool        break       buffer"
"basestring."[complete] -> "basestring.__base__
basestring.__delattr__ ... etc."
"break "[complete] -> whatever the pdb completer will offer

currently, the submitted patch only attempts to complete python
expressions with rlcompleter. I think it would be useful (and more
honest, as Rocky puts it) to offer completion for pdb commands as
well. Apart from general comments, what would be great are suggestions
on what sort of completion should follow the various pdb commands.

Stephen Emslie

On 1/22/07, stephen emslie <stephenemslie at gmail.com> wrote:
> Thanks for taking a look. I've created a patch relative to pdb.py in svn and submitted it to sourceforge here:
>
>  http://sourceforge.net/tracker/index.php?func=detail&aid=1641544&group_id=5470&atid=305470
>
>
>
> On 1/19/07, Aahz < aahz at pythoncraft.com> wrote:
> > On Fri, Jan 19, 2007, stephen emslie wrote:
> > >
> > > I've attached a patch to pdb.py (on Python 2.4.4c1). The only real
> > > difference to rlcompleter's default complete method is that because pdb
> > > changes scope as you step through a program, rlcompleter's namespace is
> > > updated to reflect the current local and global namespace.
> > >
> > > This is my first attempt at a python patch. Any suggestions or improvements
> > > are welcome.
> >
> > Please go ahead and upload this patch to SourceForge to make sure it
> > doesn't get lost.  Thanks!
> > --
> > Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/
> >
> > Help a hearing-impaired person:  http://rule6.info/hearing.html
> >
>
>


From stephenemslie at gmail.com  Thu Feb  8 11:49:32 2007
From: stephenemslie at gmail.com (stephen emslie)
Date: Thu, 8 Feb 2007 10:49:32 +0000
Subject: [Python-ideas] tab completion in pdb
In-Reply-To: <51f97e530702080244o204d6413sa8981fa24a3f4ffa@mail.gmail.com>
References: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com>
	<20070119173804.GA8679@panix.com>
	<51f97e530701220357sdc7c1d8nab5dfebca2aae605@mail.gmail.com>
	<51f97e530702080244o204d6413sa8981fa24a3f4ffa@mail.gmail.com>
Message-ID: <51f97e530702080249t50787485i3d77dab3a82d600d@mail.gmail.com>

I think its worth forwarding a cautionary snippet from Rocky that
makes a lot of sense:

On 2/4/07, R. Bernstein <rocky at panix.com> wrote:
> The caveat though in all of these cases is that it can be difficult to
> get exactly what the valid completions are. For example if you know
> you need an number, do you really want to complete on all objects that
> are a numeric type plus the digits 0-9, "+", "-", and "("? And also
> throw in functions which return a number?  So in general, I think one
> has to back off a bit. And there are two ways to back off: one is
> allowing more possible completions (e.g. any object) and another by
> allowing less. I've been offering less. But I think really it's
> dictated by what "feels" right which will no doubt be different for
> different people. "Feeling right" could also be determined by how long
> completion takes is which may vary in different situations on
> different computers.
>
> Handling command completion feels more like engineering than art or
> science.
>


From rhamph at gmail.com  Thu Feb  8 14:01:58 2007
From: rhamph at gmail.com (Adam Olsen)
Date: Thu, 8 Feb 2007 06:01:58 -0700
Subject: [Python-ideas] Packages and Import
In-Reply-To: <45C63363.2080100@ronadam.com>
References: <45C63363.2080100@ronadam.com>
Message-ID: <aac2c7cb0702080501q67e8729ejb17f0c382099538f@mail.gmail.com>

On 2/4/07, Ron Adam <rrr at ronadam.com> wrote:
>
> After exploring this a bit further on comp.lang.python, I was able to organize
> these ideas better.  The more I thought about it, the more '+'s I found, and
> about the only '-'s I can think of is the work required to actually make a patch
> to do it.
>
> It's also good to keep in mind that since most people still rely on the old
> relative import behavior, most people have not run into some of the issues I
> mention here.  But they will at some point.
>
> I did mean to keep this short, but clarity won out. (At least it's clear to me,
> but that's an entirely subjective opinion on my part.)
>
> Maybe someone will adopt this and make a real PEP out of it.  :-)

For all the complexity of module attributes and global import hooks I
think there's something that'll start to strip some of it away: make
__import__() into a method of ModuleType, then ensure there's a way to
load python modules using subclasses of ModuleType.

You could have entirely different semantics, independent of
sys.modules.  You could load from a zip file that's entirely private,
while still allowing modules within it to reach each other using
relative imports.

Once you break free of sys.modules and global hooks it becomes much
easier to design something to replace them.

I have some ideas on how to do that, but they don't seem nearly as
important as this base functionality.

-- 
Adam Olsen, aka Rhamphoryncus


From rrr at ronadam.com  Thu Feb  8 19:31:49 2007
From: rrr at ronadam.com (Ron Adam)
Date: Thu, 08 Feb 2007 12:31:49 -0600
Subject: [Python-ideas] Packages and Import
In-Reply-To: <aac2c7cb0702080501q67e8729ejb17f0c382099538f@mail.gmail.com>
References: <45C63363.2080100@ronadam.com>
	<aac2c7cb0702080501q67e8729ejb17f0c382099538f@mail.gmail.com>
Message-ID: <45CB6C95.3090907@ronadam.com>

Adam Olsen wrote:
> On 2/4/07, Ron Adam <rrr at ronadam.com> wrote:
>>
>> After exploring this a bit further on comp.lang.python, I was able to 
>> organize
>> these ideas better.  The more I thought about it, the more '+'s I 
>> found, and
>> about the only '-'s I can think of is the work required to actually 
>> make a patch
>> to do it.
>>
>> It's also good to keep in mind that since most people still rely on 
>> the old
>> relative import behavior, most people have not run into some of the 
>> issues I
>> mention here.  But they will at some point.
>>
>> I did mean to keep this short, but clarity won out. (At least it's 
>> clear to me,
>> but that's an entirely subjective opinion on my part.)
>>
>> Maybe someone will adopt this and make a real PEP out of it.  :-)
> 
> For all the complexity of module attributes and global import hooks I
> think there's something that'll start to strip some of it away: make
> __import__() into a method of ModuleType, then ensure there's a way to
> load python modules using subclasses of ModuleType.

Seems to me, weather the importer is part of this module or the new module 
doesn't make a difference.  Just a matter of ordering.


So if you want to modify the import mechanism:

Would create a new ModuleType and modify it's import mechansims and then pass it 
a name?

Or would it be better to subclass this modules import, modify it, pass it along 
with a name (to be imported) to the new ModuleType?

Or modify this modules import and then subclass this ModuleType, which will 
inherit the new importer.  So that ModuleType(__name__) will then initialize it 
self?  (but this would inherit a lot of other things you may not want.)


> You could have entirely different semantics, independent of
> sys.modules.  You could load from a zip file that's entirely private,
> while still allowing modules within it to reach each other using
> relative imports.

> Once you break free of sys.modules and global hooks it becomes much
> easier to design something to replace them.
> 
> I have some ideas on how to do that, but they don't seem nearly as
> important as this base functionality.

Then you are further along than I am.

I just know what I want it to do.  But I am trying to learn about how the actual 
imports function.  Maybe in a few days I can give Brett some suggestions that 
are a little more concrete.

Cheers,
Ron









From brett at python.org  Thu Feb  8 20:43:43 2007
From: brett at python.org (Brett Cannon)
Date: Thu, 8 Feb 2007 11:43:43 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <aac2c7cb0702080501q67e8729ejb17f0c382099538f@mail.gmail.com>
References: <45C63363.2080100@ronadam.com>
	<aac2c7cb0702080501q67e8729ejb17f0c382099538f@mail.gmail.com>
Message-ID: <bbaeab100702081143s139e3e66vf90cbc6ca6434863@mail.gmail.com>

On 2/8/07, Adam Olsen <rhamph at gmail.com> wrote:
> On 2/4/07, Ron Adam <rrr at ronadam.com> wrote:
> >
> > After exploring this a bit further on comp.lang.python, I was able to organize
> > these ideas better.  The more I thought about it, the more '+'s I found, and
> > about the only '-'s I can think of is the work required to actually make a patch
> > to do it.
> >
> > It's also good to keep in mind that since most people still rely on the old
> > relative import behavior, most people have not run into some of the issues I
> > mention here.  But they will at some point.
> >
> > I did mean to keep this short, but clarity won out. (At least it's clear to me,
> > but that's an entirely subjective opinion on my part.)
> >
> > Maybe someone will adopt this and make a real PEP out of it.  :-)
>
> For all the complexity of module attributes and global import hooks I
> think there's something that'll start to strip some of it away: make
> __import__() into a method of ModuleType, then ensure there's a way to
> load python modules using subclasses of ModuleType.
>

And so __import__ is to be bound to what module's method?  Or is it a
class method?

> You could have entirely different semantics, independent of
> sys.modules.

You can have that now if you ignored 'reload'.  You just need to
delete a module from sys.modules as soon as you import.  Then you can
set up your imports to do what you want.

>  You could load from a zip file that's entirely private,
> while still allowing modules within it to reach each other using
> relative imports.

So are you saying that imports in a module are to use the module's
__import__ method?  So instantiate a module and then use it's method
to initialize the module itself?

-Brett


From rrr at ronadam.com  Thu Feb  8 21:41:11 2007
From: rrr at ronadam.com (Ron Adam)
Date: Thu, 08 Feb 2007 14:41:11 -0600
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702072218o51bef3e3h74110fd8098eae05@mail.gmail.com>
References: <45C63363.2080100@ronadam.com>	
	<bbaeab100702071121w4e6b55e9g615a5284abfc3c2d@mail.gmail.com>	
	<45CA74A2.4060401@ronadam.com>
	<bbaeab100702072218o51bef3e3h74110fd8098eae05@mail.gmail.com>
Message-ID: <45CB8AE7.9090808@ronadam.com>

Brett Cannon wrote:
> On 2/7/07, Ron Adam <rrr at ronadam.com> wrote:
>> Brett Cannon wrote:
>> > On 2/4/07, Ron Adam <rrr at ronadam.com> wrote:


>> It would be nice if __path__ were set on all modules in packages no 
>> matter how
>> they are started.
> 
> There is a slight issue with that as the __path__ attribute represents
> the top of a package and thus that it has an __init__ module.  It has
> some significance in terms of how stuff works at the moment.

Yes, and after some reading I found __path__ isn't exactly what I was thinking.

It could be it's only a matter of getting that first initial import right.  An 
example of this is this recipe by Nick.

     http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772



>>  The real name could be worked out by comparing __path__ and
>> __file__ if someone needs that.  But I think it would be better to 
>> just go ahead
>> and add a __realname__ attribute for when __name__ is "__main__".
>>
>> __name__ == "__main__" can stay the same and still serve it's purpose 
>> to tell
>> weather a script was started directly or imported.
> 
> I think the whole __main__ thing is the wrong thing to be trying to
> keep alive for this.  I know it would break things, but it is probably
> better to come up with a better way for a module to know when it is
> being executed or do denote what code should only be run when it is
> executed.

I was trying to suggest things that would do the least harm as far as changing 
things in the eyes of the users.  If not keeping the "__main__" name in python 
3k is a real option then yes, then there may be more options.  Is it a real 
option?  Or is Guido set on keeping it?

If you remove the "__main__" name, then you will still need to have some 
attribute for python to determine the same thing.  What you would end up doing 
is just moving the [if __name__=="__main__": __main__()] line off the end of 
program so that all program have it automatically.  We just won't see it.  And 
instead of checking __name__, the interpreter would check some other attribute.

So what and where would that other attribute be?

Would it be exposed so we add if __ismain__: <body> to our programs for 
initialization purposes?

Or you could just replace it with an __ismain__ attribute then we can name our 
main functions anyhthing we want... like test().

if __ismain__:
    test()

That is shorter and maybe less confusing than the __name__ check.


>> >> (2)  import this_package.module
>> >>       import this_package.sub_package
>> >>
>> >> If this_package is the same name as the current package, then do not
>> >> look on
>> >> sys.path. Use the location of this_package.
>> >>
>> >
>> > Already does this (at least in my pure Python implementation).
>> > Searches are done on __path__ when you are within a package.
>>
>> Cool! I don't think it's like that for the non-pure version, but it 
>> may do it
>> that way if
>> "from __future__ import absolute_import" is used.
> 
> It does do it both ways, there is just a fallback on the classic
> import semantics in terms of trying it both as a relative and absolute
> import.  But I got the semantics from the current implementation so it
> is not some great inspiration of mine.  =)

I think there shouldn't be a fall back.. that will just confuse things. Raise an 
exception here because most likely falling back is not what you want.

If someone wants to import an external to a package module with the same name as 
the package, (or modules in some other package with the same name), then there 
needs to be an explicit way to do that.  But I really don't think this will come 
up that often.


<clipped general examples>

> Or you could have copied the code I wrote for the filesystem
> importer's find_module method that already does this classification.
> =)
> 
> Part of the problem of working backwards from path to dotted name is
> that it might not import that way.  

Maybe it should work that way?  If someone wants other than that behavior, then 
maybe there can be other ways to get it?

Hers's an example of a situation where you might think it would be a problem, 
but it isn't:

     pkg1:
       __init__.py
       m1.py
       spkg1:
          __init__.py
          m3.py
       dirA:
          m4.py
          pkg2:
             __init__.py
             m5.py

You might think it wouldn't work for pkg2.m5, but that's actually ok.  pkg2 is a 
package just being stored in dirA which just happens to be located inside 
another package.

Running m5.py directly will run it as a submodule of pkg2, which is what you 
want.  It's not in a sub-package of pkg1.  And m4.py is just a regular module.

Or are you thinking of other relationships?


>__path__ can be tweaked, importers
> and loaders can be written to interpret the directory structure or
> file names differently, etc.

Yes, and they will need a basic set of well defined default behaviors to build 
on.  After that, it's up to them to be sure their interpretation does what they 
want.


> Plus what about different file types
> like .ptl files from Quixote?

This is really a matter of using a corresponding file reader to get at it's 
contents or it's real (relative to python) type... Ie, is it really a module, a 
package, or a module in a package, or some other thing ... living inside of a 
zip, or some other device (or file) like container?



>> >> (4)  import module
>> >>       import package
>> >>
>> >> Module and package are not in a package, so don't look in any
>> >> packages, even
>> >> this one or sys.path locations inside of packages.
>> >>
>> >
>> > This is already done.  Absolute imports would cause this to do a
>> > shallow check on sys.path for the module or package name.
>>
>> Great! 2 down.  Almost half way there.  :-)
>>
>> But will it check the current directory if you run a module directly 
>> because
>> currently it doesn't know if it's part of a package.  Is that correct?
> 
> Absolute import semantics go straight to sys.path, period.

Which includes the current directory.  So in effect it will fall back to a 
relative type of behavior if a module with the same name is being imported exist 
in the current, inside this package direcotry, *if* you execute the module directly.

I think this should also give an error, it is the inverse of the situation 
above. (#2) In most cases (if not all) it's not what you want.

You wanted a module that is not part of this modules package, and got one that is.


>> >> MOTIVATION
>> >> ==========
>> >>
>> >> (A) Added reliability.
>> >>
>> >> There will be much less chance of errors (silent or otherwise) due to
>> >> path/import conflicts which are sometimes difficult to diagnose.
>> >>
>> >
>> > Probably, but I don't know if the implementation complexity warrants
>> > worrying about this.  But then again how many people have actually
>> > needed to implement the import machinery.  =)  I could be labeled as
>> > jaded.
>>
>> Well, I know it's not an easy thing to do.  But it's not finding the 
>> paths and
>> or weather files are modules etc... that is hard.  From what I 
>> understand the
>> hard part is making it work so it can be extended and customized.
>>
>> Is that correct?
> 
> Yes.  I really think ditching this whole __main__ name thing is going
> to be the only solid solution.  Defining a __main__() method for
> modules that gets executed makes the most sense to me.  Just import
> the module and then execute the function if it exists.  That allow
> runpy to have the name be set properly and does away with import
> problems without mucking with import semantics.  Still have the name
> problem if you specify a file directly on the command line, though.

I'll have to see more details of how this would work I think. Part of me says
sound good. And another part says, isn't this just moving stuff around? And what
exactly does that solve?


>> The import machinery could then use those to determine how to handle 
>> imports in
>> that module.
>>
>> Is that clearer?
> 
> It is, but I don't like it.  =)

It does't exactly have to work that way.  ;-)


It's the "does it do what I designed it to do" behavioral stuff of packages and 
modules that I want.  If the module, however it is run gives an error or does 
something other than what I intended, then that's a problem.

Ron




From brett at python.org  Thu Feb  8 22:34:49 2007
From: brett at python.org (Brett Cannon)
Date: Thu, 8 Feb 2007 13:34:49 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <45CB8AE7.9090808@ronadam.com>
References: <45C63363.2080100@ronadam.com>
	<bbaeab100702071121w4e6b55e9g615a5284abfc3c2d@mail.gmail.com>
	<45CA74A2.4060401@ronadam.com>
	<bbaeab100702072218o51bef3e3h74110fd8098eae05@mail.gmail.com>
	<45CB8AE7.9090808@ronadam.com>
Message-ID: <bbaeab100702081334j7d5c4207ge6bd4453525e70ff@mail.gmail.com>

On 2/8/07, Ron Adam <rrr at ronadam.com> wrote:
> Brett Cannon wrote:
> > On 2/7/07, Ron Adam <rrr at ronadam.com> wrote:
> >> Brett Cannon wrote:
> >> > On 2/4/07, Ron Adam <rrr at ronadam.com> wrote:
>
>
> >> It would be nice if __path__ were set on all modules in packages no
> >> matter how
> >> they are started.
> >
> > There is a slight issue with that as the __path__ attribute represents
> > the top of a package and thus that it has an __init__ module.  It has
> > some significance in terms of how stuff works at the moment.
>
> Yes, and after some reading I found __path__ isn't exactly what I was thinking.
>
> It could be it's only a matter of getting that first initial import right.  An
> example of this is this recipe by Nick.
>
>      http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772
>

But Nick already rolled this stuff into 2.5 when package support was
added to runpy.

>
>
> >>  The real name could be worked out by comparing __path__ and
> >> __file__ if someone needs that.  But I think it would be better to
> >> just go ahead
> >> and add a __realname__ attribute for when __name__ is "__main__".
> >>
> >> __name__ == "__main__" can stay the same and still serve it's purpose
> >> to tell
> >> weather a script was started directly or imported.
> >
> > I think the whole __main__ thing is the wrong thing to be trying to
> > keep alive for this.  I know it would break things, but it is probably
> > better to come up with a better way for a module to know when it is
> > being executed or do denote what code should only be run when it is
> > executed.
>
> I was trying to suggest things that would do the least harm as far as changing
> things in the eyes of the users.  If not keeping the "__main__" name in python
> 3k is a real option then yes, then there may be more options.  Is it a real
> option?  Or is Guido set on keeping it?
>

Beats me.  Wouldn't be hard to have 2to change ``if __name__ ==
'__main__'`` to a function definition instead.

> If you remove the "__main__" name, then you will still need to have some
> attribute for python to determine the same thing.

Why?  There is nothing saying we can't follow most other languages and
just have a reserved function name that gets executed if the module is
executed.

>  What you would end up doing
> is just moving the [if __name__=="__main__": __main__()] line off the end of
> program so that all program have it automatically.  We just won't see it.  And
> instead of checking __name__, the interpreter would check some other attribute.
>
> So what and where would that other attribute be?
>

If a thing was done like that it would be in the global namespace of
the module just like __name__ is.

> Would it be exposed so we add if __ismain__: <body> to our programs for
> initialization purposes?
>
> Or you could just replace it with an __ismain__ attribute then we can name our
> main functions anyhthing we want... like test().
>
> if __ismain__:
>     test()
>
> That is shorter and maybe less confusing than the __name__ check.
>
>
> >> >> (2)  import this_package.module
> >> >>       import this_package.sub_package
> >> >>
> >> >> If this_package is the same name as the current package, then do not
> >> >> look on
> >> >> sys.path. Use the location of this_package.
> >> >>
> >> >
> >> > Already does this (at least in my pure Python implementation).
> >> > Searches are done on __path__ when you are within a package.
> >>
> >> Cool! I don't think it's like that for the non-pure version, but it
> >> may do it
> >> that way if
> >> "from __future__ import absolute_import" is used.
> >
> > It does do it both ways, there is just a fallback on the classic
> > import semantics in terms of trying it both as a relative and absolute
> > import.  But I got the semantics from the current implementation so it
> > is not some great inspiration of mine.  =)
>
> I think there shouldn't be a fall back.. that will just confuse things. Raise an
> exception here because most likely falling back is not what you want.
>

The fallback is the old way, so don't worry about it.

> If someone wants to import an external to a package module with the same name as
> the package, (or modules in some other package with the same name), then there
> needs to be an explicit way to do that.  But I really don't think this will come
> up that often.
>
>
> <clipped general examples>
>
> > Or you could have copied the code I wrote for the filesystem
> > importer's find_module method that already does this classification.
> > =)
> >
> > Part of the problem of working backwards from path to dotted name is
> > that it might not import that way.
>
> Maybe it should work that way?  If someone wants other than that behavior, then
> maybe there can be other ways to get it?
>

That's my point; the "other way" needs to work and the default can be
based on the path.

> Hers's an example of a situation where you might think it would be a problem,
> but it isn't:
>
>      pkg1:
>        __init__.py
>        m1.py
>        spkg1:
>           __init__.py
>           m3.py
>        dirA:
>           m4.py
>           pkg2:
>              __init__.py
>              m5.py
>
> You might think it wouldn't work for pkg2.m5, but that's actually ok.  pkg2 is a
> package just being stored in dirA which just happens to be located inside
> another package.
>
> Running m5.py directly will run it as a submodule of pkg2, which is what you
> want.  It's not in a sub-package of pkg1.  And m4.py is just a regular module.
>
> Or are you thinking of other relationships?
>

I am thinking of a package's __path__ being set to a specific
directory based on the platform or something.  That totally changes
the search order for the package that does not correspond to its
directory location.

>
> >__path__ can be tweaked, importers
> > and loaders can be written to interpret the directory structure or
> > file names differently, etc.
>
> Yes, and they will need a basic set of well defined default behaviors to build
> on.  After that, it's up to them to be sure their interpretation does what they
> want.
>
>
> > Plus what about different file types
> > like .ptl files from Quixote?
>
> This is really a matter of using a corresponding file reader to get at it's
> contents or it's real (relative to python) type... Ie, is it really a module, a
> package, or a module in a package, or some other thing ... living inside of a
> zip, or some other device (or file) like container?
>
>
>
> >> >> (4)  import module
> >> >>       import package
> >> >>
> >> >> Module and package are not in a package, so don't look in any
> >> >> packages, even
> >> >> this one or sys.path locations inside of packages.
> >> >>
> >> >
> >> > This is already done.  Absolute imports would cause this to do a
> >> > shallow check on sys.path for the module or package name.
> >>
> >> Great! 2 down.  Almost half way there.  :-)
> >>
> >> But will it check the current directory if you run a module directly
> >> because
> >> currently it doesn't know if it's part of a package.  Is that correct?
> >
> > Absolute import semantics go straight to sys.path, period.
>
> Which includes the current directory.  So in effect it will fall back to a
> relative type of behavior if a module with the same name is being imported exist
> in the current, inside this package direcotry, *if* you execute the module directly.
>
> I think this should also give an error, it is the inverse of the situation
> above. (#2) In most cases (if not all) it's not what you want.
>
> You wanted a module that is not part of this modules package, and got one that is.
>
>
> >> >> MOTIVATION
> >> >> ==========
> >> >>
> >> >> (A) Added reliability.
> >> >>
> >> >> There will be much less chance of errors (silent or otherwise) due to
> >> >> path/import conflicts which are sometimes difficult to diagnose.
> >> >>
> >> >
> >> > Probably, but I don't know if the implementation complexity warrants
> >> > worrying about this.  But then again how many people have actually
> >> > needed to implement the import machinery.  =)  I could be labeled as
> >> > jaded.
> >>
> >> Well, I know it's not an easy thing to do.  But it's not finding the
> >> paths and
> >> or weather files are modules etc... that is hard.  From what I
> >> understand the
> >> hard part is making it work so it can be extended and customized.
> >>
> >> Is that correct?
> >
> > Yes.  I really think ditching this whole __main__ name thing is going
> > to be the only solid solution.  Defining a __main__() method for
> > modules that gets executed makes the most sense to me.  Just import
> > the module and then execute the function if it exists.  That allow
> > runpy to have the name be set properly and does away with import
> > problems without mucking with import semantics.  Still have the name
> > problem if you specify a file directly on the command line, though.
>
> I'll have to see more details of how this would work I think. Part of me says
> sound good. And another part says, isn't this just moving stuff around? And what
> exactly does that solve?
>

It is moving things around, but so what?  Moving it keeps __name__
sane.  At work a global could be set to the name of the module that
started the execution or have an alias in sys.modules for the
'__main__' key to the module being executed.

The point of the solution it provides is it doesn't muck with import
semantics.  It allows the execution stuff to be external to imports
and be its own thing.

Guido has rejected this idea before (see PEP 299 :
http://www.python.org/dev/peps/pep-0299/ ), but then again there was
not this issue before.

Now I see why Nick said he wouldn't touch this in PEP 338.  =)

-Brett


From rhamph at gmail.com  Fri Feb  9 03:33:00 2007
From: rhamph at gmail.com (Adam Olsen)
Date: Thu, 8 Feb 2007 19:33:00 -0700
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702081143s139e3e66vf90cbc6ca6434863@mail.gmail.com>
References: <45C63363.2080100@ronadam.com>
	<aac2c7cb0702080501q67e8729ejb17f0c382099538f@mail.gmail.com>
	<bbaeab100702081143s139e3e66vf90cbc6ca6434863@mail.gmail.com>
Message-ID: <aac2c7cb0702081833l33c19cb3i455abd7aede6f547@mail.gmail.com>

Brett, Ron: nevermind my idea.  When trying to explain it further I
found myself going around in circles.  I need a clear set of
requirements before I can make a half-decent proposal, and the
existing docs and implementations don't make them easy to determine.

-- 
Adam Olsen, aka Rhamphoryncus


From ben at redfrontdoor.org  Fri Feb  9 09:31:22 2007
From: ben at redfrontdoor.org (Ben North)
Date: Fri, 09 Feb 2007 08:31:22 +0000
Subject: [Python-ideas] New syntax for 'dynamic' attribute access
Message-ID: <1171009882.45cc315a2d639@imp.hosting365.ie>

Hi,

I'd like to describe an addition I made to Python syntax which allows
easier access to attributes where the attribute name is only known at
run-time.  For example:

        setattr(self, method_name, getattr(self.metadata, method_name))

from Lib/distutils/dist.py could be rewritten

        self.(method_name) = self.metadata.(method_name)

As noted in the PEP-style description below, I mostly did this for
fun, but I thought it might be worth bringing to the attention of
python-ideas.  A quick search through prior postings and Google for
this idea didn't come up with anything.

Ben.



- - - - 8< - - - -


PEP: XXX
Title: Syntax For Dynamic Attribute Access
Version: $Revision$
Last-Modified: $Date$
Author: Ben North <ben at redfrontdoor.org>
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 29-Jan-2007
Post-History:


Abstract

    Dynamic attribute access is currently possible using the "getattr"
    and "setattr" builtins.  The present PEP suggests a new syntax to
    make such access easier, allowing the coder for example to write

        x.('foo_%d' % n) += 1

        z = y.('foo_%d' % n).('bar_%s' % s)

    instead of

        attr_name = 'foo_%d' % n
        setattr(x, attr_name, getattr(x, attr_name) + 1)

        z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s)


Note

    (I wrote this patch mostly to advance my own understanding of and
    experiment with the python language, but I've written it up in the
    style of a PEP in case it might be a useful idea.)


Rationale

    Dictionary access and indexing both have a friendly invocation
    syntax: instead of x.__getitem__(12) the coder can write x[12].
    This also allows the use of subscripted elements in an augmented
    assignment, as in "x[12] += 1".  The present proposal brings this
    ease-of-use to dynamic attribute access too.

    Attribute access is currently possible in two ways:

    * When the attribute name is known at code-writing time, the
      ".NAME" trailer can be used, as in

          x.foo = 42
          y.bar += 100

    * When the attribute name is computed dynamically at run-time, the
      "getattr" and "setattr" builtins must be used:

          x = getattr(y, 'foo_%d' % n)
          setattr(z, 'bar_%s' % s, 99)

      The "getattr" builtin also allows the coder to specify a default
      value to be returned in the event that the object does not have
      an attribute of the given name:

          x = getattr(y, 'foo_%d' % n, 0)

    This PEP describes a new syntax for dynamic attribute access ---
    "x.(expr)" --- with examples given in the Abstract above.  The new
    syntax also allows the provision of a default value in the "get"
    case, as in:

        x = y.('foo_%d' % n, None)

    This 2-argument form of dynamic attribute access is not permitted
    as the target of an (augmented or normal) assignment.  Finally,
    the new syntax can be used with the "del" statement, as in

        del x.(attr_name)


Impact On Existing Code

    The proposed new syntax is not currently valid, so no existing
    well-formed programs have their meaning altered by this proposal.

    Across all "*.py" files in the 2.5 distribution, there are around
    600 uses of "getattr", "setattr" or "delattr".  They break down as
    follows (figures have some room for error because they were
    arrived at by partially-manual inspection):

        c.300 uses of plain "getattr(x, attr_name)", which could be
              replaced with the new syntax;

        c.150 uses of the 3-argument form, i.e., with the default
              value; these could be replaced with the 2-argument form
              of the new syntax (the cases break down into c.125 cases
              where the attribute name is a literal string, and c.25
              where it's only known at run-time);

        c.5   uses of the 2-argument form with a literal string
              attribute name, which I think could be replaced with the
              standard "x.attribute" syntax;

        c.120 uses of setattr, of which 15 use getattr to find the
              new value; all could be replaced with the new syntax,
              the 15 where getattr is also involved would show a
              particular increase in clarity;

        c.5   uses which would have to stay as "getattr" because they
              are calls of a variable named "getattr" whose default
              value is the builtin "getattr";

        c.5   uses of the 2-argument form, inside a try/except block
              which catches AttributeError and uses a default value
              instead; these could use 2-argument form of the new
              syntax;

        c.10  uses of "delattr", which could use the new syntax.

    As examples, the line

        setattr(self, attr, change_root(self.root, getattr(self, attr)))

    from Lib/distutils/command/install.py could be rewritten

        self.(attr) = change_root(self.root, self.(attr))

    and the line

        setattr(self, method_name, getattr(self.metadata, method_name))

    from Lib/distutils/dist.py could be rewritten

        self.(method_name) = self.metadata.(method_name)


Alternative Syntax For The New Feature

    Other syntaxes could be used, for example braces are currently
    invalid in a "trailer", so could be used here, giving

        x{'foo_%d' % n} += 1

    My personal preference is for the

        x.('foo_%d' % n) += 1

    syntax though: the presence of the dot shows there is attribute
    access going on; the parentheses have an analogous meaning to the
    mathematical "work this out first" meaning.  This is also the
    syntax used in the language Matlab [1] for dynamic "field" access
    (where "field" is the Matlab term analogous to Python's
    "attribute").


Error Cases

    Only strings are permitted as attribute names, so for instance the
    following error is produced:

      >>> x.(99) = 8
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        TypeError: attribute name must be string, not 'int'

    This is handled by the existing PyObject_GetAttr function.


Draft Implementation

    A draft implementation adds a new alternative to the "trailer"
    clause in Grammar/Grammar; a new AST type, "DynamicAttribute" in
    Python.asdl, with accompanying changes to symtable.c, ast.c, and
    compile.c, and three new opcodes (load/store/del) with
    accompanying changes to opcode.h and ceval.c.  The patch consists
    of c.180 additional lines in the core code, and c.100 additional
    lines of tests.


References

    [1] Using Dynamic Field Names :: Data Types (MATLAB Programming)
        http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/f2-41859.html


Copyright

    This document has been placed in the public domain.

[PAGE-BREAK GOES HERE BUT REMOVED FOR EMAIL]
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:


- - - - 8< - - - -
diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Grammar/Grammar Python-2.5/Grammar/Grammar
--- ORIG__Python-2.5/Grammar/Grammar	2006-05-25 12:25:51.000000000 +0100
+++ Python-2.5/Grammar/Grammar	2007-02-01 18:07:04.133160000 +0000
@@ -119,7 +119,7 @@
 listmaker: test ( list_for | (',' test)* [','] )
 testlist_gexp: test ( gen_for | (',' test)* [','] )
 lambdef: 'lambda' [varargslist] ':' test
-trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
+trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')'
 subscriptlist: subscript (',' subscript)* [',']
 subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop]
 sliceop: ':' [test]
diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Include/opcode.h Python-2.5/Include/opcode.h
--- ORIG__Python-2.5/Include/opcode.h	2006-02-27 22:32:47.000000000 +0000
+++ Python-2.5/Include/opcode.h	2007-02-02 13:36:58.542848000 +0000
@@ -39,6 +39,10 @@
 #define SLICE		30
 /* Also uses 31-33 */

+/* LOAD_DYNAMIC_ATTR is below because it takes an argument. */
+#define STORE_DYNAMIC_ATTR   36
+#define DELETE_DYNAMIC_ATTR  37
+
 #define STORE_SLICE	40
 /* Also uses 41-43 */

@@ -89,6 +93,9 @@
 #define UNPACK_SEQUENCE	92	/* Number of sequence items */
 #define FOR_ITER	93

+#define LOAD_DYNAMIC_ATTR 94    /* Whether default given; 0 = no, 1 = yes */
+/* STORE_DYNAMIC_ATTR, DELETE_DYNAMIC_ATTR are above; they take no argument. */
+
 #define STORE_ATTR	95	/* Index in name list */
 #define DELETE_ATTR	96	/* "" */
 #define STORE_GLOBAL	97	/* "" */
diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_dynattr.py Python-2.5/Lib/test/test_dynattr.py
--- ORIG__Python-2.5/Lib/test/test_dynattr.py	1970-01-01 01:00:00.000000000 +0100
+++ Python-2.5/Lib/test/test_dynattr.py	2007-02-02 17:37:37.982390000 +0000
@@ -0,0 +1,91 @@
+import unittest
+import warnings
+import sys
+from test import test_support
+
+class AttrHolder:
+    pass
+
+class TestDynAttr(unittest.TestCase):
+
+    def test_simple_get(self):
+        a = AttrHolder()
+        a.foo = 100
+        a.attr_42 = 1
+        self.assertEqual(a.('foo'), 100)
+        self.assertEqual(a.('fo' + 'o'), 100)
+        self.assertEqual(a.('f' + 'o' + [('o' if True else 'obar')][0]), 100)
+        self.assertEqual(a.({'FOO': 'fo'}['FOO'] + 'o'), 100)
+        self.assertEqual(a.('fo%s' % 'o'), 100)
+        self.assertEqual(a.('attr_42'), 1)
+        self.assertEqual(a.('attr_%d' % 42), 1)
+        self.assertEqual(a.('foo' if True else 'attr_42'), 100)
+
+    def test_nested_get(self):
+        a = AttrHolder()
+        a.b = AttrHolder()
+        a.b.c = 1
+        attr_name_b = 'b'
+        attr_name_c = 'c'
+        self.assertEqual(a.(attr_name_b).(attr_name_c), 1)
+
+    def test_defaulting_get(self):
+        a = AttrHolder()
+        a.foo = 100
+        self.assertEqual(a.('foo', 99), 100)
+        self.assertEqual(a.('bar', 99), 99)
+        self.assertEqual(a.('baz', 99), 99)
+        self.assertEqual(a.('foo' if True else 'attr_42', 99), 100)
+        self.assertEqual(a.('foo' if False else 'attr_42', 99), 99)
+
+    @staticmethod
+    def attempt_non_string_use(attr_name):
+        a = AttrHolder()
+        return a.(attr_name)
+
+    def test_only_strings_allowed(self):
+        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 99)
+        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, None)
+        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 1.0)
+        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, AttrHolder)
+        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, sys)
+        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, ())
+        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, [])
+        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, {})
+        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, (1, 2))
+
+    def test_augassign(self):
+        a = AttrHolder()
+        a.foo = 100
+        a.('foo') += 10
+        self.assertEqual(a.foo, 110)
+        self.assertEqual(a.('fo' + 'o'), 110)
+        a.('f' + 'o' + 'o') *= 10
+        self.assertEqual(a.foo, 1100)
+        self.assertEqual(a.('fo' + 'o'), 1100)
+        a.('foobar'[:3]) /= 5
+        self.assertEqual(a.foo, 220)
+        self.assertEqual(a.('fo' + 'o'), 220)
+        a.(['foo', 'bar', 'baz'][0]) -= 40
+        self.assertEqual(a.foo, 180)
+        self.assertEqual(a.('fo' + 'o'), 180)
+
+    def test_setattr(self):
+        a = AttrHolder()
+        a.('foo') = 99
+        self.assertEqual(a.foo, 99)
+        a.('bar' + '_baz') = 100
+        self.assertEqual(a.bar_baz, 100)
+
+    def test_delattr(self):
+        a = AttrHolder()
+        a.foo = 99
+        del a.('foo')
+        self.assertEqual(hasattr(a, 'foo'), False)
+
+
+def test_main():
+    test_support.run_unittest(TestDynAttr)
+
+if __name__ == "__main__":
+    test_main()
diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_syntax.py Python-2.5/Lib/test/test_syntax.py
--- ORIG__Python-2.5/Lib/test/test_syntax.py	2006-05-19 07:43:50.000000000 +0100
+++ Python-2.5/Lib/test/test_syntax.py	2007-02-02 17:21:04.991119000 +0000
@@ -235,6 +235,18 @@
 >>> f() += 1
 Traceback (most recent call last):
 SyntaxError: illegal expression for augmented assignment (<doctest test.test_syntax[33]>, line 1)
+
+Outlawed uses of dynamic attributes:
+
+>>> x.('foo', 0) += 1
+Traceback (most recent call last):
+SyntaxError: augmented assignment to 2-argument dynamic-attribute expression not possible (<doctest test.test_syntax[34]>, line 1)
+>>> x.('foo', 0) = 1
+Traceback (most recent call last):
+SyntaxError: can't assign to 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[35]>, line 1)
+>>> del x.('foo', 0)
+Traceback (most recent call last):
+SyntaxError: can't delete 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[36]>, line 1)
 """

 import re
diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Parser/Python.asdl Python-2.5/Parser/Python.asdl
--- ORIG__Python-2.5/Parser/Python.asdl	2006-04-04 05:00:23.000000000 +0100
+++ Python-2.5/Parser/Python.asdl	2007-02-02 12:39:36.665151000 +0000
@@ -72,6 +72,7 @@

 	     -- the following expression can appear in assignment context
 	     | Attribute(expr value, identifier attr, expr_context ctx)
+	     | DynamicAttribute(expr value, expr attr, expr? dflt, expr_context ctx)
 	     | Subscript(expr value, slice slice, expr_context ctx)
 	     | Name(identifier id, expr_context ctx)
 	     | List(expr* elts, expr_context ctx)
diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ast.c Python-2.5/Python/ast.c
--- ORIG__Python-2.5/Python/ast.c	2006-09-05 04:56:01.000000000 +0100
+++ Python-2.5/Python/ast.c	2007-02-02 13:54:51.388446000 +0000
@@ -353,6 +353,13 @@
 	    }
 	    e->v.Attribute.ctx = ctx;
 	    break;
+        case DynamicAttribute_kind:
+            if ((ctx == Store || ctx == Del)
+                && e->v.DynamicAttribute.dflt)
+                expr_name = "2-argument form of dynamic-attribute expression";
+            else
+                e->v.DynamicAttribute.ctx = ctx;
+            break;
         case Subscript_kind:
 	    e->v.Subscript.ctx = ctx;
 	    break;
@@ -1427,7 +1434,7 @@
 static expr_ty
 ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr)
 {
-    /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
+    /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')'
        subscriptlist: subscript (',' subscript)* [',']
        subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop]
      */
@@ -1440,8 +1447,30 @@
             return ast_for_call(c, CHILD(n, 1), left_expr);
     }
     else if (TYPE(CHILD(n, 0)) == DOT ) {
+        if (TYPE(CHILD(n, 1)) == NAME)
         return Attribute(left_expr, NEW_IDENTIFIER(CHILD(n, 1)), Load,
                          LINENO(n), n->n_col_offset, c->c_arena);
+        else {
+            expr_ty e_val, e_dflt;
+            REQ(CHILD(n, 1), LPAR);
+            if (!(e_val = ast_for_expr(c, CHILD(n, 2))))
+                return NULL;
+
+            if (TYPE(CHILD(n, 3)) == RPAR) {
+                assert(NCH(n) == 3);
+                e_dflt = NULL;
+            } else {
+                assert(NCH(n) == 5);
+                REQ(CHILD(n, 3), COMMA);
+                REQ(CHILD(n, 5), RPAR);
+                if (!(e_dflt = ast_for_expr(c, CHILD(n, 4))))
+                    return NULL;
+            }
+
+            return DynamicAttribute(left_expr, e_val, e_dflt, Load,
+                                    LINENO(n), n->n_col_offset,
+                                    c->c_arena);
+        }
     }
     else {
         REQ(CHILD(n, 0), LSQB);
@@ -1964,6 +1993,15 @@
             case Attribute_kind:
             case Subscript_kind:
                 break;
+            case DynamicAttribute_kind:
+                if (expr1->v.DynamicAttribute.dflt) {
+                    ast_error(ch, "augmented assignment to "
+                              "2-argument dynamic-attribute "
+                              "expression not possible");
+                    return NULL;
+                }
+                break;
+
             default:
                 ast_error(ch, "illegal expression for augmented "
                           "assignment");
diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ceval.c Python-2.5/Python/ceval.c
--- ORIG__Python-2.5/Python/ceval.c	2006-08-13 19:10:10.000000000 +0100
+++ Python-2.5/Python/ceval.c	2007-02-08 12:56:39.637479000 +0000
@@ -1787,6 +1787,17 @@
 			if (err == 0) continue;
 			break;

+		case STORE_DYNAMIC_ATTR:
+			w = POP();
+			v = POP();
+			u = POP();
+			err = PyObject_SetAttr(v, w, u); /* v.w = u */
+			Py_DECREF(w);
+			Py_DECREF(v);
+			Py_DECREF(u);
+			if (err == 0) continue;
+			break;
+
 		case DELETE_ATTR:
 			w = GETITEM(names, oparg);
 			v = POP();
@@ -1795,6 +1806,14 @@
 			Py_DECREF(v);
 			break;

+		case DELETE_DYNAMIC_ATTR:
+			w = POP();
+			v = POP();
+			err = PyObject_SetAttr(v, w, (PyObject *)NULL);
+			Py_DECREF(w);
+			Py_DECREF(v);
+			break;
+
 		case STORE_GLOBAL:
 			w = GETITEM(names, oparg);
 			v = POP();
@@ -1994,6 +2013,28 @@
 			if (x != NULL) continue;
 			break;

+		case LOAD_DYNAMIC_ATTR:
+			if (oparg)
+				u = POP();
+			else
+				u = NULL;
+			w = POP();
+			v = TOP();
+			x = PyObject_GetAttr(v, w);
+			if (x == NULL && u != NULL
+			    && PyErr_ExceptionMatches(PyExc_AttributeError))
+			{
+				PyErr_Clear();
+				Py_INCREF(u);
+				x = u;
+			}
+			Py_DECREF(v);
+			Py_DECREF(w);
+			Py_XDECREF(u); /* This one may be NULL (if no default) */
+			SET_TOP(x);
+			if (x != NULL) continue;
+			break;
+
 		case COMPARE_OP:
 			w = POP();
 			v = TOP();
diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/compile.c Python-2.5/Python/compile.c
--- ORIG__Python-2.5/Python/compile.c	2006-08-12 02:45:47.000000000 +0100
+++ Python-2.5/Python/compile.c	2007-02-08 12:55:51.958809000 +0000
@@ -1357,6 +1357,13 @@
 		case SLICE+3:
 			return -1;

+		case LOAD_DYNAMIC_ATTR:
+			return -1 - oparg;
+		case STORE_DYNAMIC_ATTR:
+			return -3;
+		case DELETE_DYNAMIC_ATTR:
+			return -2;
+
 		case STORE_SLICE+0:
 			return -2;
 		case STORE_SLICE+1:
@@ -3641,6 +3648,41 @@
 			return 0;
 		}
 		break;
+	case DynamicAttribute_kind:
+	{
+		int has_default_p = (e->v.DynamicAttribute.dflt != NULL);
+
+		if (e->v.DynamicAttribute.ctx != AugStore) {
+			VISIT(c, expr, e->v.DynamicAttribute.value);
+			VISIT(c, expr, e->v.DynamicAttribute.attr);
+			if (has_default_p)
+				VISIT(c, expr, e->v.DynamicAttribute.dflt);
+		}
+		switch (e->v.DynamicAttribute.ctx) {
+		case AugLoad:
+			assert(!has_default_p);
+			ADDOP_I(c, DUP_TOPX, 2);
+			/* Fall through to Load */
+		case Load:
+			ADDOP_I(c, LOAD_DYNAMIC_ATTR, has_default_p);
+			break;
+		case AugStore:
+			ADDOP(c, ROT_THREE);
+			/* Fall through to Store */
+		case Store:
+			assert(!has_default_p);
+			ADDOP(c, STORE_DYNAMIC_ATTR);
+			break;
+		case Del:
+			ADDOP(c, DELETE_DYNAMIC_ATTR);
+			break;
+		default:
+			PyErr_SetString(PyExc_SystemError,
+					"invalid context in dynamic-attribute expression");
+			return 0;
+		}
+		break;
+	}
 	case Subscript_kind:
 		switch (e->v.Subscript.ctx) {
 		case AugLoad:
@@ -3700,6 +3742,18 @@
 		auge->v.Attribute.ctx = AugStore;
 		VISIT(c, expr, auge);
 		break;
+	case DynamicAttribute_kind:
+		assert(e->v.DynamicAttribute.dflt == NULL);
+		auge = DynamicAttribute(e->v.DynamicAttribute.value, e->v.DynamicAttribute.attr, NULL,
+					AugLoad, e->lineno, e->col_offset, c->c_arena);
+		if (auge == NULL)
+		    return 0;
+		VISIT(c, expr, auge);
+		VISIT(c, expr, s->v.AugAssign.value);
+		ADDOP(c, inplace_binop(c, s->v.AugAssign.op));
+		auge->v.DynamicAttribute.ctx = AugStore;
+		VISIT(c, expr, auge);
+		break;
 	case Subscript_kind:
 		auge = Subscript(e->v.Subscript.value, e->v.Subscript.slice,
 				 AugLoad, e->lineno, e->col_offset, c->c_arena);
diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/symtable.c Python-2.5/Python/symtable.c
--- ORIG__Python-2.5/Python/symtable.c	2006-08-12 02:43:40.000000000 +0100
+++ Python-2.5/Python/symtable.c	2007-02-08 12:53:36.279608000 +0000
@@ -1192,6 +1192,12 @@
         case Attribute_kind:
 		VISIT(st, expr, e->v.Attribute.value);
 		break;
+	case DynamicAttribute_kind:
+		VISIT(st, expr, e->v.DynamicAttribute.value);
+		VISIT(st, expr, e->v.DynamicAttribute.attr);
+		if (e->v.DynamicAttribute.dflt)
+		       VISIT(st, expr, e->v.DynamicAttribute.dflt);
+		break;
         case Subscript_kind:
 		VISIT(st, expr, e->v.Subscript.value);
 		VISIT(st, slice, e->v.Subscript.slice);




From guido at python.org  Fri Feb  9 16:51:47 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 9 Feb 2007 07:51:47 -0800
Subject: [Python-ideas] New syntax for 'dynamic' attribute access
In-Reply-To: <1171009882.45cc315a2d639@imp.hosting365.ie>
References: <1171009882.45cc315a2d639@imp.hosting365.ie>
Message-ID: <ca471dc20702090751l578f113chc9463ad92b4bb445@mail.gmail.com>

I've thought of the same syntax. I think you should submit this to the
PEP editor and argue on Python-dev for its inclusion in Python 2.6 --
there's no benefit that I see of waiting until 3.0.

--Guido

On 2/9/07, Ben North <ben at redfrontdoor.org> wrote:
> Hi,
>
> I'd like to describe an addition I made to Python syntax which allows
> easier access to attributes where the attribute name is only known at
> run-time.  For example:
>
>         setattr(self, method_name, getattr(self.metadata, method_name))
>
> from Lib/distutils/dist.py could be rewritten
>
>         self.(method_name) = self.metadata.(method_name)
>
> As noted in the PEP-style description below, I mostly did this for
> fun, but I thought it might be worth bringing to the attention of
> python-ideas.  A quick search through prior postings and Google for
> this idea didn't come up with anything.
>
> Ben.
>
>
>
> - - - - 8< - - - -
>
>
> PEP: XXX
> Title: Syntax For Dynamic Attribute Access
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ben North <ben at redfrontdoor.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/plain
> Created: 29-Jan-2007
> Post-History:
>
>
> Abstract
>
>     Dynamic attribute access is currently possible using the "getattr"
>     and "setattr" builtins.  The present PEP suggests a new syntax to
>     make such access easier, allowing the coder for example to write
>
>         x.('foo_%d' % n) += 1
>
>         z = y.('foo_%d' % n).('bar_%s' % s)
>
>     instead of
>
>         attr_name = 'foo_%d' % n
>         setattr(x, attr_name, getattr(x, attr_name) + 1)
>
>         z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s)
>
>
> Note
>
>     (I wrote this patch mostly to advance my own understanding of and
>     experiment with the python language, but I've written it up in the
>     style of a PEP in case it might be a useful idea.)
>
>
> Rationale
>
>     Dictionary access and indexing both have a friendly invocation
>     syntax: instead of x.__getitem__(12) the coder can write x[12].
>     This also allows the use of subscripted elements in an augmented
>     assignment, as in "x[12] += 1".  The present proposal brings this
>     ease-of-use to dynamic attribute access too.
>
>     Attribute access is currently possible in two ways:
>
>     * When the attribute name is known at code-writing time, the
>       ".NAME" trailer can be used, as in
>
>           x.foo = 42
>           y.bar += 100
>
>     * When the attribute name is computed dynamically at run-time, the
>       "getattr" and "setattr" builtins must be used:
>
>           x = getattr(y, 'foo_%d' % n)
>           setattr(z, 'bar_%s' % s, 99)
>
>       The "getattr" builtin also allows the coder to specify a default
>       value to be returned in the event that the object does not have
>       an attribute of the given name:
>
>           x = getattr(y, 'foo_%d' % n, 0)
>
>     This PEP describes a new syntax for dynamic attribute access ---
>     "x.(expr)" --- with examples given in the Abstract above.  The new
>     syntax also allows the provision of a default value in the "get"
>     case, as in:
>
>         x = y.('foo_%d' % n, None)
>
>     This 2-argument form of dynamic attribute access is not permitted
>     as the target of an (augmented or normal) assignment.  Finally,
>     the new syntax can be used with the "del" statement, as in
>
>         del x.(attr_name)
>
>
> Impact On Existing Code
>
>     The proposed new syntax is not currently valid, so no existing
>     well-formed programs have their meaning altered by this proposal.
>
>     Across all "*.py" files in the 2.5 distribution, there are around
>     600 uses of "getattr", "setattr" or "delattr".  They break down as
>     follows (figures have some room for error because they were
>     arrived at by partially-manual inspection):
>
>         c.300 uses of plain "getattr(x, attr_name)", which could be
>               replaced with the new syntax;
>
>         c.150 uses of the 3-argument form, i.e., with the default
>               value; these could be replaced with the 2-argument form
>               of the new syntax (the cases break down into c.125 cases
>               where the attribute name is a literal string, and c.25
>               where it's only known at run-time);
>
>         c.5   uses of the 2-argument form with a literal string
>               attribute name, which I think could be replaced with the
>               standard "x.attribute" syntax;
>
>         c.120 uses of setattr, of which 15 use getattr to find the
>               new value; all could be replaced with the new syntax,
>               the 15 where getattr is also involved would show a
>               particular increase in clarity;
>
>         c.5   uses which would have to stay as "getattr" because they
>               are calls of a variable named "getattr" whose default
>               value is the builtin "getattr";
>
>         c.5   uses of the 2-argument form, inside a try/except block
>               which catches AttributeError and uses a default value
>               instead; these could use 2-argument form of the new
>               syntax;
>
>         c.10  uses of "delattr", which could use the new syntax.
>
>     As examples, the line
>
>         setattr(self, attr, change_root(self.root, getattr(self, attr)))
>
>     from Lib/distutils/command/install.py could be rewritten
>
>         self.(attr) = change_root(self.root, self.(attr))
>
>     and the line
>
>         setattr(self, method_name, getattr(self.metadata, method_name))
>
>     from Lib/distutils/dist.py could be rewritten
>
>         self.(method_name) = self.metadata.(method_name)
>
>
> Alternative Syntax For The New Feature
>
>     Other syntaxes could be used, for example braces are currently
>     invalid in a "trailer", so could be used here, giving
>
>         x{'foo_%d' % n} += 1
>
>     My personal preference is for the
>
>         x.('foo_%d' % n) += 1
>
>     syntax though: the presence of the dot shows there is attribute
>     access going on; the parentheses have an analogous meaning to the
>     mathematical "work this out first" meaning.  This is also the
>     syntax used in the language Matlab [1] for dynamic "field" access
>     (where "field" is the Matlab term analogous to Python's
>     "attribute").
>
>
> Error Cases
>
>     Only strings are permitted as attribute names, so for instance the
>     following error is produced:
>
>       >>> x.(99) = 8
>         Traceback (most recent call last):
>           File "<stdin>", line 1, in <module>
>         TypeError: attribute name must be string, not 'int'
>
>     This is handled by the existing PyObject_GetAttr function.
>
>
> Draft Implementation
>
>     A draft implementation adds a new alternative to the "trailer"
>     clause in Grammar/Grammar; a new AST type, "DynamicAttribute" in
>     Python.asdl, with accompanying changes to symtable.c, ast.c, and
>     compile.c, and three new opcodes (load/store/del) with
>     accompanying changes to opcode.h and ceval.c.  The patch consists
>     of c.180 additional lines in the core code, and c.100 additional
>     lines of tests.
>
>
> References
>
>     [1] Using Dynamic Field Names :: Data Types (MATLAB Programming)
>         http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/f2-41859.html
>
>
> Copyright
>
>     This document has been placed in the public domain.
>
> [PAGE-BREAK GOES HERE BUT REMOVED FOR EMAIL]
> Local Variables:
> mode: indented-text
> indent-tabs-mode: nil
> sentence-end-double-space: t
> fill-column: 70
> coding: utf-8
> End:
>
>
> - - - - 8< - - - -
> diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Grammar/Grammar Python-2.5/Grammar/Grammar
> --- ORIG__Python-2.5/Grammar/Grammar    2006-05-25 12:25:51.000000000 +0100
> +++ Python-2.5/Grammar/Grammar  2007-02-01 18:07:04.133160000 +0000
> @@ -119,7 +119,7 @@
>  listmaker: test ( list_for | (',' test)* [','] )
>  testlist_gexp: test ( gen_for | (',' test)* [','] )
>  lambdef: 'lambda' [varargslist] ':' test
> -trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
> +trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')'
>  subscriptlist: subscript (',' subscript)* [',']
>  subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop]
>  sliceop: ':' [test]
> diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Include/opcode.h Python-2.5/Include/opcode.h
> --- ORIG__Python-2.5/Include/opcode.h   2006-02-27 22:32:47.000000000 +0000
> +++ Python-2.5/Include/opcode.h 2007-02-02 13:36:58.542848000 +0000
> @@ -39,6 +39,10 @@
>  #define SLICE          30
>  /* Also uses 31-33 */
>
> +/* LOAD_DYNAMIC_ATTR is below because it takes an argument. */
> +#define STORE_DYNAMIC_ATTR   36
> +#define DELETE_DYNAMIC_ATTR  37
> +
>  #define STORE_SLICE    40
>  /* Also uses 41-43 */
>
> @@ -89,6 +93,9 @@
>  #define UNPACK_SEQUENCE        92      /* Number of sequence items */
>  #define FOR_ITER       93
>
> +#define LOAD_DYNAMIC_ATTR 94    /* Whether default given; 0 = no, 1 = yes */
> +/* STORE_DYNAMIC_ATTR, DELETE_DYNAMIC_ATTR are above; they take no argument. */
> +
>  #define STORE_ATTR     95      /* Index in name list */
>  #define DELETE_ATTR    96      /* "" */
>  #define STORE_GLOBAL   97      /* "" */
> diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_dynattr.py Python-2.5/Lib/test/test_dynattr.py
> --- ORIG__Python-2.5/Lib/test/test_dynattr.py   1970-01-01 01:00:00.000000000 +0100
> +++ Python-2.5/Lib/test/test_dynattr.py 2007-02-02 17:37:37.982390000 +0000
> @@ -0,0 +1,91 @@
> +import unittest
> +import warnings
> +import sys
> +from test import test_support
> +
> +class AttrHolder:
> +    pass
> +
> +class TestDynAttr(unittest.TestCase):
> +
> +    def test_simple_get(self):
> +        a = AttrHolder()
> +        a.foo = 100
> +        a.attr_42 = 1
> +        self.assertEqual(a.('foo'), 100)
> +        self.assertEqual(a.('fo' + 'o'), 100)
> +        self.assertEqual(a.('f' + 'o' + [('o' if True else 'obar')][0]), 100)
> +        self.assertEqual(a.({'FOO': 'fo'}['FOO'] + 'o'), 100)
> +        self.assertEqual(a.('fo%s' % 'o'), 100)
> +        self.assertEqual(a.('attr_42'), 1)
> +        self.assertEqual(a.('attr_%d' % 42), 1)
> +        self.assertEqual(a.('foo' if True else 'attr_42'), 100)
> +
> +    def test_nested_get(self):
> +        a = AttrHolder()
> +        a.b = AttrHolder()
> +        a.b.c = 1
> +        attr_name_b = 'b'
> +        attr_name_c = 'c'
> +        self.assertEqual(a.(attr_name_b).(attr_name_c), 1)
> +
> +    def test_defaulting_get(self):
> +        a = AttrHolder()
> +        a.foo = 100
> +        self.assertEqual(a.('foo', 99), 100)
> +        self.assertEqual(a.('bar', 99), 99)
> +        self.assertEqual(a.('baz', 99), 99)
> +        self.assertEqual(a.('foo' if True else 'attr_42', 99), 100)
> +        self.assertEqual(a.('foo' if False else 'attr_42', 99), 99)
> +
> +    @staticmethod
> +    def attempt_non_string_use(attr_name):
> +        a = AttrHolder()
> +        return a.(attr_name)
> +
> +    def test_only_strings_allowed(self):
> +        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 99)
> +        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, None)
> +        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 1.0)
> +        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, AttrHolder)
> +        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, sys)
> +        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, ())
> +        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, [])
> +        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, {})
> +        self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, (1, 2))
> +
> +    def test_augassign(self):
> +        a = AttrHolder()
> +        a.foo = 100
> +        a.('foo') += 10
> +        self.assertEqual(a.foo, 110)
> +        self.assertEqual(a.('fo' + 'o'), 110)
> +        a.('f' + 'o' + 'o') *= 10
> +        self.assertEqual(a.foo, 1100)
> +        self.assertEqual(a.('fo' + 'o'), 1100)
> +        a.('foobar'[:3]) /= 5
> +        self.assertEqual(a.foo, 220)
> +        self.assertEqual(a.('fo' + 'o'), 220)
> +        a.(['foo', 'bar', 'baz'][0]) -= 40
> +        self.assertEqual(a.foo, 180)
> +        self.assertEqual(a.('fo' + 'o'), 180)
> +
> +    def test_setattr(self):
> +        a = AttrHolder()
> +        a.('foo') = 99
> +        self.assertEqual(a.foo, 99)
> +        a.('bar' + '_baz') = 100
> +        self.assertEqual(a.bar_baz, 100)
> +
> +    def test_delattr(self):
> +        a = AttrHolder()
> +        a.foo = 99
> +        del a.('foo')
> +        self.assertEqual(hasattr(a, 'foo'), False)
> +
> +
> +def test_main():
> +    test_support.run_unittest(TestDynAttr)
> +
> +if __name__ == "__main__":
> +    test_main()
> diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_syntax.py Python-2.5/Lib/test/test_syntax.py
> --- ORIG__Python-2.5/Lib/test/test_syntax.py    2006-05-19 07:43:50.000000000 +0100
> +++ Python-2.5/Lib/test/test_syntax.py  2007-02-02 17:21:04.991119000 +0000
> @@ -235,6 +235,18 @@
>  >>> f() += 1
>  Traceback (most recent call last):
>  SyntaxError: illegal expression for augmented assignment (<doctest test.test_syntax[33]>, line 1)
> +
> +Outlawed uses of dynamic attributes:
> +
> +>>> x.('foo', 0) += 1
> +Traceback (most recent call last):
> +SyntaxError: augmented assignment to 2-argument dynamic-attribute expression not possible (<doctest test.test_syntax[34]>, line 1)
> +>>> x.('foo', 0) = 1
> +Traceback (most recent call last):
> +SyntaxError: can't assign to 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[35]>, line 1)
> +>>> del x.('foo', 0)
> +Traceback (most recent call last):
> +SyntaxError: can't delete 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[36]>, line 1)
>  """
>
>  import re
> diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Parser/Python.asdl Python-2.5/Parser/Python.asdl
> --- ORIG__Python-2.5/Parser/Python.asdl 2006-04-04 05:00:23.000000000 +0100
> +++ Python-2.5/Parser/Python.asdl       2007-02-02 12:39:36.665151000 +0000
> @@ -72,6 +72,7 @@
>
>              -- the following expression can appear in assignment context
>              | Attribute(expr value, identifier attr, expr_context ctx)
> +            | DynamicAttribute(expr value, expr attr, expr? dflt, expr_context ctx)
>              | Subscript(expr value, slice slice, expr_context ctx)
>              | Name(identifier id, expr_context ctx)
>              | List(expr* elts, expr_context ctx)
> diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ast.c Python-2.5/Python/ast.c
> --- ORIG__Python-2.5/Python/ast.c       2006-09-05 04:56:01.000000000 +0100
> +++ Python-2.5/Python/ast.c     2007-02-02 13:54:51.388446000 +0000
> @@ -353,6 +353,13 @@
>             }
>             e->v.Attribute.ctx = ctx;
>             break;
> +        case DynamicAttribute_kind:
> +            if ((ctx == Store || ctx == Del)
> +                && e->v.DynamicAttribute.dflt)
> +                expr_name = "2-argument form of dynamic-attribute expression";
> +            else
> +                e->v.DynamicAttribute.ctx = ctx;
> +            break;
>          case Subscript_kind:
>             e->v.Subscript.ctx = ctx;
>             break;
> @@ -1427,7 +1434,7 @@
>  static expr_ty
>  ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr)
>  {
> -    /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
> +    /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')'
>         subscriptlist: subscript (',' subscript)* [',']
>         subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop]
>       */
> @@ -1440,8 +1447,30 @@
>              return ast_for_call(c, CHILD(n, 1), left_expr);
>      }
>      else if (TYPE(CHILD(n, 0)) == DOT ) {
> +        if (TYPE(CHILD(n, 1)) == NAME)
>          return Attribute(left_expr, NEW_IDENTIFIER(CHILD(n, 1)), Load,
>                           LINENO(n), n->n_col_offset, c->c_arena);
> +        else {
> +            expr_ty e_val, e_dflt;
> +            REQ(CHILD(n, 1), LPAR);
> +            if (!(e_val = ast_for_expr(c, CHILD(n, 2))))
> +                return NULL;
> +
> +            if (TYPE(CHILD(n, 3)) == RPAR) {
> +                assert(NCH(n) == 3);
> +                e_dflt = NULL;
> +            } else {
> +                assert(NCH(n) == 5);
> +                REQ(CHILD(n, 3), COMMA);
> +                REQ(CHILD(n, 5), RPAR);
> +                if (!(e_dflt = ast_for_expr(c, CHILD(n, 4))))
> +                    return NULL;
> +            }
> +
> +            return DynamicAttribute(left_expr, e_val, e_dflt, Load,
> +                                    LINENO(n), n->n_col_offset,
> +                                    c->c_arena);
> +        }
>      }
>      else {
>          REQ(CHILD(n, 0), LSQB);
> @@ -1964,6 +1993,15 @@
>              case Attribute_kind:
>              case Subscript_kind:
>                  break;
> +            case DynamicAttribute_kind:
> +                if (expr1->v.DynamicAttribute.dflt) {
> +                    ast_error(ch, "augmented assignment to "
> +                              "2-argument dynamic-attribute "
> +                              "expression not possible");
> +                    return NULL;
> +                }
> +                break;
> +
>              default:
>                  ast_error(ch, "illegal expression for augmented "
>                            "assignment");
> diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ceval.c Python-2.5/Python/ceval.c
> --- ORIG__Python-2.5/Python/ceval.c     2006-08-13 19:10:10.000000000 +0100
> +++ Python-2.5/Python/ceval.c   2007-02-08 12:56:39.637479000 +0000
> @@ -1787,6 +1787,17 @@
>                         if (err == 0) continue;
>                         break;
>
> +               case STORE_DYNAMIC_ATTR:
> +                       w = POP();
> +                       v = POP();
> +                       u = POP();
> +                       err = PyObject_SetAttr(v, w, u); /* v.w = u */
> +                       Py_DECREF(w);
> +                       Py_DECREF(v);
> +                       Py_DECREF(u);
> +                       if (err == 0) continue;
> +                       break;
> +
>                 case DELETE_ATTR:
>                         w = GETITEM(names, oparg);
>                         v = POP();
> @@ -1795,6 +1806,14 @@
>                         Py_DECREF(v);
>                         break;
>
> +               case DELETE_DYNAMIC_ATTR:
> +                       w = POP();
> +                       v = POP();
> +                       err = PyObject_SetAttr(v, w, (PyObject *)NULL);
> +                       Py_DECREF(w);
> +                       Py_DECREF(v);
> +                       break;
> +
>                 case STORE_GLOBAL:
>                         w = GETITEM(names, oparg);
>                         v = POP();
> @@ -1994,6 +2013,28 @@
>                         if (x != NULL) continue;
>                         break;
>
> +               case LOAD_DYNAMIC_ATTR:
> +                       if (oparg)
> +                               u = POP();
> +                       else
> +                               u = NULL;
> +                       w = POP();
> +                       v = TOP();
> +                       x = PyObject_GetAttr(v, w);
> +                       if (x == NULL && u != NULL
> +                           && PyErr_ExceptionMatches(PyExc_AttributeError))
> +                       {
> +                               PyErr_Clear();
> +                               Py_INCREF(u);
> +                               x = u;
> +                       }
> +                       Py_DECREF(v);
> +                       Py_DECREF(w);
> +                       Py_XDECREF(u); /* This one may be NULL (if no default) */
> +                       SET_TOP(x);
> +                       if (x != NULL) continue;
> +                       break;
> +
>                 case COMPARE_OP:
>                         w = POP();
>                         v = TOP();
> diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/compile.c Python-2.5/Python/compile.c
> --- ORIG__Python-2.5/Python/compile.c   2006-08-12 02:45:47.000000000 +0100
> +++ Python-2.5/Python/compile.c 2007-02-08 12:55:51.958809000 +0000
> @@ -1357,6 +1357,13 @@
>                 case SLICE+3:
>                         return -1;
>
> +               case LOAD_DYNAMIC_ATTR:
> +                       return -1 - oparg;
> +               case STORE_DYNAMIC_ATTR:
> +                       return -3;
> +               case DELETE_DYNAMIC_ATTR:
> +                       return -2;
> +
>                 case STORE_SLICE+0:
>                         return -2;
>                 case STORE_SLICE+1:
> @@ -3641,6 +3648,41 @@
>                         return 0;
>                 }
>                 break;
> +       case DynamicAttribute_kind:
> +       {
> +               int has_default_p = (e->v.DynamicAttribute.dflt != NULL);
> +
> +               if (e->v.DynamicAttribute.ctx != AugStore) {
> +                       VISIT(c, expr, e->v.DynamicAttribute.value);
> +                       VISIT(c, expr, e->v.DynamicAttribute.attr);
> +                       if (has_default_p)
> +                               VISIT(c, expr, e->v.DynamicAttribute.dflt);
> +               }
> +               switch (e->v.DynamicAttribute.ctx) {
> +               case AugLoad:
> +                       assert(!has_default_p);
> +                       ADDOP_I(c, DUP_TOPX, 2);
> +                       /* Fall through to Load */
> +               case Load:
> +                       ADDOP_I(c, LOAD_DYNAMIC_ATTR, has_default_p);
> +                       break;
> +               case AugStore:
> +                       ADDOP(c, ROT_THREE);
> +                       /* Fall through to Store */
> +               case Store:
> +                       assert(!has_default_p);
> +                       ADDOP(c, STORE_DYNAMIC_ATTR);
> +                       break;
> +               case Del:
> +                       ADDOP(c, DELETE_DYNAMIC_ATTR);
> +                       break;
> +               default:
> +                       PyErr_SetString(PyExc_SystemError,
> +                                       "invalid context in dynamic-attribute expression");
> +                       return 0;
> +               }
> +               break;
> +       }
>         case Subscript_kind:
>                 switch (e->v.Subscript.ctx) {
>                 case AugLoad:
> @@ -3700,6 +3742,18 @@
>                 auge->v.Attribute.ctx = AugStore;
>                 VISIT(c, expr, auge);
>                 break;
> +       case DynamicAttribute_kind:
> +               assert(e->v.DynamicAttribute.dflt == NULL);
> +               auge = DynamicAttribute(e->v.DynamicAttribute.value, e->v.DynamicAttribute.attr, NULL,
> +                                       AugLoad, e->lineno, e->col_offset, c->c_arena);
> +               if (auge == NULL)
> +                   return 0;
> +               VISIT(c, expr, auge);
> +               VISIT(c, expr, s->v.AugAssign.value);
> +               ADDOP(c, inplace_binop(c, s->v.AugAssign.op));
> +               auge->v.DynamicAttribute.ctx = AugStore;
> +               VISIT(c, expr, auge);
> +               break;
>         case Subscript_kind:
>                 auge = Subscript(e->v.Subscript.value, e->v.Subscript.slice,
>                                  AugLoad, e->lineno, e->col_offset, c->c_arena);
> diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/symtable.c Python-2.5/Python/symtable.c
> --- ORIG__Python-2.5/Python/symtable.c  2006-08-12 02:43:40.000000000 +0100
> +++ Python-2.5/Python/symtable.c        2007-02-08 12:53:36.279608000 +0000
> @@ -1192,6 +1192,12 @@
>          case Attribute_kind:
>                 VISIT(st, expr, e->v.Attribute.value);
>                 break;
> +       case DynamicAttribute_kind:
> +               VISIT(st, expr, e->v.DynamicAttribute.value);
> +               VISIT(st, expr, e->v.DynamicAttribute.attr);
> +               if (e->v.DynamicAttribute.dflt)
> +                      VISIT(st, expr, e->v.DynamicAttribute.dflt);
> +               break;
>          case Subscript_kind:
>                 VISIT(st, expr, e->v.Subscript.value);
>                 VISIT(st, slice, e->v.Subscript.slice);
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From veloso at verylowsodium.com  Fri Feb  9 17:15:23 2007
From: veloso at verylowsodium.com (Greg Falcon)
Date: Fri, 9 Feb 2007 11:15:23 -0500
Subject: [Python-ideas] New syntax for 'dynamic' attribute access
In-Reply-To: <1171009882.45cc315a2d639@imp.hosting365.ie>
References: <1171009882.45cc315a2d639@imp.hosting365.ie>
Message-ID: <3cdcefb80702090815y64f62df3s38291213006b5376@mail.gmail.com>

On 2/9/07, Ben North <ben at redfrontdoor.org> wrote:
> Hi,
>
> I'd like to describe an addition I made to Python syntax which allows
> easier access to attributes where the attribute name is only known at
> run-time.  For example:
>
>         setattr(self, method_name, getattr(self.metadata, method_name))
>
> from Lib/distutils/dist.py could be rewritten
>
>         self.(method_name) = self.metadata.(method_name)

Wow!  I have to say this is a compelling idea.  The syntax is a bit
foreign looking, but
  obj.(expr) += 1
is just such a huge win over
  setattr(obj, expr, getattr(obj, expr) + 1)
that I feel like I could learn to like it anyway.

So a few thoughts from the peanut gallery.

The problem with the syntax is that it looks a good bit like a
function call.  The dot can get easily lost when rendered in a
proportional font.  Because of this, I keep wanting to write the
punctuation as something else, possibly .* , but I think that's my C++
bias shining through.  (And as you point, your syntax does have the
precedent of MATLAB.)

>     This PEP describes a new syntax for dynamic attribute access ---
>     "x.(expr)" --- with examples given in the Abstract above.  The new
>     syntax also allows the provision of a default value in the "get"
>     case, as in:
>         x = y.('foo_%d' % n, None)

This is the one bit I really don't like.  y.('foobar') is arguably a
natural extension to y.foobar, but y.('foobar', None) isn't analogous
to any current syntax, and the multiple arguments make it look even
more dangerously like a call.

I think the goal should be to capture all of the readable syntax of
attribute access, not to capture all of the expressiveness of getattr.
 In Python, you already have to be explicit when you're worried if the
thing you're accessing might not be there.  So you write either:

  x.foo += 1
or
  x.foo = getattr(x, foo, 0) + 1

and you write either:

  x[bar].append(1)
or
  x.setdefault(bar, []).append(1)

Even if x.(foo, bar) wasn't scary syntax, I don't think breaking
tradition here is worth it.

But in any event, it's a very interesting idea and patch.  I'll
definitely play around with it this weekend.

Greg F


From jcarlson at uci.edu  Fri Feb  9 18:14:43 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 09 Feb 2007 09:14:43 -0800
Subject: [Python-ideas] New syntax for 'dynamic' attribute access
In-Reply-To: <1171009882.45cc315a2d639@imp.hosting365.ie>
References: <1171009882.45cc315a2d639@imp.hosting365.ie>
Message-ID: <20070209081856.AC8E.JCARLSON@uci.edu>


Ben North <ben at redfrontdoor.org> wrote:
> Hi,
> 
> I'd like to describe an addition I made to Python syntax which allows
> easier access to attributes where the attribute name is only known at
> run-time.  For example:
> 
>         setattr(self, method_name, getattr(self.metadata, method_name))
> 
> from Lib/distutils/dist.py could be rewritten
> 
>         self.(method_name) = self.metadata.(method_name)
> 
> As noted in the PEP-style description below, I mostly did this for
> fun, but I thought it might be worth bringing to the attention of
> python-ideas.  A quick search through prior postings and Google for
> this idea didn't come up with anything.

My only concern with your propsed change is your draft implementation...

> Draft Implementation
> 
>     A draft implementation adds a new alternative to the "trailer"
>     clause in Grammar/Grammar; a new AST type, "DynamicAttribute" in
>     Python.asdl, with accompanying changes to symtable.c, ast.c, and
>     compile.c, and three new opcodes (load/store/del) with
>     accompanying changes to opcode.h and ceval.c.  The patch consists
>     of c.180 additional lines in the core code, and c.100 additional
>     lines of tests.

Specifically, your changes to ceval.c and the compiler may have been
easier to implement, but it may negatively affect general Python
performance.  Have you run a recent pystone before and after the changes?

Since the *value* being used for the dynamic set, get, and load should
be available, it may be possible to replace the 3 new opcodes with 1 new
opcode that shifts the the value on the top of stack into the the code
object co_names, which can then be accessed directly by the standard
[LOAD|STORE|DEL]_ATTR opcodes.

The major concern with such a change is that the co_name field would no
longer be read-only, so wouldn't be sharable between threads (and would
need to grow by at least 1 when such dynamic accesses were allowed,
though growing by 2 could help in augmented assignment cases).

We could probably get away with a single new attribute on the stack
frame, adding an alternate GETITEM implementation...

#ifndef Py_DEBUG
#define GETITEM2(v, i, s) \
    ((i) != -1) ? PyTuple_GET_ITEM((PyTupleObject *)(v), (i)) : (s))
#else
#define GETITEM2(v, i, s) \
    ((i) != -1) ? PyTuple_GetItem((v), (i)) : (s))
#endif

With that change, dynamic attribute access would result in a single
opcode DYNAMIC_ACCESS, which would copy/move the value on the top of the
stack into some stack frame attribute, which is then automatically
accessed in the [LOAD|STORE|DEL]_ATTR opcodes (the GETITEM2 macro not
being used in any other opcodes).

The problem with the alternate approach above is that the overhead of
general [LOAD|STORE|DEL]_ATTR opcodes may become measurably larger
(depending on the branch prediction efficiency of a processor). 
Something to think about.


 - Josiah



From rrr at ronadam.com  Fri Feb  9 18:31:16 2007
From: rrr at ronadam.com (Ron Adam)
Date: Fri, 09 Feb 2007 11:31:16 -0600
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702081334j7d5c4207ge6bd4453525e70ff@mail.gmail.com>
References: <45C63363.2080100@ronadam.com>	
	<bbaeab100702071121w4e6b55e9g615a5284abfc3c2d@mail.gmail.com>	
	<45CA74A2.4060401@ronadam.com>	
	<bbaeab100702072218o51bef3e3h74110fd8098eae05@mail.gmail.com>	
	<45CB8AE7.9090808@ronadam.com>
	<bbaeab100702081334j7d5c4207ge6bd4453525e70ff@mail.gmail.com>
Message-ID: <45CCAFE4.2090102@ronadam.com>

Brett Cannon wrote:
> On 2/8/07, Ron Adam <rrr at ronadam.com> wrote:
>> Brett Cannon wrote:
>> > On 2/7/07, Ron Adam <rrr at ronadam.com> wrote:
>> >> Brett Cannon wrote:
>> >> > On 2/4/07, Ron Adam <rrr at ronadam.com> wrote:


>> >> It would be nice if __path__ were set on all modules in packages no
>> >> matter how
>> >> they are started.
>> >
>> > There is a slight issue with that as the __path__ attribute represents
>> > the top of a package and thus that it has an __init__ module.  It has
>> > some significance in terms of how stuff works at the moment.
>>
>> Yes, and after some reading I found __path__ isn't exactly what I was 
>> thinking.
>>
>> It could be it's only a matter of getting that first initial import 
>> right.  An
>> example of this is this recipe by Nick.
>>
>>      http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772
> 
> But Nick already rolled this stuff into 2.5 when package support was
> added to runpy.

I'll take a look at runpy today sometime.


>> If you remove the "__main__" name, then you will still need to have some
>> attribute for python to determine the same thing.
> 
> Why?  There is nothing saying we can't follow most other languages and
> just have a reserved function name that gets executed if the module is
> executed.

Yes, but this is where python is different from other languages.  In a way, 
python's main *is* the whole module from the top to bottom.  And so the 
'__main__' name is referring to the whole module and not just a function in it.

A more specific function would be needed to get the context right.  Maybe 
__script__(),  or __run__().


Or if you want to be consistent with class's,  how about adding __call__() to 
modules?   Then the main body of the module effectively works the same way as it 
does in a class.  =)


Hey, I think that has some cool possibilities, it makes modules callable in 
general.  So if I want to run a module's __call__(), AKA main() as you call it, 
after importing I would just do...

    import module
    module()

And it would just work.  ;-)


>>  What you would end up doing
>> is just moving the [if __name__=="__main__": __main__()] line off the 
>> end of
>> program so that all program have it automatically.  We just won't see 
>> it.  And
>> instead of checking __name__, the interpreter would check some other 
>> attribute.
>>
>> So what and where would that other attribute be?
>>
> 
> If a thing was done like that it would be in the global namespace of
> the module just like __name__ is.

Forget this, I like the idea above much better!  It's fully consistent with 
class's and so it would be easy to explain as well.  A step towards unification 
of class's and modules.  The __name__ attribute isn't changed as well. ;-)



>> If someone wants to import an external to a package module with the 
>> same name as
>> the package, (or modules in some other package with the same name), 
>> then there
>> needs to be an explicit way to do that.  But I really don't think this 
>> will come
>> up that often.
>>
>>
>> <clipped general examples>
>>
>> > Or you could have copied the code I wrote for the filesystem
>> > importer's find_module method that already does this classification.
>> > =)
>> >
>> > Part of the problem of working backwards from path to dotted name is
>> > that it might not import that way.
>>
>> Maybe it should work that way?  If someone wants other than that 
>> behavior, then
>> maybe there can be other ways to get it?
>>
> 
> That's my point; the "other way" needs to work and the default can be
> based on the path.

We need to get much more specific on this.  ie... examples.   I don't think we 
will get anywhere trying to generalize this point.



>> Hers's an example of a situation where you might think it would be a 
>> problem,
>> but it isn't:
>>
>>      pkg1:
>>        __init__.py
>>        m1.py
>>        spkg1:
>>           __init__.py
>>           m3.py
>>        dirA:
>>           m4.py
>>           pkg2:
>>              __init__.py
>>              m5.py
>>
>> You might think it wouldn't work for pkg2.m5, but that's actually ok.  
>> pkg2 is a
>> package just being stored in dirA which just happens to be located inside
>> another package.
>>
>> Running m5.py directly will run it as a submodule of pkg2, which is 
>> what you
>> want.  It's not in a sub-package of pkg1.  And m4.py is just a regular 
>> module.
>>
>> Or are you thinking of other relationships?
> 
> I am thinking of a package's __path__ being set to a specific
> directory based on the platform or something.  That totally changes
> the search order for the package that does not correspond to its
> directory location.

In that case, I think the developer and anyone who tries to run the script in a 
way the developer did not intend will have to be on their own.

For example if I add a directory to __path__ to include a module that normally 
lives someplace else. Thats ok.  If I execute any of 'my' modules in 'my' 
package. It will import __init__.py and set the __path__ accordingly and 
everything will still work.

But if I execute the 'other' module directly, then python needs to run it in 
what ever context it normally lives in.  We shouldn't try to figure out what 
'other' packages it may be used in, because it may be used in many packages.  So 
the only thing to do is run it in the context it is in where we find it.  And 
not this 'special' context we put it in.

For situations where we might have several subdir's in our package that may be 
choosen from depending on platform (or other things).  We may be able to put a 
hint in the directory, such as a _init__.py file.  (Notice the single 
underscore.)  Or some variation if that's too subtle.  The idea is it's an 
inactive sub-package and the main packages __init__ file could activate a 
'reserved' sub-package using some method like renaming the _init__.py to 
__init__.py, (but I really don't like renaming as a way to do that.)  It would 
be better to have some other way.

Then we could possibly still do the search up to find the root package by 
including _init__.py files in our search in those cases as well.



>> >> >> MOTIVATION
>> >> >> ==========
>> >> >>
>> >> >> (A) Added reliability.
>> >> >>
>> >> >> There will be much less chance of errors (silent or otherwise) 
>> due to
>> >> >> path/import conflicts which are sometimes difficult to diagnose.
>> >> >>
>> >> >
>> >> > Probably, but I don't know if the implementation complexity warrants
>> >> > worrying about this.  But then again how many people have actually
>> >> > needed to implement the import machinery.  =)  I could be labeled as
>> >> > jaded.
>> >>
>> >> Well, I know it's not an easy thing to do.  But it's not finding the
>> >> paths and
>> >> or weather files are modules etc... that is hard.  From what I
>> >> understand the
>> >> hard part is making it work so it can be extended and customized.
>> >>
>> >> Is that correct?
>> >
>> > Yes.  I really think ditching this whole __main__ name thing is going
>> > to be the only solid solution.  Defining a __main__() method for
>> > modules that gets executed makes the most sense to me.  Just import
>> > the module and then execute the function if it exists.  That allow
>> > runpy to have the name be set properly and does away with import
>> > problems without mucking with import semantics.  Still have the name
>> > problem if you specify a file directly on the command line, though.
>>
>> I'll have to see more details of how this would work I think. Part of 
>> me says
>> sound good. And another part says, isn't this just moving stuff 
>> around? And what
>> exactly does that solve?
> 
> It is moving things around, but so what?  Moving it keeps __name__
> sane.  At work a global could be set to the name of the module that
> started the execution or have an alias in sys.modules for the
> '__main__' key to the module being executed.

Or just use __call__().  It already behaves in the way you want for class's.  It 
could be reused I think for modules.  The only difference is it won't have a 
self arguments.  Which I think is not a problem.


> The point of the solution it provides is it doesn't muck with import
> semantics.  It allows the execution stuff to be external to imports
> and be its own thing.
> 
> Guido has rejected this idea before (see PEP 299 :
> http://www.python.org/dev/peps/pep-0299/ ), but then again there was
> not this issue before.
> 
> Now I see why Nick said he wouldn't touch this in PEP 338.  =)

I read the thread, and backwards compatibility as well as Guido just not liking 
it were the reasons it was rejected.  Backwards compatibility is less of a 
problem for py3k, but I also agree with his reasons for not liking it.  I think 
a reserved __call__() function for modules may be a little easier to sell.  It's 
already reserved in other situations for very much the same purpose as well.

Cheers,
    Ron



From brett at python.org  Fri Feb  9 19:24:51 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 9 Feb 2007 10:24:51 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <45CCAFE4.2090102@ronadam.com>
References: <45C63363.2080100@ronadam.com>
	<bbaeab100702071121w4e6b55e9g615a5284abfc3c2d@mail.gmail.com>
	<45CA74A2.4060401@ronadam.com>
	<bbaeab100702072218o51bef3e3h74110fd8098eae05@mail.gmail.com>
	<45CB8AE7.9090808@ronadam.com>
	<bbaeab100702081334j7d5c4207ge6bd4453525e70ff@mail.gmail.com>
	<45CCAFE4.2090102@ronadam.com>
Message-ID: <bbaeab100702091024o6ebc94d4t895351646757e130@mail.gmail.com>

On 2/9/07, Ron Adam <rrr at ronadam.com> wrote:
> Brett Cannon wrote:
> > On 2/8/07, Ron Adam <rrr at ronadam.com> wrote:

[SNIP]

> >> If you remove the "__main__" name, then you will still need to have some
> >> attribute for python to determine the same thing.
> >
> > Why?  There is nothing saying we can't follow most other languages and
> > just have a reserved function name that gets executed if the module is
> > executed.
>
> Yes, but this is where python is different from other languages.  In a way,
> python's main *is* the whole module from the top to bottom.  And so the
> '__main__' name is referring to the whole module and not just a function in it.
>
> A more specific function would be needed to get the context right.  Maybe
> __script__(),  or __run__().
>
>
> Or if you want to be consistent with class's,  how about adding __call__() to
> modules?   Then the main body of the module effectively works the same way as it
> does in a class.  =)
>
>
> Hey, I think that has some cool possibilities, it makes modules callable in
> general.  So if I want to run a module's __call__(), AKA main() as you call it,
> after importing I would just do...
>
>     import module
>     module()
>
> And it would just work.  ;-)
>

I like this idea.  Makes it very obvious.  You just say "when a
specific module is specified at the command line it is called.  Could
even have it take possibly sys.argv[1:] (which I think was supposed to
turn into sys.args or sys.arg or something at some point).

What do other people think?

-Brett


From jcarlson at uci.edu  Fri Feb  9 20:04:41 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 09 Feb 2007 11:04:41 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702091024o6ebc94d4t895351646757e130@mail.gmail.com>
References: <45CCAFE4.2090102@ronadam.com>
	<bbaeab100702091024o6ebc94d4t895351646757e130@mail.gmail.com>
Message-ID: <20070209105106.AC95.JCARLSON@uci.edu>


"Brett Cannon" <brett at python.org> wrote:
> 
> On 2/9/07, Ron Adam <rrr at ronadam.com> wrote:
> > Brett Cannon wrote:
> > > On 2/8/07, Ron Adam <rrr at ronadam.com> wrote:
> 
> [SNIP]
> 
> > >> If you remove the "__main__" name, then you will still need to have some
> > >> attribute for python to determine the same thing.
> > >
> > > Why?  There is nothing saying we can't follow most other languages and
> > > just have a reserved function name that gets executed if the module is
> > > executed.
> >
> > Yes, but this is where python is different from other languages.  In a way,
> > python's main *is* the whole module from the top to bottom.  And so the
> > '__main__' name is referring to the whole module and not just a function in it.
> >
> > A more specific function would be needed to get the context right.  Maybe
> > __script__(),  or __run__().
> >
> >
> > Or if you want to be consistent with class's,  how about adding __call__() to
> > modules?   Then the main body of the module effectively works the same way as it
> > does in a class.  =)
> >
> >
> > Hey, I think that has some cool possibilities, it makes modules callable in
> > general.  So if I want to run a module's __call__(), AKA main() as you call it,
> > after importing I would just do...
> >
> >     import module
> >     module()
> >
> > And it would just work.  ;-)
> >
> 
> I like this idea.  Makes it very obvious.  You just say "when a
> specific module is specified at the command line it is called.  Could
> even have it take possibly sys.argv[1:] (which I think was supposed to
> turn into sys.args or sys.arg or something at some point).
> 
> What do other people think?

I don't like it.  Much of my dislike comes from personal aesthetics, but
then there is a logical disconnect.  When an instance of a class is
created, its __call__ method is not automatically called. By using the
semantic of 'the __call__ function in the module namespace is
automatically executed if the module is "run" from the command line', we
are introducing a different instance creation semantic (an imported
module is an instance of ModuleType).

I think we should just stick with what has been proposed for *years*, a
__main__ function that is automatically executed after the module has
been imported if its __name__ == '__main__'.  Even better, anyone who
wants to write code compatible with the updated syntax can include the
following literal block at the end of their files...

    if __name__ == '__main__':
        try:
            __main__
        except NameError:
            pass
        else:
            try:
                __main__()
            finally:
                try:
                    from __future__ import disable_run
                except SyntaxError:
                    #we are using an older Python
                    pass
                else:
                    #we are using a new Python, and
                    #disabling automatic running succeeded
                    pass

With such a semantic, current users of Python could include the above
literal block and it would *just work*...then again, the new semantic
wouldn't really be useful if people started using the above literal
block.


 - Josiah



From brett at python.org  Fri Feb  9 22:14:07 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 9 Feb 2007 13:14:07 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <20070209105106.AC95.JCARLSON@uci.edu>
References: <45CCAFE4.2090102@ronadam.com>
	<bbaeab100702091024o6ebc94d4t895351646757e130@mail.gmail.com>
	<20070209105106.AC95.JCARLSON@uci.edu>
Message-ID: <bbaeab100702091314n58dc94acvfc662ecc4a3190e5@mail.gmail.com>

On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> "Brett Cannon" <brett at python.org> wrote:
> >
> > On 2/9/07, Ron Adam <rrr at ronadam.com> wrote:
> > > Brett Cannon wrote:
> > > > On 2/8/07, Ron Adam <rrr at ronadam.com> wrote:
> >
> > [SNIP]
> >
> > > >> If you remove the "__main__" name, then you will still need to have some
> > > >> attribute for python to determine the same thing.
> > > >
> > > > Why?  There is nothing saying we can't follow most other languages and
> > > > just have a reserved function name that gets executed if the module is
> > > > executed.
> > >
> > > Yes, but this is where python is different from other languages.  In a way,
> > > python's main *is* the whole module from the top to bottom.  And so the
> > > '__main__' name is referring to the whole module and not just a function in it.
> > >
> > > A more specific function would be needed to get the context right.  Maybe
> > > __script__(),  or __run__().
> > >
> > >
> > > Or if you want to be consistent with class's,  how about adding __call__() to
> > > modules?   Then the main body of the module effectively works the same way as it
> > > does in a class.  =)
> > >
> > >
> > > Hey, I think that has some cool possibilities, it makes modules callable in
> > > general.  So if I want to run a module's __call__(), AKA main() as you call it,
> > > after importing I would just do...
> > >
> > >     import module
> > >     module()
> > >
> > > And it would just work.  ;-)
> > >
> >
> > I like this idea.  Makes it very obvious.  You just say "when a
> > specific module is specified at the command line it is called.  Could
> > even have it take possibly sys.argv[1:] (which I think was supposed to
> > turn into sys.args or sys.arg or something at some point).
> >
> > What do other people think?
>
> I don't like it.  Much of my dislike comes from personal aesthetics, but
> then there is a logical disconnect.  When an instance of a class is
> created, its __call__ method is not automatically called. By using the
> semantic of 'the __call__ function in the module namespace is
> automatically executed if the module is "run" from the command line', we
> are introducing a different instance creation semantic (an imported
> module is an instance of ModuleType).
>

But I don't see the leap of how specifying a module to execute on the
command line is any different than doing ``Class()()`` for
instantiation with an immediate call.  It would still be a separate
step.

> I think we should just stick with what has been proposed for *years*, a
> __main__ function that is automatically executed after the module has
> been imported if its __name__ == '__main__'.

But that does not solve the problem Ron has been trying to deal with;
setting __name__ to __main__ prevents the execution of a module that
uses relative imports because the import machinery can then no longer
infer what package the module is in.

-Brett


From rrr at ronadam.com  Fri Feb  9 23:46:56 2007
From: rrr at ronadam.com (Ron Adam)
Date: Fri, 09 Feb 2007 16:46:56 -0600
Subject: [Python-ideas] Packages and Import
In-Reply-To: <20070209105106.AC95.JCARLSON@uci.edu>
References: <45CCAFE4.2090102@ronadam.com>
	<bbaeab100702091024o6ebc94d4t895351646757e130@mail.gmail.com>
	<20070209105106.AC95.JCARLSON@uci.edu>
Message-ID: <45CCF9E0.50704@ronadam.com>

Josiah Carlson wrote:
> "Brett Cannon" <brett at python.org> wrote:
>> On 2/9/07, Ron Adam <rrr at ronadam.com> wrote:
>>> Brett Cannon wrote:
>>>> On 2/8/07, Ron Adam <rrr at ronadam.com> wrote:
>> [SNIP]
>>
>>>>> If you remove the "__main__" name, then you will still need to have some
>>>>> attribute for python to determine the same thing.
>>>> Why?  There is nothing saying we can't follow most other languages and
>>>> just have a reserved function name that gets executed if the module is
>>>> executed.
>>> Yes, but this is where python is different from other languages.  In a way,
>>> python's main *is* the whole module from the top to bottom.  And so the
>>> '__main__' name is referring to the whole module and not just a function in it.
>>>
>>> A more specific function would be needed to get the context right.  Maybe
>>> __script__(),  or __run__().
>>>
>>>
>>> Or if you want to be consistent with class's,  how about adding __call__() to
>>> modules?   Then the main body of the module effectively works the same way as it
>>> does in a class.  =)
>>>
>>>
>>> Hey, I think that has some cool possibilities, it makes modules callable in
>>> general.  So if I want to run a module's __call__(), AKA main() as you call it,
>>> after importing I would just do...
>>>
>>>     import module
>>>     module()
>>>
>>> And it would just work.  ;-)
>>>
>> I like this idea.  Makes it very obvious.  You just say "when a
>> specific module is specified at the command line it is called.  Could
>> even have it take possibly sys.argv[1:] (which I think was supposed to
>> turn into sys.args or sys.arg or something at some point).
>>
>> What do other people think?
> 
> I don't like it.  Much of my dislike comes from personal aesthetics, ....

Fair enough.  I have one aesthetic dislike myself, but its really minor. 
Currently you can always look at the bottom of files to see what they will do if 
you execute them.  With a main() function of any type, it may be someplace else. 
  But I can live with that if the name is always the same and doesn't change.


> .... but then there is a logical disconnect.  When an instance of a class is
> created, its __call__ method is not automatically called. By using the
> semantic of 'the __call__ function in the module namespace is
> automatically executed if the module is "run" from the command line', we
> are introducing a different instance creation semantic (an imported
> module is an instance of ModuleType).

I'm not sure I follow.  Importing a module would not call it's __call__() 
function.  As I've shown above it's an extra step.


Running a module isn't the same as importing one.  A lot of stuff goes on under 
the covers, so there really is no logical disconnect.  What it really does is 
just add one additional step to the "run from a command line" sequence of events.


> I think we should just stick with what has been proposed for *years*, a
> __main__ function that is automatically executed after the module has
> been imported if its __name__ == '__main__'.  Even better, anyone who
> wants to write code compatible with the updated syntax can include the
> following literal block at the end of their files...
> 
>     if __name__ == '__main__':
>         try:
>             __main__
>         except NameError:
>             pass
>         else:
>             try:
>                 __main__()
>             finally:
>                 try:
>                     from __future__ import disable_run
>                 except SyntaxError:
>                     #we are using an older Python
>                     pass
>                 else:
>                     #we are using a new Python, and
>                     #disabling automatic running succeeded
>                     pass
> 
> With such a semantic, current users of Python could include the above
> literal block and it would *just work*...then again, the new semantic
> wouldn't really be useful if people started using the above literal
> block.

That seems to be too much for some reason.



If __name__ returns the real name and a function ismain() is a new builtin, then 
these short one liners would work.


     #  Put at bottom of new programs so they work with older python too.

     if __name__ == '__main__': __call__()



     #  Makes old programs work with newer python.
     #  Add this above "if __name__=='__main__'".

     if hasattr(__builtins__, 'ismain') and ismain():  __name__ = '__main__'


Cheers,
    Ron







From jcarlson at uci.edu  Sat Feb 10 00:28:52 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 09 Feb 2007 15:28:52 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702091314n58dc94acvfc662ecc4a3190e5@mail.gmail.com>
References: <20070209105106.AC95.JCARLSON@uci.edu>
	<bbaeab100702091314n58dc94acvfc662ecc4a3190e5@mail.gmail.com>
Message-ID: <20070209145514.AC9E.JCARLSON@uci.edu>


"Brett Cannon" <brett at python.org> wrote:
> On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > "Brett Cannon" <brett at python.org> wrote:
> > > On 2/9/07, Ron Adam <rrr at ronadam.com> wrote:
[snip]
> > > >     import module
> > > >     module()
> > > >
> > > > And it would just work.  ;-)
> > > >
> > >
> > > I like this idea.  Makes it very obvious.  You just say "when a
> > > specific module is specified at the command line it is called.  Could
> > > even have it take possibly sys.argv[1:] (which I think was supposed to
> > > turn into sys.args or sys.arg or something at some point).
> > >
> > > What do other people think?
> >
> > I don't like it.  Much of my dislike comes from personal aesthetics, but
> > then there is a logical disconnect.  When an instance of a class is
> > created, its __call__ method is not automatically called. By using the
> > semantic of 'the __call__ function in the module namespace is
> > automatically executed if the module is "run" from the command line', we
> > are introducing a different instance creation semantic (an imported
> > module is an instance of ModuleType).
> 
> But I don't see the leap of how specifying a module to execute on the
> command line is any different than doing ``Class()()`` for
> instantiation with an immediate call.  It would still be a separate
> step.

I feel that there is a disconnect, but maybe it's personal aesthetics
again.


> > I think we should just stick with what has been proposed for *years*, a
> > __main__ function that is automatically executed after the module has
> > been imported if its __name__ == '__main__'.
> 
> But that does not solve the problem Ron has been trying to deal with;
> setting __name__ to __main__ prevents the execution of a module that
> uses relative imports because the import machinery can then no longer
> infer what package the module is in.

I may have missed it, but how are either of the following ambiguous...

    from . import foo
    from ..bar import baz

The leading dot tells me that 'relative to the path of the current
module (or __init__.py module in a package), look for packages/modules
named [everything else without a single leading dot].

Now, I tried the first of those lines in Python 2.5 and I was surpised
that having two files foo and goo, goo importing foo via the first
example above, didn't work.  What is even worse is that over a year ago
I was working on an import semantic for relative imports that would have
made the above do as I would have expected.

This leads me to believe that *something* about relative imports is
broken, but being that I mostly skipped the earlier portions of this
particular thread, I'm not certain what it is.  I would *guess* that it
has to do with the way the current importer comes up with package
relative imports, and I believe it could be fixed by switching to a
path-relative import.

That is, when module goo is performing 'from . import foo', you don't
look at goo.__name__ to determine where to look for foo, you look at
goo.__file__ .  With that change in semantic, both of the above cases
work just fine, including 'from ..bar import baz', even without the
current module being part of a package.  That is, running goo.py in the
following tree would succeed with the above two imports...

    .../
        torun/
            #include __init__.py if you want this to be a package
            goo.py
            foo.py
        bar/
            __init__.py
            baz.py
        #I don't know if it would make sense to require __init__.py here

It is still possible to handle import hooks, as the relative import
stuff is only really applicable to getting the base path from which to
start searching for sub packages (after you have stripped off all
leading periods).

It also naturally leads to a __name__ semantic that Guido had suggested
to me when I was talking about relative imports:

    goo.__name__ == '__main__'
    foo.__name__ == '__main__.foo'
    baz.__name__ == '__main__..bar.baz'

Which could more or less be used with the current importer; it just
needs a special-casing of 'from . import ...' in the __main__ module. 
It may also make sense to do path/__name__ normalization relative to
__main__ for any relative imports, so if in baz.py above you did 'from
..torun import foo', that it gave you the previously existing foo and
not a new copy.

I've got a bunch of code that implements the above name/path semantic,
but it was never tested (being that I was using Python 2.4 at the time,
and relative imports weren't proper syntax).


 - Josiah



From brett at python.org  Sat Feb 10 01:50:45 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 9 Feb 2007 16:50:45 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <20070209145514.AC9E.JCARLSON@uci.edu>
References: <20070209105106.AC95.JCARLSON@uci.edu>
	<bbaeab100702091314n58dc94acvfc662ecc4a3190e5@mail.gmail.com>
	<20070209145514.AC9E.JCARLSON@uci.edu>
Message-ID: <bbaeab100702091650v34a01d46j27fdbcc72299940c@mail.gmail.com>

On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> "Brett Cannon" <brett at python.org> wrote:
> > On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > > "Brett Cannon" <brett at python.org> wrote:
> > > > On 2/9/07, Ron Adam <rrr at ronadam.com> wrote:
> [snip]
> > > > >     import module
> > > > >     module()
> > > > >
> > > > > And it would just work.  ;-)
> > > > >
> > > >
> > > > I like this idea.  Makes it very obvious.  You just say "when a
> > > > specific module is specified at the command line it is called.  Could
> > > > even have it take possibly sys.argv[1:] (which I think was supposed to
> > > > turn into sys.args or sys.arg or something at some point).
> > > >
> > > > What do other people think?
> > >
> > > I don't like it.  Much of my dislike comes from personal aesthetics, but
> > > then there is a logical disconnect.  When an instance of a class is
> > > created, its __call__ method is not automatically called. By using the
> > > semantic of 'the __call__ function in the module namespace is
> > > automatically executed if the module is "run" from the command line', we
> > > are introducing a different instance creation semantic (an imported
> > > module is an instance of ModuleType).
> >
> > But I don't see the leap of how specifying a module to execute on the
> > command line is any different than doing ``Class()()`` for
> > instantiation with an immediate call.  It would still be a separate
> > step.
>
> I feel that there is a disconnect, but maybe it's personal aesthetics
> again.
>
>
> > > I think we should just stick with what has been proposed for *years*, a
> > > __main__ function that is automatically executed after the module has
> > > been imported if its __name__ == '__main__'.
> >
> > But that does not solve the problem Ron has been trying to deal with;
> > setting __name__ to __main__ prevents the execution of a module that
> > uses relative imports because the import machinery can then no longer
> > infer what package the module is in.
>
> I may have missed it, but how are either of the following ambiguous...
>
>     from . import foo
>     from ..bar import baz
>
> The leading dot tells me that 'relative to the path of the current
> module (or __init__.py module in a package), look for packages/modules
> named [everything else without a single leading dot].
>

Right, but "path" in this case is not file path but an ambiguous path.
 Relative imports work for modules stored in file systems, files
(e.g., zip files), databases, etc.

> Now, I tried the first of those lines in Python 2.5 and I was surpised
> that having two files foo and goo, goo importing foo via the first
> example above, didn't work.  What is even worse is that over a year ago
> I was working on an import semantic for relative imports that would have
> made the above do as I would have expected.
>
> This leads me to believe that *something* about relative imports is
> broken, but being that I mostly skipped the earlier portions of this
> particular thread, I'm not certain what it is.  I would *guess* that it
> has to do with the way the current importer comes up with package
> relative imports, and I believe it could be fixed by switching to a
> path-relative import.
>
> That is, when module goo is performing 'from . import foo', you don't
> look at goo.__name__ to determine where to look for foo, you look at
> goo.__file__ .

But what about modules stored in a sqlite3 database?  How is that
supposed to work?  What made basing relative imports off of __name__
so nice is that it allowed the import machinery figure out what the
resulting absolute module name was.  That allowed the modules to be
stored any way that you wanted without making any assumptions about
how the modules are stored or their location is determined by an
importer's find_module method.

> With that change in semantic, both of the above cases
> work just fine, including 'from ..bar import baz', even without the
> current module being part of a package.  That is, running goo.py in the
> following tree would succeed with the above two imports...
>
>     .../
>         torun/
>             #include __init__.py if you want this to be a package
>             goo.py
>             foo.py
>         bar/
>             __init__.py
>             baz.py
>         #I don't know if it would make sense to require __init__.py here
>
> It is still possible to handle import hooks, as the relative import
> stuff is only really applicable to getting the base path from which to
> start searching for sub packages (after you have stripped off all
> leading periods).
>

But it isn't a file path, it's an absolute module name that you are after.

> It also naturally leads to a __name__ semantic that Guido had suggested
> to me when I was talking about relative imports:
>
>     goo.__name__ == '__main__'
>     foo.__name__ == '__main__.foo'
>     baz.__name__ == '__main__..bar.baz'
>
> Which could more or less be used with the current importer; it just
> needs a special-casing of 'from . import ...' in the __main__ module.

And I am trying to avoid special-casing for this.

-Brett


From jcarlson at uci.edu  Sat Feb 10 02:57:44 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 09 Feb 2007 17:57:44 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702091650v34a01d46j27fdbcc72299940c@mail.gmail.com>
References: <20070209145514.AC9E.JCARLSON@uci.edu>
	<bbaeab100702091650v34a01d46j27fdbcc72299940c@mail.gmail.com>
Message-ID: <20070209171022.ACA1.JCARLSON@uci.edu>


"Brett Cannon" <brett at python.org> wrote:
> On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
[snip]

> > Now, I tried the first of those lines in Python 2.5 and I was surpised
> > that having two files foo and goo, goo importing foo via the first
> > example above, didn't work.  What is even worse is that over a year ago
> > I was working on an import semantic for relative imports that would have
> > made the above do as I would have expected.
> >
> > This leads me to believe that *something* about relative imports is
> > broken, but being that I mostly skipped the earlier portions of this
> > particular thread, I'm not certain what it is.  I would *guess* that it
> > has to do with the way the current importer comes up with package
> > relative imports, and I believe it could be fixed by switching to a
> > path-relative import.
> >
> > That is, when module goo is performing 'from . import foo', you don't
> > look at goo.__name__ to determine where to look for foo, you look at
> > goo.__file__ .
> 
> But what about modules stored in a sqlite3 database?  How is that
> supposed to work?  What made basing relative imports off of __name__
> so nice is that it allowed the import machinery figure out what the
> resulting absolute module name was.  That allowed the modules to be
> stored any way that you wanted without making any assumptions about
> how the modules are stored or their location is determined by an
> importer's find_module method.

How is it done now?  Presumably if you have some module imported from a
database (who does this, really?), it gets a name like
dbimported.name1.name2, which an import hook can recognize as being
imported from a database.  Now, is dbimported.name1.name2 really the
content of an __init__.py file (if it was a package), or is it a module
in dbimported.name1?


Right now, we can't distinguish (based on __name__) between the cases of...

    foo/
        __init__.py

and
    foo.py

But we can, trivially, distinguish just by *examining* __file__ (at
least for files from a filesystem). For example:

    >>> import bar
    >>> import baz
    >>> bar.__name__, bar.__file__
    ('bar', 'bar.py')
    >>> baz.__name__, baz.__file__
    ('baz', 'baz\\__init__.py')
    >>>

It's pretty obvious to me which one is a package and which one is a
module.

If we codify the requirement that __file__ must end with '.../__init__.X'
(where X can be; py, pyw, pyc, so, dll, pyd, etc.) if the thing we
imported is a package, then the import hooks don't need to use the
__file__ attribute for anything other than discerning between "is this a
package, or is this a module", and can then handle the __name__ mangling
as per import semantics.  The only trick is if someone were to
specifically import an __init__ module (from .package import __init__),
but even then, the results are garbage (you get the module's __init__
method).

> But it isn't a file path, it's an absolute module name that you are after.

If one were to just do "path" manipulations, afterwards you can
translate that to an absolute name (perhaps based on the path of
__main__). Pretending that you have a path can make the semantic
non-ambigous, but I prefer the alternate I just described.


> > It also naturally leads to a __name__ semantic that Guido had suggested
> > to me when I was talking about relative imports:
> >
> >     goo.__name__ == '__main__'
> >     foo.__name__ == '__main__.foo'
> >     baz.__name__ == '__main__..bar.baz'
> >
> > Which could more or less be used with the current importer; it just
> > needs a special-casing of 'from . import ...' in the __main__ module.
> 
> And I am trying to avoid special-casing for this.

And it's only really an issue because we can't currently discern between
module or package, right?  So let us choose some semantic for
determining "is this a module or package?", perhaps set by whatever did
the importing, and skip the special cases for __main__, etc.

We can use the __file__ semantic I described earlier.  Or we can specify
a new attribute on modules; __package__.  If __package__ == __name__,
then the current module is a package.  If __package__ != __name__, then
the current module is not a package.  Regardless, when doing relative
imports, the 'name' we start out with is __package__.

For example, say we have a file called foo.py that we have run from the
command line.  It's __name__ should be '__main__', as per Python history. 
However, __package__ will be ''.  When foo.py performs 'from . import
goo', we know precisely what "package" we are in, the same package (and
"path") as the '__main__' module.

Two remaining cases:
1) If goo is a module; goo.py sits next to foo.py (or equivalently in a
database, etc.)
    goo.__package__ == ''
    goo.__name__ == 'goo'
2) If goo is a package;
    goo.__package__ == 'goo'
    goo.__name__ == 'goo'


On the other hand, say foo.py sits in 'bin', did 'from ..pa import bar',
but bar did 'from ..bin import foo', we now have an issue.  How do we
determine that the foo that bar imports is the same foo that was run
from the command line?  However, this is a problem regardless of your
'package or module' semantic, if you fix the 'from .. import baz' being
run from the '__main__' module.


 - Josiah



From brett at python.org  Sat Feb 10 23:32:55 2007
From: brett at python.org (Brett Cannon)
Date: Sat, 10 Feb 2007 14:32:55 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <20070209171022.ACA1.JCARLSON@uci.edu>
References: <20070209145514.AC9E.JCARLSON@uci.edu>
	<bbaeab100702091650v34a01d46j27fdbcc72299940c@mail.gmail.com>
	<20070209171022.ACA1.JCARLSON@uci.edu>
Message-ID: <bbaeab100702101432p1d99997bvf72ccdf77e8cb4cc@mail.gmail.com>

On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> "Brett Cannon" <brett at python.org> wrote:
> > On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> [snip]
>
> > > Now, I tried the first of those lines in Python 2.5 and I was surpised
> > > that having two files foo and goo, goo importing foo via the first
> > > example above, didn't work.  What is even worse is that over a year ago
> > > I was working on an import semantic for relative imports that would have
> > > made the above do as I would have expected.
> > >
> > > This leads me to believe that *something* about relative imports is
> > > broken, but being that I mostly skipped the earlier portions of this
> > > particular thread, I'm not certain what it is.  I would *guess* that it
> > > has to do with the way the current importer comes up with package
> > > relative imports, and I believe it could be fixed by switching to a
> > > path-relative import.
> > >
> > > That is, when module goo is performing 'from . import foo', you don't
> > > look at goo.__name__ to determine where to look for foo, you look at
> > > goo.__file__ .
> >
> > But what about modules stored in a sqlite3 database?  How is that
> > supposed to work?  What made basing relative imports off of __name__
> > so nice is that it allowed the import machinery figure out what the
> > resulting absolute module name was.  That allowed the modules to be
> > stored any way that you wanted without making any assumptions about
> > how the modules are stored or their location is determined by an
> > importer's find_module method.
>
> How is it done now?

Importers and loaders; see PEP 302.

>  Presumably if you have some module imported from a
> database (who does this, really?),

No one at the moment.  But think of it as an alternative to zip files
if you use sqlite3 for the DB storage.  Several people have told me
privately they would like to see an importer and loader for this.

> it gets a name like
> dbimported.name1.name2, which an import hook can recognize as being
> imported from a database.  Now, is dbimported.name1.name2 really the
> content of an __init__.py file (if it was a package), or is it a module
> in dbimported.name1?
>
>
> Right now, we can't distinguish (based on __name__) between the cases of...
>
>     foo/
>         __init__.py
>
> and
>     foo.py
>

Right, but you can based on whether __path__ is defined, which is how
import tells.

> But we can, trivially, distinguish just by *examining* __file__ (at
> least for files from a filesystem). For example:
>
>     >>> import bar
>     >>> import baz
>     >>> bar.__name__, bar.__file__
>     ('bar', 'bar.py')
>     >>> baz.__name__, baz.__file__
>     ('baz', 'baz\\__init__.py')
>     >>>
>
> It's pretty obvious to me which one is a package and which one is a
> module.
>

Right, but as I said, this is what __path__ is for; to tell when a
module is just a module or a package.  And then the __name__ attribute
is used to tell if it is a top-level module or not.

> If we codify the requirement that __file__ must end with '.../__init__.X'
> (where X can be; py, pyw, pyc, so, dll, pyd, etc.) if the thing we
> imported is a package, then the import hooks don't need to use the
> __file__ attribute for anything other than discerning between "is this a
> package, or is this a module", and can then handle the __name__ mangling
> as per import semantics.  The only trick is if someone were to
> specifically import an __init__ module (from .package import __init__),
> but even then, the results are garbage (you get the module's __init__
> method).
>
> > But it isn't a file path, it's an absolute module name that you are after.
>
> If one were to just do "path" manipulations, afterwards you can
> translate that to an absolute name (perhaps based on the path of
> __main__). Pretending that you have a path can make the semantic
> non-ambigous, but I prefer the alternate I just described.
>
>
> > > It also naturally leads to a __name__ semantic that Guido had suggested
> > > to me when I was talking about relative imports:
> > >
> > >     goo.__name__ == '__main__'
> > >     foo.__name__ == '__main__.foo'
> > >     baz.__name__ == '__main__..bar.baz'
> > >
> > > Which could more or less be used with the current importer; it just
> > > needs a special-casing of 'from . import ...' in the __main__ module.
> >
> > And I am trying to avoid special-casing for this.
>
> And it's only really an issue because we can't currently discern between
> module or package, right?

Basically.  It's also where within a package the module is, so it
isn't quite that simple.

The issue comes down to that we lose where a module is contained
within a package hierarchy once __name__ gets set to __main__.  If we
came up with another way to delineate that a module was being executed
or some other way to specify where a module was in a package then the
problem would be solved.  I prefer trying to change the former since
the latter is perfectly handled with __name__ as-is when it isn't
changed to __main__.

-Brett


From collinw at gmail.com  Sat Feb 10 23:45:58 2007
From: collinw at gmail.com (Collin Winter)
Date: Sat, 10 Feb 2007 16:45:58 -0600
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702101432p1d99997bvf72ccdf77e8cb4cc@mail.gmail.com>
References: <20070209145514.AC9E.JCARLSON@uci.edu>
	<bbaeab100702091650v34a01d46j27fdbcc72299940c@mail.gmail.com>
	<20070209171022.ACA1.JCARLSON@uci.edu>
	<bbaeab100702101432p1d99997bvf72ccdf77e8cb4cc@mail.gmail.com>
Message-ID: <43aa6ff70702101445g1ab068a8nf8f52e91f49403d5@mail.gmail.com>

On 2/10/07, Brett Cannon <brett at python.org> wrote:
> On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> >  Presumably if you have some module imported from a
> > database (who does this, really?),
>
> No one at the moment.  But think of it as an alternative to zip files
> if you use sqlite3 for the DB storage.  Several people have told me
> privately they would like to see an importer and loader for this.

I did stuff like this a while back, writing __import__ functions to
pull modules from a database or directly from an SVN repository. I'm
not saying it's a good idea, but it is being done.

Collin Winter


From jcarlson at uci.edu  Sun Feb 11 01:42:57 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sat, 10 Feb 2007 16:42:57 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702101432p1d99997bvf72ccdf77e8cb4cc@mail.gmail.com>
References: <20070209171022.ACA1.JCARLSON@uci.edu>
	<bbaeab100702101432p1d99997bvf72ccdf77e8cb4cc@mail.gmail.com>
Message-ID: <20070210160616.ACAA.JCARLSON@uci.edu>


"Brett Cannon" <brett at python.org> wrote:
> On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> >
> > "Brett Cannon" <brett at python.org> wrote:
> > > On 2/9/07, Josiah Carlson <jcarlson at uci.edu> wrote:
[snip]
> > How is it done now?
> 
> Importers and loaders; see PEP 302.

Wow, I should have started there when I was doing my relative import
work.  That's much nicer to work with.

[snip]

> > > > It also naturally leads to a __name__ semantic that Guido had suggested
> > > > to me when I was talking about relative imports:
> > > >
> > > >     goo.__name__ == '__main__'
> > > >     foo.__name__ == '__main__.foo'
> > > >     baz.__name__ == '__main__..bar.baz'
> > > >
> > > > Which could more or less be used with the current importer; it just
> > > > needs a special-casing of 'from . import ...' in the __main__ module.
> > >
> > > And I am trying to avoid special-casing for this.
> >
> > And it's only really an issue because we can't currently discern between
> > module or package, right?
> 
> Basically.  It's also where within a package the module is, so it
> isn't quite that simple.
> 
> The issue comes down to that we lose where a module is contained
> within a package hierarchy once __name__ gets set to __main__.  If we
> came up with another way to delineate that a module was being executed
> or some other way to specify where a module was in a package then the
> problem would be solved.  I prefer trying to change the former since
> the latter is perfectly handled with __name__ as-is when it isn't
> changed to __main__.

Kind-of.  It still seems to break in a few cases, which I believe to be
the result of current __name__ semantics (which can be fixed if __name__
could be anything, but I'll talk about that again later).

Say I wanted to do relative imports in paths below the current __main__
module.  The following works, and gives me the __main__-derived package
names that Guido suggested.

    >>> __path__ = [os.getcwd()]
    >>> from . import bar
    >>> bar
    <module '__main__.bar' from 'D:\Projects\python\py25\bar.pyc'>

So far so good.  What about "cousin" relative imports (an issue I've had
to deal with myself)?

    >>> from ..py25b import baz
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: Relative importpath too deep

No dice.  Sub-packages seem to be ok with this hack to __main__, but
nothing that goes to a parent of the "root" package of __main__.

My attempts to hack __main__ with...

    __name__ = '.'.join(os.getcwd().split('\\')[1:])

just got me "parent package not loaded".  Adding fake parent packages
all the way to the root of the drive, leaving the above name mangling in
place, allowed me to do the 'from ..py25b import baz' import in the
"main" module.


I like Guido's suggestion; allow multiple trailing dots off of
__main__.  That is, suppose that we were able to do the "from ..py25b
import baz" import, the __name__ of the imported object should be
__main__..py25b.baz .  Allowing that particular semantic would let us
keep the "if __name__ == '__main__':" thing we've been doing.  This,
however, would require a bunch of extra coding.


Anyways...I hear where you are coming from with your statements of 'if
__name__ could be anything, and we could train people to use ismain(),
then all of this relative import stuff could *just work*'.  It would
require inserting a bunch of (fake?) packages in valid Python name
parent paths (just in case people want to do cousin, etc., imports from
__main__).

You have convinced me.

 - Josiah



From ben at redfrontdoor.org  Sun Feb 11 11:07:18 2007
From: ben at redfrontdoor.org (Ben North)
Date: Sun, 11 Feb 2007 10:07:18 +0000
Subject: [Python-ideas] New syntax for 'dynamic' attribute access
Message-ID: <45CEEAD6.1000906@redfrontdoor.org>

Thanks for the responses on this.

In general:

Guido van Rossum:
 > I think you should submit this to the
 > PEP editor and argue on Python-dev for its inclusion in Python 2.6 --
 > there's no benefit that I see of waiting until 3.0.

Greg Falcon:
 > Wow!  I have to say this is a compelling idea.

Josiah Carlson:
 > My only concern with your propsed change is your draft implementation.

and on the syntax in particular:

Guido van Rossum:
 > I've thought of the same syntax.

Greg Falcon:
 > The syntax is a bit foreign looking, but [...] I feel like I could
 > learn to like it anyway.

Mostly positive, then, as far as the general idea and the syntax goes.

On the two-argument form, Greg Falcon wrote:
 > >         x = y.('foo_%d' % n, None)
 >
 > This is the one bit I really don't like.  y.('foobar') is arguably a
 > natural extension to y.foobar, but y.('foobar', None) isn't analogous
 > to any current syntax, and the multiple arguments make it look even
 > more dangerously like a call.
 >
 > In Python, you already have to be explicit when you're worried if the
 > thing you're accessing might not be there.

I was definitely in two minds about whether the extension to the
two-argument form was a win or a loss.  It does increase the power of
the new syntax, but it is certainly rather clumsy-looking.  I'd be happy
to prepare a patch with just the one-argument version if the consensus
is that the balance is that way.

Josiah Carlson:
 > My only concern with your propsed change is your draft
 > implementation. [...]
 >
 > Specifically, your changes to ceval.c and the compiler may have been
 > easier to implement, but it may negatively affect general Python
 > performance.  Have you run a recent pystone before and after the changes?

I hadn't done so, no, but have now tried some tests.  In fact, there is
some evidence of a slight negative effect on the general performance.
On my laptop, repeated pystone runs with 100,000 loops are quite
variable but there might be a performance penalty of around 1% with the
new code.  I'm a bit puzzled by this because of course the new opcodes
are never invoked in the pystone code --- does a switch statement become
slower the more cases there are?  (I tried grouping the cases with the
new opcodes all together at the end of the switch, but the noisiness of
the pystone results was such that I couldn't tell whether this helped.)
Or is it just that the core bytecode-interpretation loop is bigger so
has worse processor cache performance?

On the other hand, getting/setting dynamic attributes seems to be about
40--45% faster with the new syntax, probably because you avoid the
lookup of 'getattr' and the overhead of the function call.

Anyway, I'll experiment with the other implementation suggestions made
by Josiah, and in the meantime summarise the discussion so far to
python-dev as suggested by Guido.

Thanks,

Ben.



From jcarlson at uci.edu  Sun Feb 11 11:22:04 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 11 Feb 2007 02:22:04 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <20070210160616.ACAA.JCARLSON@uci.edu>
References: <bbaeab100702101432p1d99997bvf72ccdf77e8cb4cc@mail.gmail.com>
	<20070210160616.ACAA.JCARLSON@uci.edu>
Message-ID: <20070210214446.ACB0.JCARLSON@uci.edu>


Josiah Carlson <jcarlson at uci.edu> wrote:
> Anyways...I hear where you are coming from with your statements of 'if
> __name__ could be anything, and we could train people to use ismain(),
> then all of this relative import stuff could *just work*'.  It would
> require inserting a bunch of (fake?) packages in valid Python name
> parent paths (just in case people want to do cousin, etc., imports from
> __main__).
> 
> You have convinced me.

And in that vein, I have implemented a bit of code that mangles the
__name__ of the __main__ module, sets up pseudo-packages for parent
paths with valid Python names, imports __init__.py modules in ancestor
packages, adds an ismain() function to builtins, etc.

It allows for crazy things like...

    from ..uncle import cousin
    from ..parent import sibling
    #the above equivalent to:
    from . import sibling
    from .sibling import nephew

...all executed within the __main__ module (which gets a new __name__). 
Even better, it works with vanilla Python 2.5, and doesn't even require
an import hook.

The only unfortunate thing is that because you cannot predict how far up
the tree relative imports go, you cannot know how far up the paths one
should go in creating the ancestral packages.  My current (simple)
implementation goes as far up as the root, or the parent of the deepest
path with an __init__.py[cw] .

If you are curious, I can send you a copy off-list.

 - Josiah



From jcarlson at uci.edu  Sun Feb 11 11:39:16 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 11 Feb 2007 02:39:16 -0800
Subject: [Python-ideas] New syntax for 'dynamic' attribute access
In-Reply-To: <45CEEAD6.1000906@redfrontdoor.org>
References: <45CEEAD6.1000906@redfrontdoor.org>
Message-ID: <20070211022329.ACB4.JCARLSON@uci.edu>


Ben North <ben at redfrontdoor.org> wrote:
> Josiah Carlson:
>  > My only concern with your propsed change is your draft
>  > implementation. [...]
>  >
>  > Specifically, your changes to ceval.c and the compiler may have been
>  > easier to implement, but it may negatively affect general Python
>  > performance.  Have you run a recent pystone before and after the changes?
> 
> I hadn't done so, no, but have now tried some tests.  In fact, there is
> some evidence of a slight negative effect on the general performance.
> On my laptop, repeated pystone runs with 100,000 loops are quite
> variable but there might be a performance penalty of around 1% with the
> new code.  I'm a bit puzzled by this because of course the new opcodes
> are never invoked in the pystone code --- does a switch statement become
> slower the more cases there are?  (I tried grouping the cases with the
> new opcodes all together at the end of the switch, but the noisiness of
> the pystone results was such that I couldn't tell whether this helped.)
> Or is it just that the core bytecode-interpretation loop is bigger so
> has worse processor cache performance?

There have been a few examples where nontrivial blocks of code inserted
into the mainloop of ceval.c slowed down Python.  Indeed, it turns out
that Python fit into the processor cache *just right*, and with much
more, the cache was blown.  Then again, 1% loss isn't really substantial,
I wouldn't worry too much.

Get a PEP number and talk to others in python-dev, put your patch up on
sourceforge so that people on other platforms can test it out (and
verify the 1% loss, if any), and if you are curious, then try out the
idea I offered up - which has the potential to slow down *all* attribute
lookups (though minimizes the source additions to the ceval.c mainloop).


> On the other hand, getting/setting dynamic attributes seems to be about
> 40--45% faster with the new syntax, probably because you avoid the
> lookup of 'getattr' and the overhead of the function call.

That speedup sounds reasonable.


> Anyway, I'll experiment with the other implementation suggestions made
> by Josiah, and in the meantime summarise the discussion so far to
> python-dev as suggested by Guido.

Great!  I look forward to seeing this in Python 2.6 .

 - Josiah



From ben at redfrontdoor.org  Sun Feb 11 13:51:06 2007
From: ben at redfrontdoor.org (Ben North)
Date: Sun, 11 Feb 2007 12:51:06 +0000
Subject: [Python-ideas] New syntax for 'dynamic' attribute access
In-Reply-To: <20070211022329.ACB4.JCARLSON@uci.edu>
References: <45CEEAD6.1000906@redfrontdoor.org>
	<20070211022329.ACB4.JCARLSON@uci.edu>
Message-ID: <45CF113A.4070805@redfrontdoor.org>

Josiah Carlson wrote:
 > Get a PEP number and talk to others in python-dev [...]

I've made the request, and if/when a PEP-number is allocated, I'll move
the discussion to python-dev. Thanks for the comments and suggestions.

Ben.



From brett at python.org  Sun Feb 11 18:36:01 2007
From: brett at python.org (Brett Cannon)
Date: Sun, 11 Feb 2007 09:36:01 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <20070210214446.ACB0.JCARLSON@uci.edu>
References: <bbaeab100702101432p1d99997bvf72ccdf77e8cb4cc@mail.gmail.com>
	<20070210160616.ACAA.JCARLSON@uci.edu>
	<20070210214446.ACB0.JCARLSON@uci.edu>
Message-ID: <bbaeab100702110936ufab73c7m7dfbe2b6917b009@mail.gmail.com>

On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> Josiah Carlson <jcarlson at uci.edu> wrote:
> > Anyways...I hear where you are coming from with your statements of 'if
> > __name__ could be anything, and we could train people to use ismain(),
> > then all of this relative import stuff could *just work*'.  It would
> > require inserting a bunch of (fake?) packages in valid Python name
> > parent paths (just in case people want to do cousin, etc., imports from
> > __main__).
> >
> > You have convinced me.
>
> And in that vein, I have implemented a bit of code that mangles the
> __name__ of the __main__ module, sets up pseudo-packages for parent
> paths with valid Python names, imports __init__.py modules in ancestor
> packages, adds an ismain() function to builtins, etc.
>
> It allows for crazy things like...
>
>     from ..uncle import cousin
>     from ..parent import sibling
>     #the above equivalent to:
>     from . import sibling
>     from .sibling import nephew
>
> ...all executed within the __main__ module (which gets a new __name__).
> Even better, it works with vanilla Python 2.5, and doesn't even require
> an import hook.
>
> The only unfortunate thing is that because you cannot predict how far up
> the tree relative imports go, you cannot know how far up the paths one
> should go in creating the ancestral packages.  My current (simple)
> implementation goes as far up as the root, or the parent of the deepest
> path with an __init__.py[cw] .
>

Just to make sure that I understand this correctly, __name__ is set to
__main__ for the module that is being executed.  Then other modules in
the package are also called __main__, but with the proper dots and
such to resolve to the proper depth in the package?

> If you are curious, I can send you a copy off-list.

I have way too much on my plate right now to dive into it right now,
but I assume the patch is either against runpy or my import code?

-Brett


From brett at python.org  Sun Feb 11 18:41:19 2007
From: brett at python.org (Brett Cannon)
Date: Sun, 11 Feb 2007 09:41:19 -0800
Subject: [Python-ideas] New syntax for 'dynamic' attribute access
In-Reply-To: <45CF113A.4070805@redfrontdoor.org>
References: <45CEEAD6.1000906@redfrontdoor.org>
	<20070211022329.ACB4.JCARLSON@uci.edu>
	<45CF113A.4070805@redfrontdoor.org>
Message-ID: <bbaeab100702110941h5f7495f0kdb7a3713c81ea767@mail.gmail.com>

On 2/11/07, Ben North <ben at redfrontdoor.org> wrote:
> Josiah Carlson wrote:
>  > Get a PEP number and talk to others in python-dev [...]
>
> I've made the request, and if/when a PEP-number is allocated, I'll move
> the discussion to python-dev. Thanks for the comments and suggestions.
>

I wouldn't wait for the PEP number.  The PEP editors, just like anyone
else, can get busy.  Go ahead and keep working on stuff.  If you write
a PEP, just fill the number with XXXs and say it is a draft or
proto-PEP when you post it somewhere.  Lots of people have written
entire PEPs before getting a number so it is not unexpected.

As Guido said, get something on python-dev for people to discuss, even
if it is as simple as the syntactic proposal part.

-Brett


From jcarlson at uci.edu  Sun Feb 11 19:55:33 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 11 Feb 2007 10:55:33 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702110936ufab73c7m7dfbe2b6917b009@mail.gmail.com>
References: <20070210214446.ACB0.JCARLSON@uci.edu>
	<bbaeab100702110936ufab73c7m7dfbe2b6917b009@mail.gmail.com>
Message-ID: <20070211100112.ACBA.JCARLSON@uci.edu>


"Brett Cannon" <brett at python.org> wrote:
> 
> On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> >
> > Josiah Carlson <jcarlson at uci.edu> wrote:
> > > Anyways...I hear where you are coming from with your statements of 'if
> > > __name__ could be anything, and we could train people to use ismain(),
> > > then all of this relative import stuff could *just work*'.  It would
> > > require inserting a bunch of (fake?) packages in valid Python name
> > > parent paths (just in case people want to do cousin, etc., imports from
> > > __main__).
> > >
> > > You have convinced me.
> >
> > And in that vein, I have implemented a bit of code that mangles the
> > __name__ of the __main__ module, sets up pseudo-packages for parent
> > paths with valid Python names, imports __init__.py modules in ancestor
> > packages, adds an ismain() function to builtins, etc.
> >
> > It allows for crazy things like...
> >
> >     from ..uncle import cousin
> >     from ..parent import sibling
> >     #the above equivalent to:
> >     from . import sibling
> >     from .sibling import nephew
> >
> > ...all executed within the __main__ module (which gets a new __name__).
> > Even better, it works with vanilla Python 2.5, and doesn't even require
> > an import hook.
> >
> > The only unfortunate thing is that because you cannot predict how far up
> > the tree relative imports go, you cannot know how far up the paths one
> > should go in creating the ancestral packages.  My current (simple)
> > implementation goes as far up as the root, or the parent of the deepest
> > path with an __init__.py[cw] .
> >
> 
> Just to make sure that I understand this correctly, __name__ is set to
> __main__ for the module that is being executed.  Then other modules in
> the package are also called __main__, but with the proper dots and
> such to resolve to the proper depth in the package?

No.  Say, for example, that you had a tree like the following.

    .../
        pk1/
            pk2/
                __init__.py
                pk3/
                    __init__.py
                    run.py

Also say that run.py was run from the command line, and the relative
import code that I have written gets executed.  The following assumes
that at least a "dummy" module is inserted into sys.modules['__main__']

1) A fake package called 'pk1' with __path__ == ['../pk1'] is inserted
into sys.modules.
2) 'pk1.pk2' is imported as per package rules (__init__.py is executed),
and gets a __path__ == ['../pk1/pk2/'] .
3) 'pk1.pk2.pk3' is imported as per package rules (__init__.py is
executed), and gets a __path__ == ['../pk1/pk2/pk3'] .
4) We fetch sys.packages['__main__'], give it a new __name__ of
'pk1.pk2.pk3.__main__', but don't give it a path.  Also insert the
module into sys.modules['pk1.pk2.pk3.__main__'].
5) Add ismain() to builtins.
6) The remainder of run.py is executed.


> > If you are curious, I can send you a copy off-list.
> 
> I have way too much on my plate right now to dive into it right now,
> but I assume the patch is either against runpy or my import code?

No.  It's actually a standalone module.  When imported (presumably from
__main__ as the first thing it does), it performs the mangling,
importing, etc.  I'm sure I could modify runpy to do all of this, but
only if alter_sys was True.

I could probably do the same with your import code, where can I find it?

One reason *not* to do the __main__..uncle.cousin namings is that it is
not clear how one should go about removing those __main__ trailing dots
without examining __main__'s __file__ all the time, especially with
non-filesystem imports with nonsensical __file__.


 - Josiah



From brett at python.org  Sun Feb 11 21:03:21 2007
From: brett at python.org (Brett Cannon)
Date: Sun, 11 Feb 2007 12:03:21 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <20070211100112.ACBA.JCARLSON@uci.edu>
References: <20070210214446.ACB0.JCARLSON@uci.edu>
	<bbaeab100702110936ufab73c7m7dfbe2b6917b009@mail.gmail.com>
	<20070211100112.ACBA.JCARLSON@uci.edu>
Message-ID: <bbaeab100702111203j49005164nad8236cfe58a9f5a@mail.gmail.com>

On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> "Brett Cannon" <brett at python.org> wrote:
> >
> > On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > >
> > > Josiah Carlson <jcarlson at uci.edu> wrote:
> > > > Anyways...I hear where you are coming from with your statements of 'if
> > > > __name__ could be anything, and we could train people to use ismain(),
> > > > then all of this relative import stuff could *just work*'.  It would
> > > > require inserting a bunch of (fake?) packages in valid Python name
> > > > parent paths (just in case people want to do cousin, etc., imports from
> > > > __main__).
> > > >
> > > > You have convinced me.
> > >
> > > And in that vein, I have implemented a bit of code that mangles the
> > > __name__ of the __main__ module, sets up pseudo-packages for parent
> > > paths with valid Python names, imports __init__.py modules in ancestor
> > > packages, adds an ismain() function to builtins, etc.
> > >
> > > It allows for crazy things like...
> > >
> > >     from ..uncle import cousin
> > >     from ..parent import sibling
> > >     #the above equivalent to:
> > >     from . import sibling
> > >     from .sibling import nephew
> > >
> > > ...all executed within the __main__ module (which gets a new __name__).
> > > Even better, it works with vanilla Python 2.5, and doesn't even require
> > > an import hook.
> > >
> > > The only unfortunate thing is that because you cannot predict how far up
> > > the tree relative imports go, you cannot know how far up the paths one
> > > should go in creating the ancestral packages.  My current (simple)
> > > implementation goes as far up as the root, or the parent of the deepest
> > > path with an __init__.py[cw] .
> > >
> >
> > Just to make sure that I understand this correctly, __name__ is set to
> > __main__ for the module that is being executed.  Then other modules in
> > the package are also called __main__, but with the proper dots and
> > such to resolve to the proper depth in the package?
>
> No.  Say, for example, that you had a tree like the following.
>
>     .../
>         pk1/
>             pk2/
>                 __init__.py
>                 pk3/
>                     __init__.py
>                     run.py
>
> Also say that run.py was run from the command line, and the relative
> import code that I have written gets executed.  The following assumes
> that at least a "dummy" module is inserted into sys.modules['__main__']
>
> 1) A fake package called 'pk1' with __path__ == ['../pk1'] is inserted
> into sys.modules.
> 2) 'pk1.pk2' is imported as per package rules (__init__.py is executed),
> and gets a __path__ == ['../pk1/pk2/'] .
> 3) 'pk1.pk2.pk3' is imported as per package rules (__init__.py is
> executed), and gets a __path__ == ['../pk1/pk2/pk3'] .
> 4) We fetch sys.packages['__main__'], give it a new __name__ of
> 'pk1.pk2.pk3.__main__', but don't give it a path.  Also insert the
> module into sys.modules['pk1.pk2.pk3.__main__'].
> 5) Add ismain() to builtins.
> 6) The remainder of run.py is executed.
>

Ah, OK.  Didn't realize you had gone ahead and done step 5.

>
> > > If you are curious, I can send you a copy off-list.
> >
> > I have way too much on my plate right now to dive into it right now,
> > but I assume the patch is either against runpy or my import code?
>
> No.  It's actually a standalone module.  When imported (presumably from
> __main__ as the first thing it does), it performs the mangling,
> importing, etc.  I'm sure I could modify runpy to do all of this, but
> only if alter_sys was True.
>
> I could probably do the same with your import code, where can I find it?
>

It's in the sandbox under import_in_py if you want the Python version.

-Brett


From jcarlson at uci.edu  Mon Feb 12 06:16:17 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 11 Feb 2007 21:16:17 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <bbaeab100702111203j49005164nad8236cfe58a9f5a@mail.gmail.com>
References: <20070211100112.ACBA.JCARLSON@uci.edu>
	<bbaeab100702111203j49005164nad8236cfe58a9f5a@mail.gmail.com>
Message-ID: <20070211195852.ACC2.JCARLSON@uci.edu>


"Brett Cannon" <brett at python.org> wrote:
> On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> >
> > "Brett Cannon" <brett at python.org> wrote:
> > >
> > > On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > > >
> > > > Josiah Carlson <jcarlson at uci.edu> wrote:
> > > > > Anyways...I hear where you are coming from with your statements of 'if
> > > > > __name__ could be anything, and we could train people to use ismain(),
> > > > > then all of this relative import stuff could *just work*'.  It would
> > > > > require inserting a bunch of (fake?) packages in valid Python name
> > > > > parent paths (just in case people want to do cousin, etc., imports from
> > > > > __main__).
> > > > >
> > > > > You have convinced me.
> > > >
> > > > And in that vein, I have implemented a bit of code that mangles the
> > > > __name__ of the __main__ module, sets up pseudo-packages for parent
> > > > paths with valid Python names, imports __init__.py modules in ancestor
> > > > packages, adds an ismain() function to builtins, etc.
> > > >
> > > > It allows for crazy things like...
> > > >
> > > >     from ..uncle import cousin
> > > >     from ..parent import sibling
> > > >     #the above equivalent to:
> > > >     from . import sibling
> > > >     from .sibling import nephew
> > > >
> > > > ...all executed within the __main__ module (which gets a new __name__).
> > > > Even better, it works with vanilla Python 2.5, and doesn't even require
> > > > an import hook.
> > > >
> > > > The only unfortunate thing is that because you cannot predict how far up
> > > > the tree relative imports go, you cannot know how far up the paths one
> > > > should go in creating the ancestral packages.  My current (simple)
> > > > implementation goes as far up as the root, or the parent of the deepest
> > > > path with an __init__.py[cw] .
> > > >
> > >
> > > Just to make sure that I understand this correctly, __name__ is set to
> > > __main__ for the module that is being executed.  Then other modules in
> > > the package are also called __main__, but with the proper dots and
> > > such to resolve to the proper depth in the package?
> >
> > No.  Say, for example, that you had a tree like the following.
> >
> >     .../
> >         pk1/
> >             pk2/
> >                 __init__.py
> >                 pk3/
> >                     __init__.py
> >                     run.py
> >
> > Also say that run.py was run from the command line, and the relative
> > import code that I have written gets executed.  The following assumes
> > that at least a "dummy" module is inserted into sys.modules['__main__']
> >
> > 1) A fake package called 'pk1' with __path__ == ['../pk1'] is inserted
> > into sys.modules.
> > 2) 'pk1.pk2' is imported as per package rules (__init__.py is executed),
> > and gets a __path__ == ['../pk1/pk2/'] .
> > 3) 'pk1.pk2.pk3' is imported as per package rules (__init__.py is
> > executed), and gets a __path__ == ['../pk1/pk2/pk3'] .
> > 4) We fetch sys.packages['__main__'], give it a new __name__ of
> > 'pk1.pk2.pk3.__main__', but don't give it a path.  Also insert the
> > module into sys.modules['pk1.pk2.pk3.__main__'].
> > 5) Add ismain() to builtins.
> > 6) The remainder of run.py is executed.
> >
> 
> Ah, OK.  Didn't realize you had gone ahead and done step 5.

Yep, it was easy:

    def ismain():
        try:
            raise ZeroDivisionError()
        except ZeroDivisionError:
            f = sys.exc_info()[2].tb_frame.f_back
        try:
            return sys.modules[f.f_globals['__name__']] is sys.modules['__main__']
        except KeyError:
            return False

With the current semantics, reload would also need to be changed to
update both __main__ and whatever.__main__ in sys.modules.


> It's in the sandbox under import_in_py if you want the Python version.

Great, found it.

One issue with the code that I've been writing is that it more or less
relies on the idea of a "root package", and that discovering the root
package can be done in a straightforward way.  In a filesystem import,
it looks at the path in which the __main__ module lies and ancestors up
to the root, or the parent path of a path with an __init__.py[cw] module.

For code in which __init__.py[cw] modules aren't merely placeholders to
turn a path into a package, this could result in "undesireable" code
being run prior to the __main__ module.

It is also ambiguous when confronted with database imports in which the
command line is something like 'python -m dbimport.sub1.sub2.runme'.  Do
we also create/insert pseudo packages for the current path in the
filesystem, potentially changing the "name" to something like
"pkg1.pkg2.dbimport.sub1.sub2.runme"?  And really, this question is
applicable to any 'python -m' command line.


We obviously have a few options.  Among them;
1) make the above behavior optional with a __future__ import, must be
done at the top of a __main__ module (ignored in all other cases)
2) along with 1, only perform the above when we use imports in a
filesystem (zip imports are fine).
3) allow for a module variable to define how many ancestral paths are
inserted (to prevent unwanted/unnecessary __init__ modules from being
executed).
4) come up with a semantic for database and other non-filesystem imports.
5) toss the stuff I've hacked and more or less proposed.


Having read PEP 328 again, it doesn't specify how non-filesystem imports
should be handled, nor how to handle things like 'python -m', so we may
want to just ignore them, and do the mangling just prior to the
execution of the code for the __main__ module.


 - Josiah



From brett at python.org  Mon Feb 12 22:43:55 2007
From: brett at python.org (Brett Cannon)
Date: Mon, 12 Feb 2007 13:43:55 -0800
Subject: [Python-ideas] Packages and Import
In-Reply-To: <20070211195852.ACC2.JCARLSON@uci.edu>
References: <20070211100112.ACBA.JCARLSON@uci.edu>
	<bbaeab100702111203j49005164nad8236cfe58a9f5a@mail.gmail.com>
	<20070211195852.ACC2.JCARLSON@uci.edu>
Message-ID: <bbaeab100702121343r34032c58nd71deec9daa5ad@mail.gmail.com>

On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> "Brett Cannon" <brett at python.org> wrote:
> > On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > >
> > > "Brett Cannon" <brett at python.org> wrote:
> > > >
> > > > On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > > > >
> > > > > Josiah Carlson <jcarlson at uci.edu> wrote:
> > > > > > Anyways...I hear where you are coming from with your statements of 'if
> > > > > > __name__ could be anything, and we could train people to use ismain(),
> > > > > > then all of this relative import stuff could *just work*'.  It would
> > > > > > require inserting a bunch of (fake?) packages in valid Python name
> > > > > > parent paths (just in case people want to do cousin, etc., imports from
> > > > > > __main__).
> > > > > >
> > > > > > You have convinced me.
> > > > >
> > > > > And in that vein, I have implemented a bit of code that mangles the
> > > > > __name__ of the __main__ module, sets up pseudo-packages for parent
> > > > > paths with valid Python names, imports __init__.py modules in ancestor
> > > > > packages, adds an ismain() function to builtins, etc.
> > > > >
> > > > > It allows for crazy things like...
> > > > >
> > > > >     from ..uncle import cousin
> > > > >     from ..parent import sibling
> > > > >     #the above equivalent to:
> > > > >     from . import sibling
> > > > >     from .sibling import nephew
> > > > >
> > > > > ...all executed within the __main__ module (which gets a new __name__).
> > > > > Even better, it works with vanilla Python 2.5, and doesn't even require
> > > > > an import hook.
> > > > >
> > > > > The only unfortunate thing is that because you cannot predict how far up
> > > > > the tree relative imports go, you cannot know how far up the paths one
> > > > > should go in creating the ancestral packages.  My current (simple)
> > > > > implementation goes as far up as the root, or the parent of the deepest
> > > > > path with an __init__.py[cw] .
> > > > >
> > > >
> > > > Just to make sure that I understand this correctly, __name__ is set to
> > > > __main__ for the module that is being executed.  Then other modules in
> > > > the package are also called __main__, but with the proper dots and
> > > > such to resolve to the proper depth in the package?
> > >
> > > No.  Say, for example, that you had a tree like the following.
> > >
> > >     .../
> > >         pk1/
> > >             pk2/
> > >                 __init__.py
> > >                 pk3/
> > >                     __init__.py
> > >                     run.py
> > >
> > > Also say that run.py was run from the command line, and the relative
> > > import code that I have written gets executed.  The following assumes
> > > that at least a "dummy" module is inserted into sys.modules['__main__']
> > >
> > > 1) A fake package called 'pk1' with __path__ == ['../pk1'] is inserted
> > > into sys.modules.
> > > 2) 'pk1.pk2' is imported as per package rules (__init__.py is executed),
> > > and gets a __path__ == ['../pk1/pk2/'] .
> > > 3) 'pk1.pk2.pk3' is imported as per package rules (__init__.py is
> > > executed), and gets a __path__ == ['../pk1/pk2/pk3'] .
> > > 4) We fetch sys.packages['__main__'], give it a new __name__ of
> > > 'pk1.pk2.pk3.__main__', but don't give it a path.  Also insert the
> > > module into sys.modules['pk1.pk2.pk3.__main__'].
> > > 5) Add ismain() to builtins.
> > > 6) The remainder of run.py is executed.
> > >
> >
> > Ah, OK.  Didn't realize you had gone ahead and done step 5.
>
> Yep, it was easy:
>
>     def ismain():
>         try:
>             raise ZeroDivisionError()
>         except ZeroDivisionError:
>             f = sys.exc_info()[2].tb_frame.f_back
>         try:
>             return sys.modules[f.f_globals['__name__']] is sys.modules['__main__']
>         except KeyError:
>             return False
>
> With the current semantics, reload would also need to be changed to
> update both __main__ and whatever.__main__ in sys.modules.
>
>
> > It's in the sandbox under import_in_py if you want the Python version.
>
> Great, found it.
>
> One issue with the code that I've been writing is that it more or less
> relies on the idea of a "root package", and that discovering the root
> package can be done in a straightforward way.  In a filesystem import,
> it looks at the path in which the __main__ module lies and ancestors up
> to the root, or the parent path of a path with an __init__.py[cw] module.
>
> For code in which __init__.py[cw] modules aren't merely placeholders to
> turn a path into a package, this could result in "undesireable" code
> being run prior to the __main__ module.
>
> It is also ambiguous when confronted with database imports in which the
> command line is something like 'python -m dbimport.sub1.sub2.runme'.  Do
> we also create/insert pseudo packages for the current path in the
> filesystem, potentially changing the "name" to something like
> "pkg1.pkg2.dbimport.sub1.sub2.runme"?  And really, this question is
> applicable to any 'python -m' command line.
>
>
> We obviously have a few options.  Among them;
> 1) make the above behavior optional with a __future__ import, must be
> done at the top of a __main__ module (ignored in all other cases)
> 2) along with 1, only perform the above when we use imports in a
> filesystem (zip imports are fine).
> 3) allow for a module variable to define how many ancestral paths are
> inserted (to prevent unwanted/unnecessary __init__ modules from being
> executed).
> 4) come up with a semantic for database and other non-filesystem imports.
> 5) toss the stuff I've hacked and more or less proposed.
>

Beats me.  =)  My brain is fried at the moment so I don't have a good
answer at the moment.

-Brett


From rrr at ronadam.com  Tue Feb 13 03:19:14 2007
From: rrr at ronadam.com (Ron Adam)
Date: Mon, 12 Feb 2007 20:19:14 -0600
Subject: [Python-ideas] Packages and Import
In-Reply-To: <20070211195852.ACC2.JCARLSON@uci.edu>
References: <20070211100112.ACBA.JCARLSON@uci.edu>	<bbaeab100702111203j49005164nad8236cfe58a9f5a@mail.gmail.com>
	<20070211195852.ACC2.JCARLSON@uci.edu>
Message-ID: <45D12022.4030704@ronadam.com>

Josiah Carlson wrote:
> "Brett Cannon" <brett at python.org> wrote:
>> On 2/11/07, Josiah Carlson <jcarlson at uci.edu> wrote:

>>> Also say that run.py was run from the command line, and the relative
>>> import code that I have written gets executed.  The following assumes
>>> that at least a "dummy" module is inserted into sys.modules['__main__']
>>>
>>> 1) A fake package called 'pk1' with __path__ == ['../pk1'] is inserted
>>> into sys.modules.

For some reason I don't like the idea of fake packages.  Seems too much like a 
hack to me.  That could be just me though.


>>> 2) 'pk1.pk2' is imported as per package rules (__init__.py is executed),
>>> and gets a __path__ == ['../pk1/pk2/'] .
>>> 3) 'pk1.pk2.pk3' is imported as per package rules (__init__.py is
>>> executed), and gets a __path__ == ['../pk1/pk2/pk3'] .
>>> 4) We fetch sys.packages['__main__'], give it a new __name__ of
>>> 'pk1.pk2.pk3.__main__', but don't give it a path.  Also insert the
>>> module into sys.modules['pk1.pk2.pk3.__main__'].
>>> 5) Add ismain() to builtins.
>>> 6) The remainder of run.py is executed.
>>>
>> Ah, OK.  Didn't realize you had gone ahead and done step 5.
> 
> Yep, it was easy:
> 
>     def ismain():
>         try:
>             raise ZeroDivisionError()
>         except ZeroDivisionError:
>             f = sys.exc_info()[2].tb_frame.f_back
>         try:
>             return sys.modules[f.f_globals['__name__']] is sys.modules['__main__']
>         except KeyError:
>             return False
> 
> With the current semantics, reload would also need to be changed to
> update both __main__ and whatever.__main__ in sys.modules.
> 
> 
>> It's in the sandbox under import_in_py if you want the Python version.
> 
> Great, found it.
> 
> One issue with the code that I've been writing is that it more or less
> relies on the idea of a "root package", and that discovering the root
> package can be done in a straightforward way.  In a filesystem import,
> it looks at the path in which the __main__ module lies and ancestors up
> to the root, or the parent path of a path with an __init__.py[cw] module.
 >
 > For code in which __init__.py[cw] modules aren't merely placeholders to
 > turn a path into a package, this could result in "undesireable" code
 > being run prior to the __main__ module.


I think the idea of a "root package", Is good.  I agree with that.

Think of it this way... You aren't running a module in a package as if it were a 
top level module;  you are entering a package from a different access point. 
The package should still be cohesive.


If someone wants to run a module as if it were not in a package, but have it 
within a package's directory structure, then they can put it in a sub directory 
that doesn't have an __init__ file, and add that directory to sys.path. Those 
modules would then be treated as top level modules if you execute them directly. 
  You would also import them as if they were top level modules with no package 
prefix.  (they are on sys.path)

Then "package" modules can always use a "package" module importer to start them, 
and modules not part of packages can always use a simpler "module" importer.  It 
would be good if there was no question as to which is which and which importer 
to use.


(This part repeats some things I wrote earlier.)

If you are running a module that depends on the __init__ to add it's path to the 
package, then it's not part of the package until the __init__ is executed.  And 
of course the module can't know that before then, so it should be executed as it 
is, where it is, if there's not an __init__ in the same directory. It should 
then be treated as a top level module.

It should be up to the package designer to take this into account, and not the 
import designer to determine what the package designer intended.  Modules used 
in such ways can't know how many packages, or which packages, will use them in 
this indirect way.


> It is also ambiguous when confronted with database imports in which the
> command line is something like 'python -m dbimport.sub1.sub2.runme'.  Do
> we also create/insert pseudo packages for the current path in the
> filesystem, potentially changing the "name" to something like
> "pkg1.pkg2.dbimport.sub1.sub2.runme"?  And really, this question is
> applicable to any 'python -m' command line.


Is the 'python -m' meant to run a module in a package as if it was a top level 
module?  Or is it meant, (as I beleave), to allow you to use the python name 
instead of the file name?  Help isn't clear on this point.

       -m mod : run library module as a script (terminates option list)


One of my main points when starting this thread was...

> Make pythons concept of a package, (currently an informal type), be stronger 
> than that of the underlying file system search path and directory structure.

Which would mean to me that the __init__(s) in packages should always be run 
before modules in packages.  That would simplify the import problem I think, and 
dummy packages would not be needed.

Also this only needs to be done for the first module in the package that is 
imported. After that, then any additional modules added to the package by the 
__init__ becomes importable.  You just can't use those modules as an entry point 
the package.

It also makes the code clearer from a reading standpoint as it's becomes easier 
to determine what the relationships are.


> We obviously have a few options.  Among them;
> 1) make the above behavior optional with a __future__ import, must be
> done at the top of a __main__ module (ignored in all other cases)
> 2) along with 1, only perform the above when we use imports in a
> filesystem (zip imports are fine).
> 3) allow for a module variable to define how many ancestral paths are
> inserted (to prevent unwanted/unnecessary __init__ modules from being
> executed).
> 4) come up with a semantic for database and other non-filesystem imports.
> 5) toss the stuff I've hacked and more or less proposed.
> 
> 
> Having read PEP 328 again, it doesn't specify how non-filesystem imports
> should be handled, nor how to handle things like 'python -m', so we may
> want to just ignore them, and do the mangling just prior to the
> execution of the code for the __main__ module.

Brett said things get a simpler if __name__ was always the module name.  How 
about adding a pair of attributes to modules:

     __package__  -> package name  # full package.sub-packages... etc.
     __module__   -> module name   # is "" if it's a package.

If a module isn't in a package then __package__ is "".

Then __name__ == "__main__" could still work until ismain() is introduced. 
__name__ wouldn't be needed for import uses in this case.

Cheers,
   Ron




From jimjjewett at gmail.com  Sun Feb 18 04:18:53 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sat, 17 Feb 2007 22:18:53 -0500
Subject: [Python-ideas] immutable classes [was: pre-PEP: Default Argument
	Expressions]
Message-ID: <fb6fbf560702171918y33d79991j727b2ccdfeef8513@mail.gmail.com>

I have added python-ideas to the Cc list, and suggest removing
python-3000 from additional replies.

BJ?rn Lindqvist gave an example explaining why he might want to
re-evaluate mutable default arguments.  It still looks like like buggy
code, but it isn't the error I was expecting -- and I think it comes
from the difficulty of declaring something immutable.

On 2/15/07, BJ?rn Lindqvist <bjourne at gmail.com> wrote:
> On 2/15/07, Jim Jewett <jimjjewett at gmail.com> wrote:

> > Then are there *any* good use cases for [non-persistent mutable defaults]

> > (1)  Not really (treated as) mutable.  ==> Doesn't care

> >     >>> def f(extra_settings={}) ...

> > usually doesn't modify or even store extra_settings;  ...

> That is dangerous code. Sooner or later someone will modify the
> extra_settings dict.

How?

    >>> f.func_defaults[0]['key']=value

may be misguided, but it probably isn't an accident.

BJ?rn's example does store the mutable directly, but it makes a bit
more sense because it looks like a complex object rather than just a
mapping.

>     class Vector:
>         def __init__(self, x, y, z):
>             self.x = x
>             self.y = y
>             self.z = z

>     class Ray:
>         def __init__(self, direction, origin = Vector(0, 0, 0)):
>             self.direction = direction
>             self.origin = origin
>
>     ray1 = Ray(Vector(0, 0, -1))
>     ray2 = Ray(Vector(0, 0, 1))
>     ray3 = Ray(Vector(-1, 0, 0), Vector(2, 3, 4))

> The above code looks quite nice, but is wrong.

Why is vector mutable?

Is the real problem that it is too hard to declare objects or
attributes immutable?

My solution is below, but I'll grant that it isn't as straightforward
as I would have liked.  Is this something that could be solved with a
recipe, or a factory to make immutable classes?

>>> class Vector3D(tuple):
...         def __new__(self, x, y, z):
...             return super(Vector3D, self).__new__(self, (x, y, z))
...         x=property(lambda self: self[0])
...         y=property(lambda self: self[1])
...         z=property(lambda self: self[2])

-jJ


From jcarlson at uci.edu  Sun Feb 18 05:25:29 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sat, 17 Feb 2007 20:25:29 -0800
Subject: [Python-ideas] immutable classes [was: pre-PEP: Default
	Argument Expressions]
In-Reply-To: <fb6fbf560702171918y33d79991j727b2ccdfeef8513@mail.gmail.com>
References: <fb6fbf560702171918y33d79991j727b2ccdfeef8513@mail.gmail.com>
Message-ID: <20070217202422.AD91.JCARLSON@uci.edu>


"Jim Jewett" <jimjjewett at gmail.com> wrote:
> >>> class Vector3D(tuple):
> ...         def __new__(self, x, y, z):
> ...             return super(Vector3D, self).__new__(self, (x, y, z))
> ...         x=property(lambda self: self[0])
> ...         y=property(lambda self: self[1])
> ...         z=property(lambda self: self[2])

That looks very much like Raymond Hettinger's named tuple recipe,
recently offered in python-dev:

    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261

 - Josiah



From jimjjewett at gmail.com  Sun Feb 18 17:25:55 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sun, 18 Feb 2007 11:25:55 -0500
Subject: [Python-ideas] immutable classes [was: pre-PEP: Default
	Argument Expressions]
In-Reply-To: <20070217202422.AD91.JCARLSON@uci.edu>
References: <fb6fbf560702171918y33d79991j727b2ccdfeef8513@mail.gmail.com>
	<20070217202422.AD91.JCARLSON@uci.edu>
Message-ID: <fb6fbf560702180825u4b7de14fjd2ba9f61e63b0291@mail.gmail.com>

Well, except that Raymond's is indeed better -- so consider this a
request for inclusion in 2.6

On 2/17/07, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> "Jim Jewett" <jimjjewett at gmail.com> wrote:
> > >>> class Vector3D(tuple):
> > ...         def __new__(self, x, y, z):
> > ...             return super(Vector3D, self).__new__(self, (x, y, z))
> > ...         x=property(lambda self: self[0])
> > ...         y=property(lambda self: self[1])
> > ...         z=property(lambda self: self[2])
>
> That looks very much like Raymond Hettinger's named tuple recipe,
> recently offered in python-dev:
>
>     http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261

-jJ


From pym.aldebaran at gmail.com  Sun Feb 18 21:16:59 2007
From: pym.aldebaran at gmail.com (Pierre-Yves Martin)
Date: Sun, 18 Feb 2007 21:16:59 +0100
Subject: [Python-ideas] local variable access : __getlocal__ and __setlocal__
In-Reply-To: <324b1c9b0702180159x7bc59296o44c01eb2b7d13f56@mail.gmail.com>
References: <324b1c9b0702180159x7bc59296o44c01eb2b7d13f56@mail.gmail.com>
Message-ID: <324b1c9b0702181216w5f0468dcme2ae79a6b5ecc546@mail.gmail.com>

Currently it's possible to customize attribute access via methods :

__getattribute__(self, name)
__getattr__(self, name)
__setattr__(self, name, value)


but it is not possible to customize local variable access. It would be
useful for example to allow implementation of a local constant or any other
on-access/on-modification behavior. The function should could be:

__getlocal__(name)
__setlocal__(name, value)

By default they are just access to the local dictionnary for reading and
writting (AFAIK the only current way to access this dictionnary is using the
locals()  built-in).
Here is a sample code of typical usage :

#######################
#Sample 1 : Tracing local access

old_getlocal = __getlocal__
old_setlocal = __setlocal__

#redefine the built-ins
def __getlocal__(name):
    print 'accessing %s' % name
    return old_getlocal(name)

def __setlocal__(name, value):
    print 'setting %s with value %s' % (name, value)
    old_setlocal(name, value)

a = 1
b = 2
c = 3

#setting b and accessing  a
b = a + 1

###################
#Sample 2 : Constants
old_getlocal = __getlocal__
old_setlocal = __setlocal__
class ConstantError(Exception):pass
constants = []

#redefine the built-ins
def __setlocal__(name, value):
    if old_getlocal(name, value) in constants:
        raise ConstantError()
    else:
        old_setlocal(name, value)

a = 1
b = 2
c = 3

constants = [b, c]

a = 4 #OK
b = 5 #IMPOSSIBLE

constant.remove(c)

c = 6 #OK


I found such approach more flexible than introducing a 'const' in python
(not a good thing I think). And it enable many other customizations...

Note : the fact that in my example I DO NOT access to any kind of writable
local dictionary! This on purpose, the only way to access this dictionary
should be : locals(), __getlocal__() and __setlocal__() built-ins.

Pierre-Yves Martin
pym<dot>aldebaran< a t >gmail<dot>com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070218/7a3c26c6/attachment.html>

From rnd at onego.ru  Sun Feb 18 21:21:16 2007
From: rnd at onego.ru (Roman Susi)
Date: Sun, 18 Feb 2007 22:21:16 +0200
Subject: [Python-ideas] Pictograms in Python?
Message-ID: <45D8B53C.4050607@onego.ru>

Hi!

I have recently came across this part of Unicode spectrum:

http://www.nttdocomo.co.jp/english/service/imode/make/content/pictograph/list/index.html

While it is understood that the visual representation depends on many
things and doesn't depend on Python that much, it could perhaps be
useful to have unicodedata contain at least translations, so that

unicodedata.name(unichr(0xE63F))

be something like: "PICTOGRAM CLOUDY"

Still, I donno how standard is that range of Unicode characters, but if
Unicode really has those their support could be funny...

Hopefully, BDFL will 0xE70B that 0xE6F0

See you 0xE6B7,
Roman


From jcarlson at uci.edu  Sun Feb 18 22:12:21 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 18 Feb 2007 13:12:21 -0800
Subject: [Python-ideas] local variable access : __getlocal__ and
	__setlocal__
In-Reply-To: <324b1c9b0702181216w5f0468dcme2ae79a6b5ecc546@mail.gmail.com>
References: <324b1c9b0702180159x7bc59296o44c01eb2b7d13f56@mail.gmail.com>
	<324b1c9b0702181216w5f0468dcme2ae79a6b5ecc546@mail.gmail.com>
Message-ID: <20070218125633.AD94.JCARLSON@uci.edu>


"Pierre-Yves Martin" <pym.aldebaran at gmail.com> wrote:
> Currently it's possible to customize attribute access via methods :
> 
> __getattribute__(self, name)
> __getattr__(self, name)
> __setattr__(self, name, value)
> 
> 
> but it is not possible to customize local variable access. It would be
> useful for example to allow implementation of a local constant or any other
> on-access/on-modification behavior. The function should could be:
[snip]

To offer the ability to control such access in a function-local scope
would result in significant slowdown during the execution of the
function body.  This issue would propagate everywhere, as with your
offered semantics, every name lookup would result in a function call
(except for the __getlocal__ and __setlocal__ lookups apparently), and
function calls in Python are significantly slower than dictionary
lookups (globals, etc.) or array lookups (function local accesses).

There are many other reasons to dislike your proposal, but in an attempt
to get something done today, I'll restrain myself.


> #Sample 2 : Constants
[snip]

I believe this is the wrong approach.  If you want to define constants,
create a class with read-only properties.  Alternatively, recent Pythons
allow for passing a dict subclasses to the global and local dictionaries
in the exec statement which implement your constant definitions.  Both
options can offer read-only constants, are available *now*, and even
better, won't destroy the performance of Python.

If you couldn't guess, -1 .

 - Josiah



From ian.bollinger at gmail.com  Sun Feb 18 22:07:00 2007
From: ian.bollinger at gmail.com (Ian D. Bollinger)
Date: Sun, 18 Feb 2007 16:07:00 -0500
Subject: [Python-ideas] Pictograms in Python?
In-Reply-To: <45D8B53C.4050607@onego.ru>
References: <45D8B53C.4050607@onego.ru>
Message-ID: <45D8BFF4.1090004@gmail.com>

U+E000...U+F8FF are for private use, i.e. not part of the official 
Unicode specification. Some of those symbols are part of the standard, 
but obviously have different code points. For instance, the CLOUD is 0x2601.

Roman Susi wrote:
> Hi!
>
> I have recently came across this part of Unicode spectrum:
>
> http://www.nttdocomo.co.jp/english/service/imode/make/content/pictograph/list/index.html
>
> While it is understood that the visual representation depends on many
> things and doesn't depend on Python that much, it could perhaps be
> useful to have unicodedata contain at least translations, so that
>
> unicodedata.name(unichr(0xE63F))
>
> be something like: "PICTOGRAM CLOUDY"
>
> Still, I donno how standard is that range of Unicode characters, but if
> Unicode really has those their support could be funny...
>
> Hopefully, BDFL will 0xE70B that 0xE6F0
>
> See you 0xE6B7,
> Roman
>   


From taleinat at gmail.com  Sun Feb 18 22:31:06 2007
From: taleinat at gmail.com (Tal Einat)
Date: Sun, 18 Feb 2007 23:31:06 +0200
Subject: [Python-ideas] local variable access : __getlocal__ and
	__setlocal__
In-Reply-To: <324b1c9b0702181216w5f0468dcme2ae79a6b5ecc546@mail.gmail.com>
References: <324b1c9b0702180159x7bc59296o44c01eb2b7d13f56@mail.gmail.com>
	<324b1c9b0702181216w5f0468dcme2ae79a6b5ecc546@mail.gmail.com>
Message-ID: <7afdee2f0702181331u3bd81519rfcaaac0a12bc5f06@mail.gmail.com>

On 2/18/07, Pierre-Yves Martin <pym.aldebaran at gmail.com> wrote:
>
> Currently it's possible to customize attribute access via methods :
>
> __getattribute__(self, name)
> __getattr__(self, name)
> __setattr__(self, name, value)
>
>
> but it is not possible to customize local variable access. It would be
> useful for example to allow implementation of a local constant or any other
> on-access/on-modification behavior. The function should could be:
>
> __getlocal__(name)
> __setlocal__(name, value)


-1

IMO this would allow changing the basic behavior of Python too much.
Variable assignment should be variable assignment.

I am generally wary of such behind-the-scenes magic, since it can easily be
taken too far, resulting in highly unreadable code -- not very Pythonic.

- Tal Einat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070218/a65cc773/attachment.html>

From jan.kanis at phil.uu.nl  Mon Feb 19 23:55:45 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Mon, 19 Feb 2007 23:55:45 +0100
Subject: [Python-ideas] immutable classes [was: pre-PEP: Default
	Argument Expressions]
In-Reply-To: <fb6fbf560702171918y33d79991j727b2ccdfeef8513@mail.gmail.com>
References: <fb6fbf560702171918y33d79991j727b2ccdfeef8513@mail.gmail.com>
Message-ID: <op.tn0py8iid64u53@jan-lenovo>

On the 'using old semantics when you really want to' part, that's very  
well possible with a decorator under the proposed semantics:

def caching(**cachevars):
	def inner(func):
		def wrapper(**argdict):
			for var in cachevars:
				if not var in argdict:
					argdict[var] = cachevars[var]
			return func(**argdict)
		return wrapper
	return inner

@caching(cache={})
def foo(in, cache):
	result = bar(in)
	cache[in] = result
	return result

This implementation of caching doesn't handle positional args, but it can  
be made to. One such decorator would still be a net win of several hundred  
lines of code in the standard lib.


Of course, IMHO, the real fix to this is to 1) have default expressions be  
evaluated at calltime, and 2) have _all_ lexical variables be bound at  
definition time and 3) make them immutable. Then something like

lst = []
for i in range(10):
	lst.append(lambda i: i*i)

would work. That would be a real win for functional programming. (good  
thing)
Unfortunately Guido's decided not to support (1), and (2) has been  
proposed some time ago and didn't make it. In both cases because it would  
be to big a departure from how Python currently works. (3) is quite  
impossible in a language like python. <mode=dreaming> I just hope if  
python were designed today it would have done these. </mode>

- Jan


From jcarlson at uci.edu  Tue Feb 20 00:17:53 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Mon, 19 Feb 2007 15:17:53 -0800
Subject: [Python-ideas] immutable classes [was: pre-PEP: Default
	Argument Expressions]
In-Reply-To: <op.tn0py8iid64u53@jan-lenovo>
References: <fb6fbf560702171918y33d79991j727b2ccdfeef8513@mail.gmail.com>
	<op.tn0py8iid64u53@jan-lenovo>
Message-ID: <20070219151513.ADB4.JCARLSON@uci.edu>


"Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
[snip]
> lst = []
> for i in range(10):
> 	lst.append(lambda i: i*i)

You must mean something like...
    lambda j: i*j

> <mode=dreaming> I just hope if  
> python were designed today it would have done these. </mode>

Probably not.  Value binding breaks closures.

 - Josiah



From ncoghlan at gmail.com  Tue Feb 20 14:28:23 2007
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 20 Feb 2007 23:28:23 +1000
Subject: [Python-ideas] [Python-3000] immutable classes [was: pre-PEP:
 Default Argument Expressions]
In-Reply-To: <op.tn0py8iid64u53@jan-lenovo>
References: <fb6fbf560702171918y33d79991j727b2ccdfeef8513@mail.gmail.com>
	<op.tn0py8iid64u53@jan-lenovo>
Message-ID: <45DAF777.8070708@gmail.com>

Jan Kanis wrote:
<mode=dreaming> I just hope if
> python were designed today it would have done these. </mode>

If Python had done these, it wouldn't be Python ;)

There are many, many programming language design decisions which have 
good arguments on each side (and some which seem obviously correct may 
involve hidden costs which aren't appreciated until after it is too late 
to change them). That's one of the major reasons why there are so many 
different programming languages out there.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org


From greg.ewing at canterbury.ac.nz  Thu Feb 22 02:31:33 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 22 Feb 2007 14:31:33 +1300
Subject: [Python-ideas] [Python-Dev] Making builtins more efficient
In-Reply-To: <erinut$mg8$1@sea.gmane.org>
References: <1141879691.11091.78.camel@grey> <440FF9CB.5030407@gmail.com>
	<79990c6b0603090400h25dd2c7ev3d5c379f6529f3c2@mail.gmail.com>
	<1141915806.11091.127.camel@grey> <4410BE69.3080004@canterbury.ac.nz>
	<1171984065.22648.47.camel@grey> <erinut$mg8$1@sea.gmane.org>
Message-ID: <45DCF275.2040601@canterbury.ac.nz>

Giovanni Bajo wrote:

> Are you aware of this patch, which is still awaiting review?
> https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1616125&group_id=5470

That scheme seems unnecessarily complicated to me.
I don't see a need to mess around with timestamps,
except to preserve the ability to change builtins
on the fly, and I don't think it's worth going to
great lengths to preserve that.

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | Carpe post meridiem!          	  |
Christchurch, New Zealand	   | (I'm not a morning person.)          |
greg.ewing at canterbury.ac.nz	   +--------------------------------------+


From greg.ewing at canterbury.ac.nz  Thu Feb 22 02:25:56 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 22 Feb 2007 14:25:56 +1300
Subject: [Python-ideas] [Python-Dev] Making builtins more efficient
In-Reply-To: <1171984065.22648.47.camel@grey>
References: <1141879691.11091.78.camel@grey> <440FF9CB.5030407@gmail.com>
	<79990c6b0603090400h25dd2c7ev3d5c379f6529f3c2@mail.gmail.com>
	<1141915806.11091.127.camel@grey> <4410BE69.3080004@canterbury.ac.nz>
	<1171984065.22648.47.camel@grey>
Message-ID: <45DCF124.6040101@canterbury.ac.nz>

Steven Elliott wrote:

> What I have in mind may be close to what you are suggesting above.

My idea is somewhat more uniform and general than that.

For the module dict, you use a special mapping type that
allows selected items to be accessed by an index as well
as a name. The set of such names is determined when the
module's code is compiled -- it's simply the names used
in that module to refer to globals or builtins.

The first time a given builtin is referenced in the module,
it will be unbound in the module dict, so it is looked up
in the usual way and then written into the module dict,
so it can subsequently be retrieved by index.

The advantages of this scheme over yours are that it speeds
access to module-level names as well as builtins, and it
doesn't require the compiler to have knowledge of a
predefined set of names.

It does entail a slight semantic change, as changes made
to a builtin won't be seen by a module that has already
used that builtin for the first time. But from what Guido
has said before, it seems he is willing to accept a change
like that if it will help.

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | Carpe post meridiem!          	  |
Christchurch, New Zealand	   | (I'm not a morning person.)          |
greg.ewing at canterbury.ac.nz	   +--------------------------------------+


From greg.ewing at canterbury.ac.nz  Sat Feb 24 00:36:49 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 24 Feb 2007 12:36:49 +1300
Subject: [Python-ideas] [Python-Dev] bool conversion wart?
In-Reply-To: <45DF0C84.8020403@ronadam.com>
References: <erlclc$ehu$1@sea.gmane.org>
	<d06a5cd30702221708j475301cate0fb810de6de73e7@mail.gmail.com>
	<erlf32$pob$1@sea.gmane.org>
	<3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com>
	<erlgsv$2g8$1@sea.gmane.org> <45DEC783.8040601@canterbury.ac.nz>
	<45DF0C84.8020403@ronadam.com>
Message-ID: <45DF7A91.8020402@canterbury.ac.nz>

Ron Adam wrote:

> Ok, so what if...  instead of bool being a type, it be a keyword that is 
> just a nicer way to spell 'not not'?

The main benefit of having a bool type is so that its
values display as "True" and "False" rather than whatever
surrogate values you happen to have used to represent
true and false.

The fact that bool() can be used canonicalise a truth
value is secondary -- the same thing could just as well
be achieved by a plain function (as it was before bool
became a type). It's an extremely rare thing to need
to do in any case.

If any change is to be made to bool in 3.0, my preference
would be to keep it but make it a separate type.

 > Are 'and' and 'or' going to be changed in 3.0 to return bool types too?

I very much hope not! Their current behaviour is very
useful, and I would hate to lose it.

--
Greg


From larry at hastings.org  Sat Feb 24 02:28:12 2007
From: larry at hastings.org (Larry Hastings)
Date: Fri, 23 Feb 2007 17:28:12 -0800
Subject: [Python-ideas] A Pythonic replacement for ConfigParser?
Message-ID: <45DF94AC.7040903@hastings.org>


Ah, yes, ConfigParser, the red-headed stepchild of Python 
serialization.  It seems ConfigParser has been around for a /very/ long 
time... and it shows.  In particular, its interface is pretty clunky 
when viewed with a modern Python programmer's eye.

I propose hacking up ConfigParser so it has a more Pythonic 
interface--specifically, so it acts more-or-less like a two-level nested 
dict.  I'll call the rewritten module "configdict" for now.

Here's the full list of what I have in mind:
    * a configdict object behaves like a dict, but can only contain 
configdict.section objects
    * a configdict.section object behaves like a dict, but can only 
contain strings
    * obviously, configdict and configdict.section would be new-style 
classes
    * write the initial implementation as a wrapper around RawConfigParser
    * "defaults" argument to constructor -> configdict.update() or 
configdict[section].update()
    * sections() -> configdict.iternames()
    * add_section(s) -> configdict[s] = section
    * has_section(s) -> s in configdict
    * readfp(fp) -> configdict.read(fp)
    * add a new function parse(s) which parses s as if it were a file 
(probably just wraps in stringio and calls read())
    * get(section, option) -> configdict[section][option] or 
configdict[section].get(option)
    * items(section) -> configdict[section].iteritems()
    * write(fp) -> unchanged
    * remove_option(section, option) -> del configdict[section][option]
    * remove_section(section) -> del configdict[section]
    * drop read(filenames), getint(), getboolean()
    * not sure how I feel about optionxform(), could keep it, could drop it
    * drop the spooky "magic interpolation feature" entirely (explicit, 
not implicit)

Worth pursuing?

Cheers,


/larry/


From mail at manuzhai.nl  Sat Feb 24 08:48:16 2007
From: mail at manuzhai.nl (Manuzhai)
Date: Sat, 24 Feb 2007 08:48:16 +0100
Subject: [Python-ideas] A Pythonic replacement for ConfigParser?
In-Reply-To: <45DF94AC.7040903@hastings.org>
References: <45DF94AC.7040903@hastings.org>
Message-ID: <eroqk5$e6p$1@sea.gmane.org>

Larry Hastings wrote:
> Worth pursuing?

See also: http://www.voidspace.org.uk/python/configobj.html

Regards,

Manuzhai



From fdrake at acm.org  Sun Feb 25 17:52:05 2007
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Sun, 25 Feb 2007 11:52:05 -0500
Subject: [Python-ideas] A Pythonic replacement for ConfigParser?
In-Reply-To: <45DF94AC.7040903@hastings.org>
References: <45DF94AC.7040903@hastings.org>
Message-ID: <200702251152.06124.fdrake@acm.org>

On Friday 23 February 2007 20:28, Larry Hastings wrote:
 > Ah, yes, ConfigParser, the red-headed stepchild of Python
 > serialization.  It seems ConfigParser has been around for a /very/ long
 > time... and it shows.  In particular, its interface is pretty clunky
 > when viewed with a modern Python programmer's eye.

There are a couple of aspects that need to be considered when selecting a 
replacement for ConfigParser.  The API is something everyone seems to 
understand needs improvements.  There are different ways to approach it in 
detail, and I don't think it's terribly important (though it would be nice).

The other is whether existing ConfigParser files needs to be supported.  If 
arbitrary files compatible with ConfigParser must work with the replacement, 
there are a lot of syntactical weirdnesses that need to be supported, and I 
don't know that re-implementing that is valuable at all.

Something that strictly supports ".ini" file syntax has the problem that it 
doesn't support arbitrary ConfigParser files, and that there's no appearant 
real specification for the format.

There are many libraries designed to provide support for general configuration 
files without supporting either the old ConfigParser files, these can provide  
a more "modern" API, but share the issue of compatibility with existing data 
files.

Given that data is harder to change than code, I'm not inclined to replace 
ConfigParser at all.  A new library (newly written or newly selected) could 
be added to avoid the ConfigParser issues for configuration files for new 
applications, but there's not really much value in a ConfigParser 
replacement.


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>


From jan.kanis at phil.uu.nl  Sun Feb 25 20:46:17 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Sun, 25 Feb 2007 20:46:17 +0100
Subject: [Python-ideas] immutable classes [was: pre-PEP: Default
	Argument Expressions]
In-Reply-To: <20070219151513.ADB4.JCARLSON@uci.edu>
References: <fb6fbf560702171918y33d79991j727b2ccdfeef8513@mail.gmail.com>
	<op.tn0py8iid64u53@jan-lenovo>
	<20070219151513.ADB4.JCARLSON@uci.edu>
Message-ID: <op.tobk7fr5d64u53@jan-lenovo>

On Tue, 20 Feb 2007 00:17:53 +0100, Josiah Carlson <jcarlson at uci.edu>  
wrote:
>
> "Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
> [snip]
>> lst = []
>> for i in range(10):
>> 	lst.append(lambda i: i*i)
>
> You must mean something like...
>     lambda j: i*j

yes. my mistake.

>> <mode=dreaming> I just hope if
>> python were designed today it would have done these. </mode>
>
> Probably not.  Value binding breaks closures.

That depends on how you exactly define closures. The basics of having an  
inner function with free variables and initializing those free variables  
to the values they have in the parent scope still works.

>
>  - Josiah

On Tue, 20 Feb 2007 14:28:23 +0100, Nick Coghlan <ncoghlan at gmail.com>  
wrote:
> Jan Kanis wrote:
> <mode=dreaming> I just hope if
>> python were designed today it would have done these. </mode>
>
> If Python had done these, it wouldn't be Python ;)

True. Let's make the better-than-python language. (aiming high) ;)


From jcarlson at uci.edu  Sun Feb 25 23:45:08 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 25 Feb 2007 14:45:08 -0800
Subject: [Python-ideas] immutable classes [was: pre-PEP: Default
	Argument Expressions]
In-Reply-To: <op.tobk7fr5d64u53@jan-lenovo>
References: <20070219151513.ADB4.JCARLSON@uci.edu>
	<op.tobk7fr5d64u53@jan-lenovo>
Message-ID: <20070225144206.AE45.JCARLSON@uci.edu>


"Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
> On Tue, 20 Feb 2007 00:17:53 +0100, Josiah Carlson <jcarlson at uci.edu>  
> wrote:
> > "Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
> >> <mode=dreaming> I just hope if
> >> python were designed today it would have done these. </mode>
> >
> > Probably not.  Value binding breaks closures.
> 
> That depends on how you exactly define closures. The basics of having an  
> inner function with free variables and initializing those free variables  
> to the values they have in the parent scope still works.

It would break closures as defined for Python version 1.? to Python 2.5,
Python 3.x, and beyond.

Changing the semantics of Python closures is not the right thing to do,
whether in 2.6, 3.x or otherwise.


 - Josiah



From jan.kanis at phil.uu.nl  Wed Feb 28 16:09:52 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Wed, 28 Feb 2007 16:09:52 +0100
Subject: [Python-ideas] immutable classes [was: pre-PEP: Default
	Argument Expressions]
In-Reply-To: <20070225144206.AE45.JCARLSON@uci.edu>
References: <20070219151513.ADB4.JCARLSON@uci.edu>
	<op.tobk7fr5d64u53@jan-lenovo>
	<20070225144206.AE45.JCARLSON@uci.edu>
Message-ID: <op.togseqfld64u53@jan-lenovo>

On Sun, 25 Feb 2007 23:45:08 +0100, Josiah Carlson <jcarlson at uci.edu>  
wrote:

>
> "Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
>> On Tue, 20 Feb 2007 00:17:53 +0100, Josiah Carlson <jcarlson at uci.edu>
>> wrote:
>> > "Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
>> >> <mode=dreaming> I just hope if
>> >> python were designed today it would have done these. </mode>
>> >
>> > Probably not.  Value binding breaks closures.
>>
>> That depends on how you exactly define closures. The basics of having an
>> inner function with free variables and initializing those free variables
>> to the values they have in the parent scope still works.
>
> It would break closures as defined for Python version 1.? to Python 2.5,
> Python 3.x, and beyond.
>
> Changing the semantics of Python closures is not the right thing to do,
> whether in 2.6, 3.x or otherwise.
>

Yup. That's why I used '<mode=dreaming>' and talked about if python were  
designed today, i.e. it didn't exist yet. Of course, that's not actually  
the case so it's a purely theoretical point.